Upcoming hotness from RelEng

To kick off the new year, I’d like to share some of the exciting projects we have underway in Release Engineering.


First off we have Balrog
, our next generation update server. Work on Balrog has been underway for quite some time. Last fall we switched beta users
to use it. Shortly after, we did some additional load testing to see if we were ready to flip over release traffic. The load testing revealed some areas that needed optimization, which isn’t surprising since almost no optimization work had been done up to that point!

and Nick
added the required caching, and our subsequent load testing
was a huge success. We’re planning on flipping the switch to divert release users over on January 19th
. o/


Next up we have Funsize
. (Don’t ask about the name; it’s all Laura
‘s fault). Funsize is a web service to generate partial updates between two versions of Firefox. There are a number of places where we want to generate these partial updates, so wrapping the logic up into a service makes a lot of sense, and also affords the possibility of faster generation due to caching.

We’re aiming to have nightly builds use funsize for partial update generation this quarter.

I’d really like to see us get away from the model where the “nightly build” job is responsible for not only the builds, but generating and publishing the complete and partial updates. The problem with this is that the single job is responsible for too many deliverables, and touches too many systems. It’s hard to make and test changes in isolation.

The model we’re trying to move to is where the build jobs are responsible only for generating the required binaries. It should be the responsibility of a separate system to generate partials and publish updates to users. I believe splitting up these functions into their own systems will allow us to be more flexible in how we work on changes to each piece independently.

S3 uploads from automation

This quarter we’re also working on migrating build and test files off our aging file server infrastructure (aka “FTP”, which is a bit of a misnomer…) and onto S3. All of our build and test binaries are currently uploaded and downloaded via a central file server in our data center. It doesn’t make sense to do this when most of our builds and tests are being generated and consumed inside AWS now. In addition, we can get much better cost-per-GB by moving the storage to S3.

No reboots

has been doing awesome work with runner
. One of the primary aims here is to stop rebooting build and test machines between every job. We’re hoping that by not rebooting between builds, we can get a small speedup in build times since a lot of the build tree should be cached in memory already. Also, by not rebooting we can have shorter turnaround times between jobs on a single machine; we can effectively save 3-4 minutes of overhead per job by not rebooting. There’s also the opportunity to move lots of machine maintenance work from inside the build/test jobs themselves to instead run before buildbot starts.

Release build promotion

Finally I’d like to share some ideas we have about how to radically change how we do release builds of Firefox.

Our plan is to create a new release pipeline that works with already built binaries and “promotes” them to the release/beta channel. The release pipeline we have today creates a fresh new set of release builds that are distinct from the builds created as part of continuous integration.

This new approach should cut the amount of time required to release nearly in half, since we only need to do one set of builds instead of two. It also has the benefit of aligning the release and continuous-integration pipelines, which should simplify a lot of our code.

… and much more!

This is certainly not an exhaustive list of the things we have planned for this year. Expect to hear more from us over the coming weeks!

稿源:chris' random ramblings (源链) | 关于 | 阅读提示

本站遵循[CC BY-NC-SA 4.0]。如您有版权、意见投诉等问题,请通过eMail联系我们处理。
酷辣虫 » 综合技术 » Upcoming hotness from RelEng

喜欢 (0)or分享给?

专业 x 专注 x 聚合 x 分享 CC BY-NC-SA 4.0

使用声明 | 英豪名录