This is the second of a series of blog posts I am publishing for sharing some considerations about the challenges of maintaining a downstream of Chromium.
The first part, Maintaining downstreams of Chromium: why downstreams?, provided initial definitions, and an analysis of why someone would need to maintain a downstream.
In this post I will focus on the possible different strategies for tracking the Chromium upstream project changes, and integrating the downstream changes.
Applying changes
The first problem is related to the fact that our downstream will include additional changes on top of upstream (otherwise we would not need a downstream, right?).
There are two different approaches for this, based on the Git strategy used: rebase vs. merge.
Merge strategy
This consists on maintaining an upstream branch and periodically merging its changes to the downstream branch.
In the case of Chromium, the size of the repository and the amount and frequency of the changes is really big, so the chances that merging changes cause any conflict are higher than in other smaller projects.
Rebase strategy
This consists on maintaining the downstream changes as a series of patches, that are applied on top of an upstream baseline.
Updating means changing the baseline and applying all the patches. As this is done, not all patches will cause a conflict. And, for the ones that do, the complexity is far smaller.
Tip: keep the downstream as small as possible
Don’t hesitate to remove features that are more prone to cause conflicts. And at least, think carefully, for each change, if it can be done in a way that matches better upstream codebase.
And, if some of your changes are good for the commons, just contribute them to upstream!
When to update?
A critical aspect of applying upstream changes is how often to do that.
The size and structure of the team involved is highly dependent on this decision. And of course, planning the resources.
Usually this needs to be as much predictable as possible, and bound to the upstream release cycle.
Some examples used in downstream projects:
- Weekly or daily tracking.
- Major releases.
Upstream release policy and rebases
It is not only important how often you track upstream repository, but also what you track.
Chromium, nowadays, follows this procedure for each major release (first number in the version):
- Main development happens in
mainbranch. Several releases are tagged daily. And, mostly every week, one of them is released to thedevchannel for users to try (with additional testing). - At a point, an stabilization branch is created, entering
betastage. The quality bar for what is landed in this branch is raised. Only stabilization work is done. More releases per day are tagged, and again, approximately once per week one is released tobetachannel. - When planned, the branch becomes
stable. This means it is ready for wide user adoption, so thestablechannel will pick its releases from this branch.
That means main (development branch) targets version N, then N-1 branch is the beta (or stabilization branch), and N-2 is the stable branch. Nowadays Chromium targets publishing a new major stable version every four weeks (and that also means a release spends 4 weeks in beta channel).
You can read the official release cycle documentation if you want to know more, including the extended stable (or long term support) channel.
Some downstreams will track upstream main branch. Some others will just start applying the downstream changes when first release lands the stable channel. And some may just start when main is branched and stabilization work begins.
Tip: update often
The more you update, the smaller the change set you need to consider. And that means reducing the complexity of the changes.
This is specially important when merging instead of rebasing, as applying the new changes is done in one step.
This could be hard, though, depending on the size of the downstream. And, from time to time some refactors upstream will still imply non-trivial work to adapt the downstream changes.
Tip: update NOT so often
OK, OK. I just told the opposite, right?
Now, think of a complex change upstream, that implies many intermediate steps, with a moving architecture that evolves as it is developed (something not unfrequent in Chromium). Updating often means you will need to adapt your downstream to all the intermediate steps.
That definitely could mean more work.
A nice strategy for matching upstream stable releases
If you want to publish an stable release almost the same day or week of the official upstream stable release, and your downstream changes are not trivial, then a good moment for starting applying changes is when the stabilization branch happens.
But the first days there is a lot of stabilization work happening, as the quality bar is raised. So… waiting a few days after the beta branch happens could be a good idea.
An idea: just wait for the second version published in beta channel (the first one happens right after branching). That should give still three full weeks before the version is promoted to the stable channel.
Tracking main branch: automate everything!
In case you want to follow upstream main branch, in a daily basis (or even per commit), then it is just not practical to do that manually.
The solution for that is automation, at different levels:
- Applying the changes.
- Resolving the conflicts.
- And, more important, testing the result is working.
Good automation is, though, expensive. It requires computing power for building Chromium often, and run tests. But, increasing test coverage will detect issues earlier, and give an extra margin for resolving them.
In any case, no matter your update strategy is, automation will always be helpful.
Next
This is the end of the post. So, what comes next?
In the next post in this series, I will talk about the downstream size problem, and different approaches for keeping it under control.