We do trunk development. For those who aren’t familiar, all new code from all projects goes into the same mainline branch (a.k.a. trunk). We cut a release branch only when a release is about ready for production.
We didn’t always do this. In fact, up until not too long ago, the idea sounded quite crazy to me. Everything I knew about software development told me that individual projects needed to exist in their own sandboxes. I mean, how else could you release each project independently of others, right?
Setting aside that concern for a moment, let’s talk about why trunk. The primary reason is a shortened feedback loop. In software, the sooner you find that your change impacted something else, the easier it is to fix it. If a change made for project A collides with project B, it’s really good to discover that 2 minutes after it happens vs 2 months after.
The second benefit is that trunk development eliminates The Merge. Most devs have a horror story about how it took 5 guys 2 weeks to merge two feature branches together after each has been going at it on their own for 6 months. “There were 564 breaking changes” they’d say “and we lost 2 men trying to reconcile them“.
Why Not Trunk?
Of course there is no free lunch. Trunk development is great, but it has a cost. First, if you are developing multiple features concurrently and need the flexibility to release them independently, you have to use something like a feature toggle. I won’t recap Martin Fowler’s excellent write-up on this, but needless to say it takes extra effort.
Second, depending on the quality of the code base and the amount of automated tests you have, verifying that your product is ready for release is not trivial. After all, literally anyone on any team can introduce a change at any time that could break your soon-to-be-released change.
Third, working on trunk raises the bar for how you treat the code base because consequences of a broken build are more severe. If you forget to get latest or to run a local build before checking in, resulting broken build could stop work on all teams (yes, you can mitigate the impact of broken builds with build pipelines and such, but the point stands).
Yes, But Does it Scale?
It may sound to you like working on trunk can only work for small teams and code bases. Well, that’s definitely not the case. Trunk development can scale and the most illustrative example of that I could find is Google.
Almost every single project Google has is in a single trunk repository. Like most things Google, the scale of this is thing mind-boggling:
- Over 5,000 developers worldwide access the trunk
- Over 2,000 projects are in active development
- Over 50,000 builds per day (close to 90,000 during weekdays)
- Over 100,000,000 (yes 100 million) test cases run each day
- 20+ code changes per minute
To find out more, check out this great talk from Google’s Ashish Kumar.
How Google is able to do this is not really rocket science. They rely heavily on code reviews (which are actually enforced on submission) and lots of test automation. Oh and they have a lot of good developers.
You may also like:
Did you love / hate / were unmoved by this post?
Then show your support / disgust / indifference by following me on Twitter!