Impact of Continuous Delivery Beyond Engineering

It seems like the practice of Continuous Delivery (CD) has finally entered the mainstream of software development. More and more companies like Etsy, Facebook, and GitHub are doing multiple releases a day using a fully automated pipeline.

CD works on simple premise: every code change is deployed directly to production. This seemingly innocent requirement has a major impact on every discipline involved with building software products.

Most people (or at least most devs) tend to focus on the impact CD has on engineering activities. And there is no question that the impact is significant because CD requires a very high degree of confidence in the pipeline. The act of automatically building, testing, and deploying software has to be reliable enough that a miscue in any of those steps is essentially a Six Sigma event.

And yet, although the impact on engineering practices is significant, I would argue that the nature of that impact is evolutionary for agile teams already practicing Continuous Integration (Of course, for teams which aren’t already doing CI, CD would be revolutionary. That said, moving directly to CD without first doing CI is probably a bridge too far.)

On the other hand, the impact of CD on non-engineering activities like quality assurance, product development, and release management can be revolutionary. And it’s this impact I want to explore.

Impact on Quality Assurance

The goal of traditional QA is to prevent bugs from going to production. Obviously, assuring quality has non-trivial labor and opportunity costs. Someone has to test the app to ensure that it works as expected. And, while this testing is going on, users aren’t getting the benefits of new capabilities.

So why do we spend so much time and money on QA? Is it because we believe that it’s possible to prevent all bugs? Maybe, though I don’t know if anyone seriously expects a software system of any complexity to ever be bug-free.

A more likely motivator is the underlying assumption that bugs are much costlier to fix once they’re in production. And while it’s debatable whether this assumption was ever valid, fixing production bugs certainly has overhead. That overhead comes in the form of whatever ceremonies your shop requires before deploying a change to production.

So, QA spends a lot of time trying to find and fix bugs before software ever sees customers. We have all kinds of fun activities like code freezes, bug bashes, and manual regression tests designed to prevent bugs from going out. And again, these activities can be time consuming and expensive.

But what if the cost of fixing production bugs was the same as or less than the cost of preventing them? What would that do to the practice of Quality Assurance? Would it still be relevant? How would it need to change?

Well, CD does have an impact on the economics of bugs. It not only makes showstopping defects potentially less likely, but it also makes it possible to quickly fix them. Therefore, the practice of QA must evolve to take full advantage of what CD has to offer.

As Noah Sussman so eloquently pointed out in his talk on continuous improvement, the focus has to shift from prevention to rapid discovery and resolution. In other words, the most cost-effective thing to do in CD shops is to spend time monitoring real production activity to find and quickly fix bugs.

Another interesting perspective is to expand the definition of bugs. Instead of only considering actual code defects, think of the overall user experience. Is a specific path through the application taking too long? How about a new button not being used, in spite of expectations to the contrary.

One last point on QA. By no means am I suggesting that proactive application testing is irrelevant. On the contrary, having a good set of automated tests is necessary for Continuous Delivery.

Impact on Product Development

What about product development? Continuous Delivery impacts it in at least two ways: reducing feedback loops and decoupling production deployments from releases.

Reduced feedback loops

In traditional agile shops, the time between a feature idea and real-life usage of that feature is measured in weeks, possibly months. That’s a pretty long time to wait for feedback. Moreover, how quickly can a product manager reasonably expect to iterate on that feedback? Again, think weeks, not days.

So, we have all types of practices to mitigate the duration of this feedback loop: user testing sessions, betas, and so on. Now, these activities can certainly be effective, but they’re often not a good substitute for real world usage.

With CD, it’s conceivable to get feedback much sooner, especially with techniques like A/B tests. So, what does this mean to product development? In a word, instrumentation: instrument features to understand usage patterns, use the data gathered to gain insights into user needs, and make modifications based on those insights.

Decoupling releases from deployments

Continuous Delivery decouples feature releases from code deployments. Even though every change makes its way to production immediately, it may not be visible to the end user. The act of actually “releasing a feature” becomes a matter of flipping a switch (known as config flags, “feature flips”, feature toggles, etc).

Every story is either a part of a larger feature or can be deployed independently (as is often the case with defect fixes). This requirement forces a certain level of discipline because story sets must be planned ahead of time. At the same time, product managers get a new level of flexibility because the definition of a “finished feature” is completely fluid and can be redefined as needed.

Impact on Release Management

Finally, consider release management. In shops which release infrequently, releases are big events. And as with any big event, someone has the responsibility of managing it.

Code branches must be stabilized, environments created, regression tests run, bugs triaged and resolved (or deferred), and code moved to production. With so many moving pieces and so many players, someone needs to play the role of a air traffic controller.

But what if none of this ceremony was needed because everything is automated? What role if any does release management play? What value does it add? Once the release machinery becomes self-managing, is the role of a release manager still needed?

Final Thought

Adoption of Continuous Delivery will act as a major forcing function for the entire product development organization. Every role is impacted because a lot of the assumptions which drive traditional practices may no longer be valid.

References

http://www.slideshare.net/noahsussman/continuous-improvement-devops-day-mountain-view-2012

http://codeascraft.etsy.com/2011/02/04/how-does-etsy-manage-development-and-operations/

http://code.flickr.com/blog/2009/12/02/flipping-out/

You may also like:

• The Microsoft Tax
• Continuous Delivery For the Enterprise
• Painless Production Deployments

• ContinuousDelivery
• ProductDevelopment
• QA

Did you love / hate / were unmoved by this post?
Then show your support / disgust / indifference by following me on Twitter!