Production Bugs: Whose responsibility are they?

Bug fixing is rarely fun. This is especially true for bugs you didn’t create.

If you work on an established application, there is a good chance that a production bug was introduced by someone who is not you. Yet someone who is most likely you needs to fix it. Moreover, production bugs usually demand immediate attention and put extra pressure on you to fix them. All that makes production bug fixing especially not fun.

Fortunately, there are multiple ways of dealing with production bug fixing so that it sucks a bit less for all involved. Unfortunately, none of those ways are especially great.

All For One And One For All

One approach is to have everyone be responsible for fixing production defects. As in, every engineer could work on any defect reported from the field.

The primary appeal of this approach is fairness. When everyone is on the hook for production bugs, no one person (or a small group of people) gets “stuck” fixing the seemingly never-ending stream of bugs while the others are living the good life working on new features.

Making this work in reality is not trivial because it can be distracting and disruptive. If you’re in the middle of working out a domain model for a complex new feature, interrupting that work to fix an urgent IE7 styling bug is not ideal for productivity. Context switching can get expensive.

On top of that, making reliable delivery commitments becomes very difficult. I mean, imagine that you’re half way through an iteration when suddenly all hell breaks loose and you’ve got 2 urgent defects that must be fixed yesterday. What happens to those 20 story points you thought were going to easily get done?

Sure, you could try to plan it better. You could try and pick bugs that are at least in the ball park of what you’re currently doing. And, since you’re already “under the hood”, fixing a few loose ends shouldn’t be too bad.

Unfortunately, you rarely get the luxury of deciding the order in which to fix production bugs. Much more likely is the scenario where your customers decide based on how painful the issues are.
FDA approved generic medications also have successfully established bio equivalence which means that it works in the same mechanism that has the ability to preserve the heart of a man, and prostrate. pfizer online viagra But it is a 50mg viagra sale http://mouthsofthesouth.com/wp-content/uploads/2013/03/MOTS-Tart-4-20 common feature in Vortex Binoculars and is the Viper HD has it on all air-to-glass surfaces. Course curriculum decides levitra uk the quality of driving lessons that are taught to the students. It is basically Sildenafil and hinders cGMP and PDE5. check levitra generika

The Lone Wolf

Another idea is to rotate one or two engineers out of the main team to focus on bug fixing. This way, the pain is still spread evenly across the team, but it’s not nearly as disruptive.

The difficulty with this approach is making transitions seamless. Imagine a situation where you get a new bug a couple of days before you’re scheduled to roll off production support. At first blush, the bug looks reasonable and you decide to take it on.

Fast forward two days. Fixing the bug turned into an archaeological dig. You sort of figured out where to start fixing it, but you’re probably days away from actually finishing. And now you have to stop now and roll off.

What happens to the bug? Well, you could attempt a Vulcan mind meld with the incoming engineer so that he can pick up where you left off. Or you could let him enjoy the same journey of discovery and pain you just went through. Either way, there is non-trivial overhead of transitioning.

A Band of Brothers

Finally, you can have a team dedicated to fixing production bugs. This approach is the middle ground between everyone doing it and just one person doing it. And like most middle grounds, it has something for everyone to dislike.

First, there is a distinct feeling among devs that being on such a team is not unlike being sent to Gulag. This is especially true if you have a hostile code base to deal with.

Second, because this team isn’t actively participating in feature development, there is a tendency to disconnect. After a while you’ll find that understanding of the technology stack is not nearly as deep, dev practices aren’t followed as rigorously, etc.

Final Thought

Fixing production bugs can be painful no matter what organizational structure you adopt. Your challenge is to figure out ways to make it less painful, maybe even enjoyable. But that’s a topic for another post.

You may also like:

• Hostile Code Bases

• hostileCodeBase
• productionSupport

Did you love / hate / were unmoved by this post?
Then show your support / disgust / indifference by following me on Twitter!