What Are Software Factories Good For?

I’ve been thinking a lot lately about software factories and where they can be most helpful. Specifically, I wanted to understand how they fit with the types of work software developers typically do. And to do that, I think it’s useful to classify development work in terms of uncertainty.

Uncertainty Matrix

We can plot software development on a two-dimensional matrix of uncertainty - what we want to build (requirements) and how we want to build it (technology):

To build production code, we need both types of certainty: a clear spec and a workable technical approach (let’s call it technical design). If we have both, we’re ready to build. But, if we’re missing one or the other (or both), we need to figure it out first.

So, if the requirements are unclear, we build a prototype, have users react to it, and use what we learned to write the spec. If the implementation is unclear, we spike a few technical approaches until one holds up, and that becomes the technical design. Either way, the prototype or the spike gets thrown away since it’s simply a cheap way to remove uncertainty before the expensive part - building the real thing.

AI Enters the Chat

Now, what I described above is what we used to do when we had to write code by hand like animals. However, with the advent of AI, things changed in a pretty profound way because AI completely upended the economics of writing software. Building became so cheap that we often don’t bother with the spec/design at all. Instead, we sit with Claude, try something, react to it, and adjust - until the prototype gradually becomes the feature. Finding out what to build and building it have merged into one loop. The spec still exists, but nobody writes it down first. It takes shape in the conversation and manifests in our heads and chat transcripts.

By the way, this is a perfectly reasonable way to do agentic engineering, especially considering how good the frontier models have gotten at producing clean, well-architected code. However, it does not fit well with software factories for two reasons:

• It’s much too interactive - software factories are meant for long-running coding sessions without much (if any) human input along the way.
• No spec = No Definition of Done - to be maximally effective, software factories need a detailed specification of what to build because they need to check their work against it.

This dynamic brings us to an interesting challenge - can software factories still be useful for work done outside them? And if so, how? Well, I think they can, and it boils down to one key idea - a software factory can help the engineer underwrite the code produced by the agents, even if that code was produced outside of the factory.

Outside Work

Inspired by this idea, I added a feature to my software factory Fabrika called Outside Work. This feature is designed to help the human make sense of what her agent wrote:

Given a branch to look at, Fabrika does a few things:

1. Analyzes what was built and, as much as possible, what should have been built. I should note that, since Fabrika doesn’t have a spec to evaluate the code against, it tries to ascertain intent by looking at commit messages and tests.
2. Performs a code review to try and find any weaknesses or potential defects.
3. Runs application-level checks to make sure no regressions were introduced.

Once Fabrika evaluates the branch, it creates the equivalent of a review packet to help the human understand what was built. If there are specific findings that should be addressed, Fabrika can either fix them itself or give the human a prompt for an external agent. If the human decides to have Fabrika fix stuff, it’ll run the code through a repair loop:

I should also note that Fabrika will not modify the branch it was given. Instead, any code changes (such as creating test probes) are written to a separate branch under fabrika/outside/<branch>.

Review Packet

As mentioned earlier, Fabrika prepares a Review Packet for the human by looking at the code on the branch it was given. This packet is designed to answer two key questions:

1. Is there anything that should be resolved before shipping?
2. What was actually changed, and how?

The information is presented in four tabs - findings, what was built, the code, and checks.

1. Findings

In the Findings tab, Fabrika shows the human any potential issues found during code review. If the human decides to fix them, she can either have Fabrika do it or get a prompt to hand to an external agent:

The goal is to ultimately get the branch to a PR:

2. What Was Built

This tab describes the code found on the branch in four sections:

1. What it adds - a diagram of changes separated into user-facing components, endpoints, and data changes:

2. Key Flows - sequence diagrams of new flows:

3. Where it landed - subsystems where the changes were made:

4. What it claims - what the tests are claiming to validate and what the probes discovered:

3. The Code

The Code tab shows all the diffs for this branch organized into novel, conventional, mechanical, and tests:

The idea here is to help the human conduct an efficient code review by focusing mainly on the important stuff.

4. Checks

Finally, the Checks tab shows the results of running checks that have been defined for this repository:

How Outside Work Fits

So, to summarize this whole thing - while there are obvious advantages to using Fabrika for the entire build (red teaming, confirmed acceptance criteria, etc.), it can help even if it wasn’t used that way:

Why Is This Even in a Factory?

I realize that what I’m describing here may not sound like a software factory concern at all. To many people, software factories are purely about setting up a bunch of agents to work independently on a task given to them. And, going beyond that remit is doing too much.

Obviously, software factories can be really good at helping humans build code faster. There’s no arguing that, and that’s basically what everyone is trying to do. But should they stop there, or should we expect them to help in other ways too?

As I see it, the primary constraint in the modern age of agentic engineering isn’t how quickly we can bang out code, but how well we can keep up with it. Therefore, a system designed to help humans grow the codebase should also help them keep up with that codebase.

If you pull on that thread, a question quickly emerges - assuming that you do want to keep up with your codebase, how do you do it? Well, I think you need purpose-built tooling that goes beyond just reading the diffs to help you quickly digest massive amounts of agentic code.

Finally, I fully allow for the possibility that this is not an evergreen need. One day the models may get so good that reviewing their code will seem like a silly waste of time. But that day isn’t today, and until it comes, we need something to tide us over.