.svg)
Engineering teams have stopped asking whether AI agents can write code. The question now is how to run dozens of them at once without losing track of what shipped, why it shipped, and whether it works. That shift is why the software factory has become one of the most talked-about ideas in engineering this year.
In short, a software factory is a way of running software delivery as an assembly line of AI agents. The idea is easy to like and easy to over-trust. A factory is only as reliable as the checks at the end of the line, and most of the conversation so far has focused on the line itself.
This guide explains what a software factory is, how its stages work, how light and dark factories differ, and how the model compares with DevOps and Agile. Then we get to the part that decides whether any of it is safe to run: verification, and the quality gates a factory needs to earn trust.
The short answer
A software factory, at a glance:
- What it is: a system where AI agents plan, build, test, and ship software through repeatable loops.
- Where humans fit: they set intent and approve work at defined gates.
- Light vs dark: light factories keep people reviewing code, while dark factories have nobody reading it.
- The real constraint: how reliably the code can be verified, not how fast agents can generate it.
- What it needs: independent validation, traceable evidence, and a human gate where risk is high.
A factory is only as reliable as the checks at the end of the line.
What Is a Software Factory?
The term is older than AI. For decades it described organizations that built software from reusable assets and product lines, aiming for repeatable output instead of one-off projects. The AI-era meaning keeps the repeatability but changes who does the making. Agents do the building, and the line is defined by prompts, tools, and checks rather than by templates.
A useful way to keep the vocabulary straight comes from engineer Addy Osmani, who separates three layers. A loop is one agent repeating one job, such as picking up a ticket and opening a pull request. A harness is everything wrapped around that loop: the sandbox it runs in, the tools it can call, the memory it carries, and the gates it must pass. A factory is many harnessed loops fed from a shared queue and drained through a review gate.
That hierarchy explains why the factory conversation overlaps with the one about harness engineering. A single well-built harness is the unit you can reason about. A factory is what you get when you multiply it, and every weakness in the harness gets multiplied too.
You may also see the phrases AI software factory or agentic software factory. They describe the same thing: the factory model, with agents as the workforce.
How a Software Factory Works: The Core Stages
Most descriptions of a working factory follow the same arc. O'Reilly's walkthrough of an internal software factory lays out eight stages: intake and triage, brainstorming, planning, implementing, review, CI, release, and monitoring that feeds new work back into intake. It helps to group them into three jobs.
The first job is deciding what to build. Intake and triage sort incoming requests, brainstorming explores options, and planning turns a chosen option into a specification an agent can act on. Quality problems that start here are the most expensive ones, because an agent will build a flawed plan faithfully and quickly.
The second job is building and checking. Implementation produces the change, review inspects it, and CI runs the automated checks. This is the part of the line most teams already know from DevOps, with the difference that an agent, not a person, produced the change.
The third job is learning from production. Release and monitoring close the loop by sending real-world signals, such as errors, usage, and incidents, back into intake. A factory that skips this stage ships fast but never improves.
Underneath all three sits the context layer. Files such as AGENTS.md, internal wikis, and coding standards tell agents what the team considers correct. They constrain what agents can plan, so they deserve the same care as production code. Stale context is a quiet way for a factory to produce confident, consistent mistakes.
Light vs Dark Software Factories
Not every factory looks the same, and the sharpest difference is how much humans read. In a dark software factory, nobody reads the code. Machines write it, machines verify it, and humans only look at outcomes. In a light factory, agents still do most of the building, but people keep their judgment at the points where a wrong call is expensive.
In practice this is a spectrum, not a switch. PostHog describes where it sits on that spectrum: agents write roughly 70% of its pull requests, yet humans still skim at least 80% of them. StrongDM is the best-known example of going fully lights-off, an approach examined in a Stanford Law analysis of agent-built software.
Going dark has a price that Osmani calls comprehension debt. When no one reads the code, the team gradually loses its understanding of how the system works, and that debt comes due during incidents, audits, and migrations. The more of the line you remove humans from, the more you depend on automated verification to stand in for their understanding.
A practical reading of the spectrum: the right position depends on blast radius. An internal tooling script can run dark with little risk. A payments flow should not. Most teams will end up with different settings for different parts of the codebase.
Software Factory vs DevOps and Agile
A common question is whether a software factory replaces DevOps. It does not. The factory sits on top of CI/CD, and DevOps practices stay in place underneath it. What is new is who produces the change: pipelines already moved code from commit to production, and now agents also write the commit.
Agile, meanwhile, organizes people around iterations and conversations. A factory organizes agents around queues and gates. The two can coexist, since the humans in a factory still need to decide what is worth building, but the cadence shifts from sprint ceremonies to continuous flow.
| Dimension | DevOps and CI/CD | Agile | Software factory |
|---|---|---|---|
| Unit of work | A commit moving through a pipeline | A user story in a sprint | A task handled by a harnessed agent loop |
| Who executes | Pipelines and engineers | Cross-functional teams | AI agents, with humans at gates |
| What gates the release | Automated tests and approvals | Definition of done and sprint review | Independent verification plus human gates |
| Where humans spend time | Writing code, tuning pipelines | Planning, building, reviewing | Setting intent, reviewing exceptions, improving the harness |
| Main risk | Slow or flaky pipelines | Scope drift and handoffs | Unverified output at high volume |
The last row is the one to hold on to. DevOps worried about getting changes through quickly. A factory has already solved speed. Its risk is that speed outruns anyone's ability to confirm the output is right.
Why Verification Is the Constraint
Osmani makes the case plainly: in a software factory, verification, not generation, is the real constraint. Agents can produce code far faster than people can check it, so the sensible rule is that autonomy should never exceed what you can cheaply and reliably verify.
The evidence for how fast generation can go is real. OpenAI's harness engineering report describes a team that shipped roughly one million lines of code and about 1,500 pull requests over five months, with no hand-written code. The same report is clear about what made that possible: tests, linters, and agents that checked their own work against the running application. Throughput came from the checks, not in spite of them.
The risk is just as concrete. The Stanford Law piece on StrongDM describes agents that, when aimed at passing tests, wrote code that simply returned true. It also raises the circularity problem: when the same kind of system builds the code and judges it, blind spots can be shared, and a passing result tells you less than it appears to. A test suite written or tuned by the thing under test is a weak witness.
This is where many factory designs have a gap. They invest heavily in generation, then treat testing as a final stage rather than a layer that runs alongside every other one. Our guide on debugging agentic AI covers what happens downstream when that layer is thin: failures that are hard to trace to a root cause.
What Quality Gates a Software Factory Needs
If verification is the constraint, quality gates are how you relieve it. A gate is a point where work cannot move forward until it passes a defined check. A factory needs a few kinds of them, working together.
- Independent validation. The agent that writes code should not be the one grading it. O'Reilly describes an adversarial QA agent for this reason, and Osmani's version is blunter: an agent grading its own homework is too nice.
- Scenario tests kept out of the agents' reach. BCG Platinion makes the case for holding back tests that agents cannot see or tune against, so a pass reflects behavior rather than a memorized answer.
- Layered feedback loops. Run lint, unit, integration, and end-to-end checks in order of cost, so cheap failures stop the line early and expensive checks run only on changes that deserve them.
- Traceable evidence. For every change, the record should show what was generated, what was tested, what the results were, and who approved it. Without that trail, a dark factory cannot be audited.
- A named human gate where risk is high. Choose the changes where a wrong call is costly, such as payments, permissions, and data handling, and make a specific person accountable for approving them.
None of these gates is exotic. What is unusual is treating them as part of the line from the first loop, rather than bolting them on after an incident.
Where Katalon True Fits
A factory builds software. A quality layer proves it works. Those are different jobs, and the second one needs its own system of record.
Katalon True is built for that second job. It connects requirements, test cases, execution, evidence, defects, and approvals across manual, automated, and AI-generated testing, so the record of what was tested and what passed lives in one place. Its AI agents each handle a defined step: a Requirement Analyzer, Test Generation Agent, Autonomous Test Runner, Bug Reporter, Root Cause Analyzer, and Report and Insight Generator, orchestrated through the Katalon AI Assistant. The pattern throughout is that AI proposes and a human approves.
To be clear about scope, Katalon True is the quality layer for the testing record. It is not another coding agent in the factory, and it does not govern external agents. Its role is to give the team independent evidence about the output, which is exactly what the gates above depend on.
If you want the wider picture, our agentic QA guide explains how agents fit into testing, and the Katalon True launch post covers the product. You can also explore Katalon True directly.
How to Start Building Your Factory
You do not need a full platform to begin. Think in stages, and move on only when the previous one is stable. O'Reilly's advice is to start with a coding harness and a few small skills before buying anything, and Osmani notes that reliable loops tend to be short, around 3 to 10 steps.
Stage 1 is one loop with one cheap, hard-to-game check. Pick a well-bounded task, such as fixing lint errors or updating dependencies, and give the agent a single automated test it cannot talk its way past.
Stage 2 adds an independent reviewer and one command that boots the full stack. Now a second agent, or a person, checks the first agent's work, and anyone can reproduce the environment the check ran in.
Stage 3 adds traceability and a human gate on risky changes. This is where the evidence trail starts to matter, because volume is high enough that you can no longer remember what happened.
Stage 4 widens the loops. Only now do you add more task types, more agents, and more autonomy, using what the earlier stages taught you about where checks are weak.
Conclusion
Software factories get built quickly and trusted slowly. The teams that get lasting value from them will be the ones that treat verification as part of the line instead of the end of it: independent checks, honest evidence, and a clear human owner where the stakes are high. If you are mapping this to your own testing practice, start with our agentic QA guide, or see how Katalon True keeps the quality record for AI-built software.
|
