How I get AI to write production code

Intro

Product OS is the system I use to build production software with AI. It takes the practices professional engineering teams have relied on for decades, version control, code review, automated testing, continuous integration, security audits, and applies them in an AI native workflow, enforced by automation rather than left to discipline.

This page covers the problems in AI development that Product OS was designed to solve, and how it solves them.

The problems with AI coding

AI writes code quickly, and much of it is good. The failures cluster around everything that surrounds the code:

• Inconsistency. The same prompt, given twice, can produce two different implementations at two different levels of quality. Results depend on phrasing and on chance.
• Hallucination. Models produce code that looks plausible but references functions that don’t exist, misuses an API, or quietly does the wrong thing.
• Completion without evidence. Work gets reported as finished without tests run, checks passed, or anything a reviewer can verify.
• Weak testing. Tests are skipped, written thin, or written by the same model that wrote the code, which means they encode the author’s understanding, including its mistakes.
• Security as an afterthought. Reviews happen late, if at all, and sensitive changes get the same scrutiny as trivial ones.
• Stale context. Each session starts fresh. If the documentation has drifted from the code, the model works from a picture of the codebase that is no longer true.
• No memory of lessons. A correction made in one session is forgotten by the next, so the same mistakes repeat.

These aren’t prompting problems, and better prompts don’t fix them. They are process problems, and process is what fixes them.

Quality is a property of the process, not of the prompt.

How Product OS solves them

Each part of the process exists to close a specific gap:

• Acceptance criteria come before code. Every task gets plain language Given/When/Then statements, written and approved up front, so the builder, the verifier, and I all work from the same definition of done.
• One verify script runs every quality check. Typecheck, lint, tests, and build in a single command, identical on my machine and in CI, so “passing” means the same thing everywhere.
• CI runs the verify script on every pull request to double check for errors before anything can merge. No change is exempt.
• An automated check inspects every pull request to confirm the acceptance criteria and screenshots are attached, and to set the required level of rigor based on the size and sensitivity of the change.
• A second AI verifies the work independently. It writes its own tests from the acceptance criteria, without seeing the builder’s tests, so a misreading of the requirements surfaces as a failing test before merge instead of a bug after release.
• Testing happens in layers. Unit tests are written alongside each task, end to end tests cover the critical paths, and UI changes are verified in a real browser with screenshots, so a defect has to get through several independent checks to reach production.
• Security is reviewed at three points: when a feature is planned, when its code changes, and before launch, because security issues are cheapest to fix early and most expensive after release.
• Branch protection blocks unverified merges. Nothing reaches the main branch until every check passes and I approve it.
• Documentation is regenerated from the code after every merge, so each new session starts with a complete, current picture of the codebase. This closes the context gap behind much of the hallucination problem.
• Lessons carry forward. When something changes how future work should be done, it is recorded and loaded into every later session, so the system improves with each build instead of repeating mistakes.

The machinery itself is verified at setup: the system opens a deliberately failing pull request to confirm the gates actually block it before any real work begins.

EVERY CHANGE PASSES THE SAME GATES

Where consistency comes from

Give an AI the same prompt twice and you can get two different results. That variability is built into how the models work, and no amount of careful prompting removes it.

Product OS reduces it by fixing everything around the generation step. The spec fixes what is being built. The acceptance criteria fix what counts as done. The regenerated docs fix what the model knows about the codebase. The gates fix the quality bar. The model may still write different code on different days, but every version has to satisfy the same criteria, pass the same tests, and clear the same independent review, so the results land in the same narrow range.

That is the practical difference a standardized process makes: the outcome stops depending on how a prompt was phrased or which direction the model happened to take that day.

My role in the process

Once a project is set up, the process calls for my judgment at three points.

There is also one early checkpoint. Before refinement begins, the build is paused and shown to me running locally, so a wrong direction gets caught in minutes rather than after the full pipeline has run.

How the work flows

Projects enter one of two ways. A new product starts from a spec. An existing codebase is audited first: the system maps every feature it finds, I decide what to keep, cut, or defer, and the output is a rebuild plan that enters the pipeline like any other spec.

Setup happens once per project. The spec becomes a build plan with acceptance criteria for every task, and the machinery is installed and verified before any product code is written.

From there, the work runs as a loop. Plan a feature, agree on the acceptance criteria and one measurable definition of success, build on a branch with tests, let the checks go green, walk the preview, merge. The success metric is checked after launch, because a feature that shipped is not necessarily a feature that worked.

TWO ENTRY POINTS, ONE REPEATABLE LOOP

What has to be true before anything merges

Not every change needs the full process, and the decision is not left to judgment. The pull request check computes the required level of rigor from the diff itself: changes that are large or touch sensitive areas, authentication, payments, migrations, dependencies, automatically require the complete process, including independent verification. Small, contained changes move through a lighter version of the same gates. Because the assignment is computed, a risky change cannot be waved through as a small one.

The same principle scales across projects. A prototype gets version control and tests. A deployed product adds CI and branch protection. A product with real users gets the complete set, through feature flags, tagged releases, rollback, and monitoring.

What this produces

The comparison that matters is with the alternative. Without a standardized process, the quality of AI built software depends on how each prompt was phrased, what the model happened to know that session, and how carefully someone reviewed the result. Some days that goes well. Across a whole product, it doesn’t hold.

With Product OS, every change follows the same path: written criteria, layered tests, independent verification, security review, current documentation, and a recorded history. The output is production grade code, correct, tested, secure, and documented, produced consistently rather than occasionally. And because the documentation regenerates and the lessons persist, reliability grows with the product: the hundredth change goes through the same gates as the first.

That is what the process is for: making the quality of the result depend on the system, not on any single session.