Everyone's Building the Outer Loop

By Jason Whitehorn · July 13, 2026

I hate writing tests. I know they matter, and I write them anyway, but the work is mechanical and I'd rather be building the thing than filling in the hundredth assertion. So when the coding agents got good, the first job I handed over was the boring one.

Tests and documentation. That was the deal I made with myself, and back in March I said so publicly - AI writes those, I write the code that matters. I suspect a fair number of you drew the line in about the same place.

I had it backwards.

Here's what I ran into. The agent writes a nice test. Well named, reads clean, asserts something specific about how the system behaves. It passes, too. And then you sit there looking at it and realize you have no idea whether the assertion is true - not "does it pass," it passes, it was written against the code - but is that actually how this thing is supposed to work? Or did the model make it up?

Bad code fails loudly. It crashes, it won't compile, somebody trips over it. A bad test doesn't fail. It sits there quietly certifying the bug, in green, indefinitely. So the suite I disliked writing but leaned on completely became a suite I couldn't trust, which made the system underneath it a system I couldn't trust either.

The question I actually had wasn't "can AI write my code." It was: how do I know the test is right?

Everything below came out of chasing that one down.

The Hole In The Middle Of Loop Engineering

If you've been anywhere near this stuff lately you've seen the loop talk. Boris Cherny, who runs Claude Code at Anthropic: "I don't prompt Claude anymore. I have loops that are running. They're the ones that are prompting Claude." Peter Steinberger said much the same thing on X and got five million views in a day. Addy Osmani gave it a name - loop engineering - and now there's a whole tidy progression people recite: prompt engineering, then context engineering, then harness engineering, then loops.

The vocabulary is useful, so let's use it. The inner loop is what the agent already does on its own every turn - look at the state, reason, call a tool, look at the result, reason again. The harness gives you that one for free. The outer loop is the part you build: the thing that feeds the agent work, runs it without you sitting there typing, checks the result, and decides what's next.

Fine. Good. I run one. Mine is literally /loop /task whatever is next, and it turns over unattended until the backlog is empty.

But every loop needs a stop condition, and in practically every loop being built right now the stop condition is the tests pass. Yegge's Gas Town gates on a verifier. The field guides all end up at some variation of "your own code runs test.sh and decides whether to keep going." Anthropic ships a verification skill.

And in an agentic project, the model wrote the tests.

One of those loop-engineering guides says the quiet part out loud, in passing, and then just carries on talking about schedulers: "The model that wrote the code is far too generous grading its own homework." Everybody knows. Nobody stops.

So you can build an immaculate outer loop - worktrees, stop hooks, sub-agents, durable state, the works - and have it converge, tirelessly and unattended, on the model's opinion of itself. The loop isn't the problem. The loop is fine. The problem is what the loop asks when it wants to know if it's done.

Everyone's building the outer loop. Nobody's fixed the inner one.

What I Do Instead

Diagram of the development loop: a Jira To Do backlog feeds the Claude Code outer loop, which runs six stages - fetch ticket, architecture review, probe oracle, TDD red phase, TDD green phase, final code review - before moving the ticket to Ready to Release. An Oracle Validation inner loop runs between architecture review and probe oracle. A TDD/code review inner loop rejects back to architecture review. Ready to Release feeds CI/CD, which produces build artifacts and test results, which feed manual testing and human oversight, which dotted-lines back into the backlog. — Figure 1 - the whole thing. The purple region is the outer loop.

That's my whole setup. The purple region is the Claude Code outer loop - fetch a ticket, six stages, done, next. Inside it there are two inner loops, and one of them is why any of this works.

Everything outside the purple box is an injection. CI runs the suite in shuffle mode to shake out fragile tests, and I take DR DOS 9 out to real hardware and emulators and run commercial software on it, because third-party compatibility is the test that really counts. Neither is in the loop - they just file tickets into it, and so do I, on the dotted line. If I disappeared for a week the loop would keep turning.

Look at the stage order. Probe Oracle is stage 3. TDD Red is stage 4.

That's it. That's the entire thing. Before the agent is allowed to write down what the code should do, it has to go find out.

The Machine That Already Knows

One of the things I'm building is DR DOS 9.0, a 16-bit DOS reimplementation in x86 assembly. Which means every test I write is an assertion about how DOS behaves - and I've got a copy of DR DOS 6.0 sitting right there. If I want to know what DOS does, I can boot DOS and look.

So that's what the agent does.

The project keeps a DR DOS 6.0 image in QEMU, wired up to Claude as an MCP server. Two tools, that's the whole surface. One runs DOS commands on a freshly-booted 6.0 and hands back what printed to the console. The other takes a .COM file off my machine, uploads it, runs it, and hands back its output. Every call is a clean boot - one batch, one answer, no follow-up keystrokes.

So when the agent needs a fact it can't get from a document, it doesn't reach into its memory. It writes a small NASM program whose only job is to walk out onto 6.0, take one measurement, and print it. Build it and drop it where the harness picks it up:

nasm -f bin -o tools/drdos-oracle/HDDROOT/MYPROBE.COM tools/drdos-oracle/src/myprobe.asm

Anything in HDDROOT/ gets overlaid onto the boot image on every run, so the probe is on the disk next time DOS comes up. Then the agent calls the tool and reads what the machine has to say.

That's the Oracle Validation Inner Loop on the left of the diagram - the agent reasons, writes a probe, measures, reasons again, measures again, as many times as it takes. And then, and only then, stage 4.

The name for that machine, in computer science literature, is an oracle: a black box you can ask questions of but can't open. You hand it a question, it hands back an answer, and you don't get to see how it decided. The oracle problem - the program ran, but was that right? - is one of the genuinely hard problems in testing, and it's most of why testing is expensive.

But go back to the question I actually had. How do I know the test is right?

Now I know. The test says the MCB header is laid out like that because I went and looked at one. The assertion isn't the model's belief about DOS - it's a transcript. And that's the difference between a suite I can lean on and a suite that's just the model agreeing with itself in a nicer font.

Which turns into a rule, and it's the one rule I'd keep if you took everything else away: no behavioral claim gets into this project from a model's memory, regardless of whether it happens to be correct.

Regardless of whether it's correct. That part matters. A model that guesses the right structure layout has still failed, because from inside the system a lucky guess and a bad guess look identical. You can't audit it, you can't build on it, and the next guess will be wrong - and you'll have no way to tell which was which. Which puts you right back to not trusting the suite, which is where I started.

It's provenance, not accuracy. That's the standard.

And it's a standard TDD never needed. TDD's oracle is the developer's intent, sitting in the developer's head, and for a human that's usually good enough. Hand that same loop to a language model and "the developer's intent" quietly becomes "the model's priors," and you're grading your own homework again. The fix isn't a better test-writing prompt. It's that the content of the assertion has to come from somewhere other than the model.

What A Probe Actually Looks Like

A probe is small, boring, and single-purpose. Here's a real one - the agent needed to know how DR DOS 6.0 handles renaming a directory, so it wrote renprobe.asm, which calls INT 21h AH=56h directly and prints the carry flag and error code for each case. What came back:

DIR-rename-same-parent   56=.5600  chk=.0000 a=10   <- succeeds
DIR-move-new-parent      56=!0005  chk=!0012 a=DD   <- refused, error 5
DIR-move-to-ROOT         56=!0005  chk=!0012 a=DD   <- refused, error 5
FILE-rename-same-parent  56=.5600  chk=.0000 a=20   <- control
FILE-move-new-parent     56=.5600  chk=.0000 a=20   <- control
FILE-move-to-ROOT        56=.5600  chk=.0000 a=20   <- control
rename-nonexistent       56=!0002                   <- control

So: 6.0 renames a directory happily, as long as it stays in the same parent. What it refuses is moving one to a different parent - error 5, access denied.

Two things worth pointing out, because they're the difference between a probe and a guess.

The controls are doing real work. Those three FILE lines aren't padding. They prove the destination directories - including the root - were valid, writable targets, which means the error 5 is specifically about the source being a directory and not about the destination being unreachable. Without them you've measured something, but you don't know what. A probe with no controls is just a more expensive hunch.

Probe the layer you're implementing. The obvious first move here is to type REN C:\RNOLD RNNEW at the DOS prompt, and if you do, you get "File not found" - which reads like 6.0 can't rename directories at all. That's wrong. REN is COMMAND.COM being restrictive on top of a kernel that could do it all along. The shell was hiding the kernel. If you're implementing INT 21h, probe INT 21h.

And then that measurement becomes the assertion. Stage 4, TDD Red: a test that says a cross-parent directory move returns error 5, source intact, destination absent. Not because the model thinks so. Because the machine said so.

The Test Suite Is A Cache, And It Gets Re-Checked

Just validating against the oracle once isn't enough.

Once an oracle answer is written into a test, that test is a cached measurement - and a cache nobody revalidates is just one refactor away from becoming a liability for all the reasons discussed above. So ./test.py --oracle re-runs the cached suite against DR DOS 6.0, and it runs as part of the cycle, not as something I remember to do on a Sunday.

Which means every assertion has to hold two ways, on every ticket, forever:

our kernel passes it, and
6.0 still agrees with it.

A test that drifts away from the machine is a failing test. Not stale, not a note in a backlog - a failure, same as any other. The cache can't rot, because rot is red.

And the agent doesn't get to make red go away. If a test passes on us and fails on 6.0, that is the single most interesting thing the suite can tell you, and it is exactly the moment a tired agent would love to quietly drop it. So the harness won't let it:

for banned in ("skip", "exclude", "disable", "ignore", "omit", "except"):
    if cfg.get(banned):
        raise TestHarnessError(
            f"oracle.json '{banned}' is not allowed. The oracle must run EVERY "
            f"test in scenario '{src_name}'; excluding tests is forbidden. "
            f"Record hangs/crashes as results and fix them, do not hide them.")

A rule in a CLAUDE.md is a suggestion to a system that is, in a real sense, optimizing against you. A raise is a fact. Wherever the invariant actually matters, put it in the code.

That's the property I was missing back when the agent was writing my tests unsupervised. The suite wasn't so much wrong as unanchored - it asserted things, and nothing on earth was checking whether those things were still true.

You Probably Have One Already

None of this is about DOS, or even legacy systems.

The DOS project is just an unusually clean example, because the thing that knows the right answer is a disk image I can boot on demand. But the shape is general, and the shape is the point: your loop needs something to ask that is held to a higher standard than the model is.

That's the whole premise. Everything else is implementation details.

An oracle has to be four things:

Fast to call. If it doesn't fit inside a stage, it can't be in the loop.
Cheap to call. Same reason. Cost is what decides how often the agent can afford to ask.
Accurate, and consistent with itself. Ask it twice, get the same answer. An oracle that wavers is worse than none, because now you're debugging the oracle.
Not an LLM.

That last one is the one people argue with, so let me say why. Reach for a judge model with a nice rubric and you've quietly put the generator's own priors back in the loop - same training data, same habits, same blind spots. Push on it and you don't converge on the truth, you converge on the judge's failure modes. It'll feel like verification. It'll produce a number. It just won't be checking anything the model didn't already believe.

So where do you find one? Odds are you're already sitting on it:

A conformance suite somebody else wrote. POSIX. test262. The Khronos CTS. W3C. Protocol conformance kits. Tax authority test cases. Entire industries ship with a ready-made oracle and treat it as a compliance chore to be endured once a year, rather than the thing that could be grading every commit.

Your production history. If you're building a billing engine, last quarter's actual invoices are an oracle. Real inputs, real outputs, generated by a system under real conditions long before any model touched the code. Same for a pricing engine, a tax calculator, a claims processor, a report. You already know what the right answer was, because you shipped it.

The system you're replacing. Every rewrite, every migration, every strangler-fig has an oracle running in production right now, and hardly anybody uses it as one. Replay real traffic through both. Diff the answers. That's the entire technique, and it's the same one I'm using - my DR DOS 6.0 image is a legacy system that happens to be thirty-odd years old.

There are others. I'd argue that every industry, every domain, and every project could construct an oracle. Some might be trivial, while others difficult, but none should be impossible. And, the effort put into constructing your oracle will yield ROI quickly.

I run this same setup on a commercial compiler - it targets the JVM, and it's verified against real hardware running the vendor's own reference compiler. Different language, different century, same loop. That's what convinced me the pattern wasn't a quirk of the DOS project.

Wrap Up

Here's the whole thing as one question. You can ask it about any project, before you write a line of code:

Can I write a program that, given a candidate and an input, decides correct or incorrect - without using the model that produced the candidate?

If yes, an unattended loop is a highly effecient process for scaling your efforts. If no, you're doing supervised craft, and no amount of loop engineering changes that. Because what you're missing isn't a scheduler. It's autonomous determinism.

I set out to get out of writing tests, and ended up somewhere I didn't expect. The agent still writes them. I still don't have to. But every assertion starts as something a machine said out loud, and neither of us is allowed to throw one away afterward.

Which means I trust the suite. Not because the agent is careful, and not because I reviewed every line - I didn't - but because the suite isn't the agent's opinion in the first place. It's a pile of measurements, and something other than the model checks them, over and over, forever.

If you can't say that about your suite, then whatever your outer loop is converging on, it isn't correctness.

My spec doesn't tell the agent what the right answer is. It tells it where to go ask.

If you've built one of these - especially if you had to manufacture your oracle rather than inherit one - I'd like to hear about it. As always, thoughts, feedback, and criticisms are welcome.