Making My AI Argue With Itself: Building a Self-Review Loop

Local AI models lie with confidence. Ask a 9B-parameter model for a bash

one-liner and you'll get something that *looks* right — and sorts your files

alphabetically when you asked for largest-first. I got tired of being the QA

department for my own homelab assistant, so I built a system that forces the

AI to check its own work: one model drafts, a second model plays hostile

reviewer, and the draft gets revised until it passes or provably plateaus.

Simple idea. Two days of debugging. (If you read the war story about my assistant going silent, consider this the sequel — same stack, new lessons.)

The design

The orchestrator is a small Python script — no framework, no LangChain, about

250 lines. It talks to local models over an OpenAI-compatible API:

1. Generator model drafts the deliverable

2. Judge model critiques it against explicit pass criteria, answering in

   strict JSON: pass/fail plus a list of concrete defects

3. Fail → the generator revises against those specific issues

4. Repeat until pass, a hard iteration cap, or convergence — if two drafts

   come back nearly identical without passing, the model has plateaued and

   the loop exits honestly instead of burning compute

That last part matters. Small local models hit their ceiling fast, and a

quality loop that can't admit defeat is just an expensive way to get the

same wrong answer three times.

What actually went wrong (everything)

The loop itself worked in an afternoon. Wiring it into my AI assistant stack

so I could trigger it from my phone is where the two days went. A partial

list of what broke, in order of discovery:

The judge wouldn't speak JSON. My first judge model answered in

Python-dict syntax half the time — single quotes, capital-T True. Prompting

harder didn't fix it. A parser that accepts both dialects did.

Thinking models produce empty answers. Newer model families "think"

before responding. Under a token budget, they'll sometimes burn the entire

budget reasoning about your one-liner and emit nothing. Every documented

off-switch failed intermittently. The fix: older-generation instruct models

for loop duty. They can't overthink because they can't think.

Small models ignore instructions about tools. The assistant had explicit,

impossible-to-misread rules: when the user says X, run this script. My

9B-class models found every loophole — answering from memory, running the

task's command themselves, saving the answer to a file — anything but the

one instructed action. Stepping up to 14B fixed dispatch almost overnight.

Instruction-following at the routing layer is a capability cliff, not a

gradient.

Two inference engines on one box is a murder-suicide. The subtlest one.

The agent's model and the loop's model ran on separate engines on the same

24GB Mac mini. Each worked alone. But the crash always came at the same

moment: the agent resuming *after* the loop finished — both engines maximally

resident, and macOS's memory killer took out the biggest process. The fix

was architectural: the loop's model now runs on a different machine entirely

(a Steam Deck, because a homelab that doesn't repurpose hardware isn't

trying). The mini runs exactly one engine, one model. Zero crashes since.

The punchline

Here's the part that justified the whole build: while debugging, I asked

various models for that disk-usage one-liner maybe a dozen times *without*

the loop. Every single answer contained a bug. Sorting by name while

claiming to sort by size. Human-readable flags missing. Off-by-one on the

process count. Confident explanations attached to every one.

The loop's judge — a 7B model with a hostile system prompt — caught most of

them. Not all; same-model self-review has blind spots, and a dedicated judge

model is next on the list. But "most" beats the zero percent I was catching

by trusting the first answer.

If you're running local models for anything that matters: don't trust the

first draft. Make the machine argue with itself. It's cheaper than being

the argument yourself.

Donnie P.