How to Run a Cheap Model Inside a Coding Agent Loop (and When Not To)

If you run a coding agent continuously, the bill arrives before the results do. The usual advice is to switch to a smaller model and hope quality holds. That is the wrong knob.

Here is the one that works, stated as a rule you can apply to a task in about five seconds:

Does correctness live outside the model?

If a fixed list of failing tests already defines “done”, the work is mechanical — hand it to the cheap model. If deciding what “done” means is part of the job, you are buying judgment, and cheap tokens will buy you confident wrong answers instead.

This post is the recipe for the first half, the measured cost of running it over seven libraries, and the two places it failed badly enough that I stopped giving it that class of work.

The setup

Two lanes run on cron against the same estate of sites.

One drives a frontier model. It reviews the portfolio, decides priorities, plans tasks, executes them, verifies, deploys. It decides what to work on.

The second is deliberately the opposite shape. No planner and no strategist. It works an ordered queue of jobs a human wrote, one step per tick. For each job it receives a specification and a set of acceptance vectors authored up front by the expensive model, and it may not touch them. Its only move is: run the vectors, read the failures, rewrite the implementation, repeat.

The recipe

1. Write the specification and the vectors with the expensive model, before any implementation exists.

This is where the real work is, and it is worth being blunt: specifying one library took me far longer than the $0.0007 it cost to satisfy. The bottleneck was never the implementation. If you take one thing from this post, take that the expensive model’s time is best spent authoring the answer key, not the code.

Write the spec as numbered rules, where every value a test pins is derivable from a rule. Then write the vectors as data, separately from the harness that runs them.

2. Lock the vectors so the implementer cannot edit them.

Hash the vector file when the job is registered and refuse to continue if it changes:

const vecNow = sha(fs.readFileSync(vecPath, 'utf8'));
if (vecNow !== cfg.vectorsSha) {
  console.log('ABORT: vectors.json changed since --init');
  process.exit(2);
}

This is not paranoia about a specific model. If whatever writes the code can also edit the tests, it will eventually make them agree while both are wrong, and it will do so silently. The lock is the entire safety property; everything else is convenience.

3. Make the harness report failures as data, and never throw.

The harness prints one line of JSON — {"total":N,"passed":N,"failures":[...]} — and reports a missing or crashing module as every vector failing rather than as an exception. That matters because the failure list is the entire instruction the cheap model receives on the next iteration. Each failure carries what was got and what was wanted:

resolve: ./utils/parse — .target = null, want "./src/utils/parse.js"

A failing vector phrased like that is a precise, mechanical instruction. That is the thing cheap models are good at.

4. Gate the output in code, and make rejection free.

Before an attempt is accepted, check it mechanically: no external require, no clock reads, no network, no randomness — and above all a regression gate that refuses any attempt passing fewer vectors than the current best. At a tenth of a cent per attempt you can afford to be brutal. That is the real unlock. With an expensive model you are tempted to salvage bad output; here you throw it away and move on.

5. Let it iterate, and cap it.

Give it an iteration ceiling so a task with a contradictory spec fails cheaply instead of forever. If it stalls twice with no gain, escalate that single job to a thinking mode — the cost is 3× of almost nothing.

What it cost

Seven libraries, all pure dependency-free parsers with a published standard behind them:

419 vectors, $0.0733 all in.

pkg-exports simulates Node’s package.json exports/imports resolution — conditions, nesting, wildcard subpaths, null targets, array fallbacks. Fifty-one vectors, passed on the first attempt, for seven hundredths of a cent.

content-disposition (RFC 6266 filename encoding) scored 80 of 82 cold, with no implementation to start from. The two misses were a mishandled empty parameter segment and a missing null guard, both fixed on the next iteration.

That cold score is also how you check your own spec. A contradictory specification cannot score 80/82 on the first try, so the number grades the author as much as the model.

For contrast, on the same estate in the same week, the frontier lane spent 220,346 tokens across two runs and verified zero tasks, then correctly paused itself. Its working rate when it is producing is roughly 155,000 tokens per shipped task — but that task includes reading a live site, judging what is worth doing, and deploying. Those are not the same unit of work, and that is the point: judgment is what you are paying for, and most tasks in a loop do not need any.

Where the cheap model failed

Prose, at roughly one fabrication in seven documentation tasks. Asked to write a README, it invented four functions with confident, plausible signatures. None existed. The audit meant to catch this passed them, because it only checked that an export was mentioned in the README, never that the claim was true. Every README in this estate is now written by hand.

Notice this is the same result as the code half, inverted. Nothing external defined what a good README says, so there was nothing to iterate against — and with no answer key you get something that looks right.

Output encoding decided success outright. Asking for file contents nested inside JSON failed 3 of 3 attempts on unparseable escaping. Switching to plain delimiters:

<<<FILE index.js
...contents...
>>>END

succeeded 5 of 5. Same model, same task, same day. If you are extracting generated files, do not make the model escape one language inside another.

The failure nobody warns you about: the queue can lie

This one is not about cheap models, and it cost more than everything above.

The frontier lane stopped planning work. Three runs in a row ended with endReason: frontier — “nothing left to do” — and a watchdog paused it. Three separate reviews concluded the backlog was exhausted.

It was not exhausted. It was invisible. A config file gave one property a cadenceDays: 6 throttle to pace its content generation, and the engine implements that by deleting the property from the registry the planner receives. That property’s last commit was 4.6 days old — and both remaining open tasks belonged to it. So 100% of the board was unreachable, and every run correctly reported an empty frontier for a board it could not see.

There is a second-order trap worth naming, because it only appears in unattended systems: the throttle would have expired by itself the next day, but the watchdog had already paused the lane — and a paused lane never reaches its own self-heal. Two individually correct safety mechanisms composed into a deadlock.

The rule I would put on a wall: “nothing to do” is a claim about your scheduler until you have proved the queue was visible. Cadence throttles, lane filters and dirty-tree skips all delete work silently, and deleted work looks exactly like finished work from the inside.

Honest limits

None of this has made money. The estate’s revenue is €0. Seven cents of libraries is not a business, and I am not claiming a saving on work that has not yet earned anything.

Seven libraries is a small sample, all in one genre: pure, dependency-free parsers with an RFC behind them. That is the most favourable possible case for the argument, because the specification already exists and is unambiguous. I do not know where this stops working; I have not found the edge yet.

The download numbers on the published packages are not evidence of adoption. They sit between 106 and 152 a week across every package regardless of purpose or publish date, which is the shape of registry mirrors and CI caches rather than users. I would rather say that than quote it as traction.

The takeaway

If your agent bill feels disproportionate to the output, do not start by changing models. Start by sorting the work: how much of this has an answer key, and am I paying frontier prices to do it anyway?

Put the expensive model on the specification and the tests — carefully, because that artifact is now the only thing between you and a cheap model that will happily make the tests agree with its bugs. Let the cheap model iterate until green. Throw away anything that fails a gate, which you can afford to do because a rejected attempt costs a tenth of a cent.

And keep the judgment — what to build, whether it is good, whether the claim is true — on the model you are willing to pay for. That part has not got cheaper.

← Back to all posts

Follow new posts by RSS: paste dankdev.com/feed.xml into any feed reader. It is a plain XML file — no signup, no email address, no account. Posts land there the same day they go up, and there is nothing to unsubscribe from later.