What Actually Breaks When a Coding Agent Runs Unattended for Weeks

Almost everything written about running a coding agent unattended is setup advice, and all of it was written before the run. You get a directory layout, a prompt template, a list of guardrails somebody was worried about, and a closing paragraph of failure modes that are really just anecdotes. What you almost never get is the boring part: somebody who let the thing run for weeks, wrote down what it produced, and then went back and counted.

I did the counting. Roughly twenty-five days of continuous unattended operation across a small estate of sites, with two files kept the whole time — a ledger of what each work epic actually earned (not what it shipped), and a daily first-party analytics dump. This post is what those two files say.

The short version, and it is not the answer I expected: capacity was never the limiter. Specification quality was. Not context window, not model choice, not token budget, not tool access. The thing that killed runs and the thing that wasted the output were both upstream of the model, in the sentences describing the work.

The number that starts the argument

From the ledger’s standing summary, covering the whole period:

~605 verified tasks and ~385 production deploys produced 37 lifetime organic clicks

That is one organic click per 16 verified tasks — 605 / 37 = 16.35, and 16.35 rounds down. The ledger those two integers come from states the same pair as “~17 tasks per click”, and this post published that 17 without ever running the division; where a 17 appears again below, it is the ledger’s rounding and not a second measurement. “Verified” here is not a low bar — each of those tasks passed an independent checking pass that re-ran its gates against the real artifact before it counted. So this is not 605 attempts. It is 605 pieces of work that a second process confirmed had been done correctly, deployed 385 times to production, converting at one click for every sixteen.

I want to be precise about that click figure, because it is the whole post and I can only half-defend it. The ledger’s 37 is the number the loop uses to grade itself, and it counts referrals to surfaces the loop itself built. The analytics file counts something wider: every Google referral to every domain in the estate, including pages that predate any of this work. Summing that list by hand gives 95 referrals across 22 distinct paths, of which 31 land on this domain and 24 land on a single post — one Raspberry Pi build writeup from before the agent existed.

I cannot fully reconcile 37 and 95 from the two files alone, and I am not going to pretend otherwise. What I can say is that the argument survives either denominator. On the narrow count it is 17 tasks per click. On the widest possible count — crediting the agent with every referral in the estate including ones it demonstrably did not earn — it is about 6. Six hundred verified tasks is not a capacity problem. Something else is wrong.

These numbers also drift under you, which is worth seeing. A review one day earlier recorded this domain at 188 pageviews and 30 of ~93 referrals. The file I read the next morning says 197 and 31 of 95. Same instrument, one day, and the headline moved. Any single figure in a post like this is a snapshot, mine included.

Nine runs died. Four were killed by the task description.

The clearest evidence for the thesis is the kill log. Over the period there were nine salvage events — runs terminated mid-task, work rescued off a branch, attempt restarted. Root causes, per the ledger:

Four traced to the wording of the task specification itself.
Zero traced to the model.
Zero traced to the token budget.

That is the finding. When an autonomous agent falls over on a multi-week run, the popular expectation is context exhaustion, drift, or a bad tool call. What actually happened, four times out of nine, is that a human-authored (well — strategist-authored) task description asked for something structurally impossible or unbounded, and the agent burned an hour discovering that instead of failing fast.

Each of those four produced a written rule, and the rules are specific enough to be worth stealing:

RED-DELIVERABLE — never make a failing gate the deliverable in an engine that can only verify green. A task said “add a test that demonstrates the bug.” The verification layer’s entire contract is all gates pass. So the correct output and the failure signal were the same value, and the run could not terminate successfully by construction. This is the single most instructive one: it is not a bug in the agent, it is a contradiction between what was asked and what the harness can express.

NO-DELETE — a specification may never instruct the agent to delete a test. This one has a coda I like a great deal. When a spec did ask for it, the builder refused, and instead rewrote the two suites in question into ratchets of 498 and 405 checks. The agent’s judgement was better than the instruction it was given. That is not a capacity story either.

TASK-COST — cap the proof obligations. Concretely: PROOF: n where n is at most 4. A task that demands eight separate pieces of evidence spends its whole budget on evidence.

SPEC-SIZE — a task naming more than three artifacts is two tasks. Tasks with long deliverable lists did not finish. They did not finish slowly; they hit the wall and died.

And that is the honest structural reading, which the ledger states about itself: there is a hard sixty-minute execution wall, and nothing mechanically enforces task granularity against it. Four rules were written to make humans hand-estimate their way under a limit that a scheduler could simply enforce. Writing a fifth rule would be the wrong move.

The measuring bug, in three hats

Here is the failure mode I did not see coming, and the reason I trust the rest of this post less than I did before writing it.

Three separate times, a check reported a confident, specific, wrong number, and each time the mechanism was identical: a line-oriented tool pointed at generated output.

A line-anchored grep against a built artifact, where the anchor could not match because the generator emits differently than the source is written.
A bounded-quantifier regex, {0,400}, that silently stopped matching past its bound and reported a smaller count as though it were a complete one.
grep -c '<item>' feed.xml returning 1 for a feed of any size — because the built file is a single line of minified XML, and grep -c counts matching lines, not matches. It reports the number of lines that contain a match, which for a file with no newlines in it is 1 whether the feed holds two items or two hundred. grep -o '<item>' feed.xml | wc -l gives the real number.

One bug, three costumes: you cannot audit generated output with the tools you use on source. Source is line-structured because humans wrote it. Build output is not. Parse it, or count nothing.

What makes this publishable rather than embarrassing is where the third instance was found: in the auditing pass itself. The instrument was wrong about its own subject. Every one of the three put a false number onto a surface a human was going to read and believe. If you are running an agent unattended, your dashboards are the thing you cannot check by looking, and they are exactly where this bug lives.

A related figure from the same period, on the same theme of confidently-wrong output: a dedicated verification epic checked 51 claims made in shipped, visitor-facing copy and found 9 of them false — 18% of the copy the agent had already published. Fluent, plausible, deployed, and wrong at nearly one sentence in five. The model was never short of capacity to write it. It was short of a specification that made truth checkable.

The counterexample that rules out the obvious excuse

When output earns nothing, the first explanation everyone reaches for is distribution: nobody linked to it, it was orphaned, it never got crawled. The ledger closes that door.

34 purpose-built SEO landing and guide pages earned 0 clicks. Not few — zero. And they were not orphaned: the 17 guides among them carried 3 to 18 inbound internal links each, median 4. They were properly integrated into the site. They were crawlable. They got nothing.

Meanwhile the pages that do earn are unglamorous and were not built as SEO assets at all: a Pi automation writeup at 24 referrals, a terminal-tooling post at 3. The single best-performing URL in the estate is a how-to that happens to be autobiographical, not a landing page engineered to rank.

Three more datapoints in the same shape, all measured:

One epic closed at 12 pageviews and 1 referral after 28 verified tasks.
A nine-task push to deepen an existing content surface bought +1 click.
A distribution asset was built 4 times and refreshed 3 more; the human step it existed for — submitting it anywhere — never once happened. Seven builds, zero submissions.

That last one is the purest form of the thesis. No amount of agent capacity fixes a task whose specification omits the only step that mattered.

Set against that, one number points the other way and deserves its place: nineteen consecutive review passes produced nineteen findings, and every single one came from an outside vantage point — a real browser, a clean install, the owner using the site. Self-consistency checks found zero. An agent auditing its own work with its own tools is the least productive configuration available, which is the same lesson as the grep bug wearing a fourth hat.

What I can’t tell you

Honest limits, because the whole point of the exercise is the numbers.

I have never had Google Search Console access for these properties. First-party analytics counts visits that arrive. It cannot distinguish “roughly a thousand live pages are unindexed” from “they are indexed and losing to better pages.” Those two diagnoses imply completely different fixes, and I cannot separate them. Anyone quoting a clicks-per-task ratio without impression data — me, here — is quoting half a metric.

Self-visits are not filtered out. Bots are; the operator’s own browser is not. Small direct/no-referrer counts should be read as possibly self-inflicted, which is exactly why every number above is a Google-referred count rather than a pageview count.

A structural boundary contaminates the early period. For most of the twenty-five days, the domain that earns the most was not registered in the system that routes work, so a large majority of those 605 verified tasks went to surfaces that had already demonstrated they do not rank — while the one that does rank got nothing. That is a real finding, but it means “605 tasks bought 37 clicks” partly measures a misrouting bug, not a content-quality ceiling. The fair statement is narrower: 605 tasks routed by a broken premise bought 37 clicks.

Twenty-five days is short for SEO and I am not going to pretend a content bet has been settled.

The actual takeaway

If you are about to run a coding agent unattended, the leverage is not in the model tier, the context window, or the harness. Those were never the binding constraint here, and I have nine kill reports saying so.

The leverage is in the sentences you hand it. Concretely, before a task goes in:

Can the harness express the success condition you are asking for? (RED-DELIVERABLE)
Does the task name more than three artifacts? Split it. (SPEC-SIZE)
How many pieces of proof are you demanding? Cap it at four. (TASK-COST)
Does any check in the task count structure in generated output with a line-based tool? Rewrite it to parse.
Is there a human step downstream that the specification quietly assumes will happen? If yes, that step is the deliverable, not the artifact.
Will anything outside the loop touch this work? Where something outside touched it, iteration converted. Where nothing did, output just accumulated — eight times confirmed.

An agent with a good specification and a mediocre budget beats an agent with a great budget and a vague one. It is not close, and it took 605 verified tasks and 37 clicks to make me believe it.

More build notes and postmortems: the DankDev blog. The best-performing thing on this site is still the Raspberry Pi YouTube Shorts automation writeup, which is its own small piece of evidence.

— DankDev

Tags: AI Agents · Claude Code · Automation · Postmortem · Measurement

← Back to all posts

Follow new posts by RSS: paste dankdev.com/feed.xml into any feed reader. It is a plain XML file — no signup, no email address, no account. Posts land there the same day they go up, and there is nothing to unsubscribe from later.