Your Work Queue Can Lie to You: Three Ways a Scheduler Hid a Whole Backlog

If you run anything that picks its own work — a cron-driven agent, a job runner, a planner that reads a board and dispatches — you will eventually get this output:

no eligible work; nothing to do

The question you have to ask, and that I did not ask for weeks, is: is that a fact about my backlog, or a fact about my scheduler?

On one system I operate, it was a fact about the scheduler three separate times, by three unrelated mechanisms. Each one was invisible in exactly the same way: the component that reported “empty” was telling the truth about the list it received. Something upstream had already removed the work, and nothing logged the removal.

The distinction that matters

There are two different states, and most systems print the same message for both:

• The queue is empty. There is genuinely no work. Correct behaviour: idle.
• The queue arrived empty. There is work, and it was removed in transit. Correct behaviour: scream.

A scheduler that cannot tell these apart cannot report its own worst failure mode. Every incident below is a case of the second state being reported as the first.

The three ways it hid

1. A throttle that deleted the row instead of skipping it

A per-property cadence rule was supposed to mean “don’t start new work on this repo yet.” It was implemented by filtering the registry before handing it to the planner. The planner therefore never saw that property at all — not as throttled, not as anything.

So the planner did what it should do given its input: it looked at the board it was given, found nothing actionable, and concluded the backlog was exhausted. It concluded that on three consecutive reviews, while two tasks sat on that property’s board the entire time.

The bug is a category error that is very easy to write. “Not now” and “does not exist” are different states, and a filter that deletes the row collapses them. If the throttle had passed the property through with a throttled_until field, the planner would have printed “1 property throttled, 2 tasks waiting” — an obviously different message from “nothing to do.”

2. A drop filter plus a cap equals a deadlock

The second one is nastier because no single component was wrong.

A safety regex scanned each planned task’s check commands and dropped any task whose commands named a deployment path. Reasonable rule: an unattended agent should not be able to write to a live webroot as part of “verifying” its work. The filter dropped the task silently — no log line, no counter, no exit code.

On its own that costs you one task. Composed with two other perfectly sensible rules it cost everything:

• a one-task-per-repo cap, so at most one task from each repo was in flight;
• a homogeneous board, where the eligible tasks were near-identical in shape, so the planner picked the same one every time.

Each run: pick the top task, get it silently dropped, hit the cap, emit nothing, exit clean. Deterministic input, deterministic filter, deterministic re-pick. The system was not slow, it was in a permanent deadlock — and its own report of that state was “no eligible work.”

The bill: four runs, roughly 437,000 tokens, three tasks visibly on the board, zero builds.

3. The same filter, tripped by a prohibition

This is the one I’d put on a slide.

Weeks later the same regex dropped the top task again. This time the blocked word did not appear in a path the task would write to. It appeared inside a filename — a filename the task’s own instructions named in order to say do not modify this file.

The planner, being obedient, wrote a check command that referenced the file it had been told to leave alone, so that the check could prove it had been left alone. The check command therefore contained the forbidden token. The filter dropped the task.

The prohibition manufactured the very command that tripped the filter. Two runs, roughly 380,000 tokens, thirteen visible tasks, zero builds.

I want to be precise about why this is worth writing down. A regex over a path-like string is a string control being used as a behaviour control. The behaviour it means to forbid is “write to the live webroot.” The string it actually matches is a substring, anywhere, in any context — a filename, a comment, a quoted example, the word inside a longer word. Every string control has this gap. It only bites you when something upstream starts reliably generating the string for an innocent reason, and a rule that says “never touch X” is a very effective generator of the token X.

The fix that shipped, and why it wasn’t enough

After incident 2, I pinned each task’s check commands verbatim in the task record, so no downstream agent could improvise a new one. That is a good fix. It removes the whole class of “an agent wrote a command that tripped a filter.” It shipped and worked, on three consecutive runs.

Then incident 3 happened anyway, and it is the reason this post exists.

The other half of the fix was a rule for whoever writes the task: don’t put the dangerous token in a check command. That rule was real, it was written down, and it was correct. It lived in a strategist digest — a document the planner never reads.

That is the whole diagnosis, and it generalises further than anything else here:

A control has an audience. If the agent that must obey it does not read where you wrote it, you have not written a control. You have written a wish.

This is not an LLM-specific failure. It is the same failure as a security policy in a wiki nobody opens, or a lint rule documented in the README instead of the linter. The difference with agents is only that the audience is enumerable: each agent has a literal, inspectable context window, and you can check whether your rule is in it. I had never checked. The answer was no.

The measured outcome

Write-ups of this kind usually stop at the diagnosis. Here is what the fix actually did.

The rule was moved out of the strategist digest and into the last clause of every task line — the text the planner demonstrably reads, because it reads nothing else about the task. Nothing else changed: same filter, same cap, same board.

The queue drained on the very next run. It has now drained on five consecutive runs — two tasks planned, two verified and two deployed on each — immediately after two consecutive runs that planned zero.

Five is why I’m publishing this. One run would have been an anecdote, and this system has produced a green run by accident before. Five consecutive runs with the same shape is a trend, and it is the only evidence I have that the diagnosis was right rather than merely satisfying.

Honest limits: these are five runs on one system, and I moved one thing, so the causal claim is as strong as a single-variable change with n=5 ever is — suggestive, not proof. It also does not fix the underlying string control. The regex is still a substring match, and a task author who writes the token into a check command by hand will still be dropped, silently. What changed is that the rule now reaches the agent that generates those commands.

Two rules I’d apply anywhere

1. “Nothing to do” is a claim about your scheduler until you have proved the queue reached a worker. Instrument the boundary, not the endpoints. Count items in to the planner and items out, and log the delta with a reason. Every incident above would have been a one-line diagnosis if any component had printed received 3 candidates, dropped 3 (filter: deploy-path), emitted 0. All three printed emitted 0.

2. A drop filter needs a safety valve, or it becomes a deadlock the system cannot report as one. If a filter can remove the last remaining item, it must be able to say so. The cheapest version is a rule: if the filter drops every candidate, that is an error, not an empty queue. A filter that silently returns zero on a non-empty input has turned your scheduler into something that fails closed and reports success — the worst combination available.

Neither rule requires new infrastructure. Both are counters and a log line, and between them they would have saved something like 800,000 tokens and six runs of doing absolutely nothing, very reliably.

← Back to all posts

Follow new posts by RSS: paste dankdev.com/feed.xml into any feed reader. It is a plain XML file — no signup, no email address, no account. Posts land there the same day they go up, and there is nothing to unsubscribe from later.