The rule that didn't hold
My long Claude Code sessions kept running past the handoff rule I pasted into them, so I built Stu. My cost per PR then fell by three quarters. Then I measured how much of that was Stu.
Jon Yoder · · 9 min read
I build Keavi mostly with Claude Code, in long sessions that keep working while I’m away. All summer those sessions had two problems. They needed me. Every campaign started with long instructions I pasted in by hand, and without them the work drifted off track. And they ran long. Nearly all of my spend is cache reads, and every turn re-reads the whole context, so a session costs roughly context size times number of turns.
Few sessions grew past 300K tokens, but turns past that line were two fifths of the bill
Keavi, July 1 to August 21, before Stu. Orange is anything past 300K tokens of context, about where my August rule later said to hand off.
Share of sessions
1,293 main sessions, by the largest context each reached
Share of spend
All Claude spend, by the context size of each turn
Show the numbers
| Under 300K | Over 300K | |
|---|---|---|
| Main sessions | 1,183 | 110 (16 past 800K) |
| Share of spend | 59% | 41% |
What I used to paste
The instructions never lived in a config file. In July I started each campaign with a hand-written manual. The longest ran to 6,221 characters and covered scope, which PRs to merge and which to leave for me, a workflow for every subagent, the review cycle, and the rules for pushing and merging. By August I had cut it to one paragraph, with a new rule for context, and was pasting it into dozens of sessions:
Please work autonomously, because I will be leaving, and you have permission to make decisions on my behalf. [...] Prefer a subagent-driven approach, and please use Sonnet or Haiku subagents whenever appropriate to keep costs down. Be very token aware, and keep this session's context lean. When you reach 25 to 30% context in this session, please hand off and pause so I can move this work to a new session. But work without me and merge PRs as you are able.
The agents usually tried. In most of those sessions they started writing a handoff somewhere past 200K tokens. But the rule didn’t hold: 30 of the 98 sessions went past 300K anyway. Those 30 cost about $1,000, half of what all 98 cost.
On August 22 I pasted it again and added: “I’ve found that the explicit prompts really help keep this on track, but I’d love to make it more automatic.” Nine days later the first version of Stu went in.
Stu is a Claude Code plugin for long unattended runs. It turns your main session into a supervisor that hands all the work to short-lived subagents and checks what they report, so its own context stays small. The rules I used to paste became checks that run whether or not I remember them.
The bill fell
By October my cost per PR had fallen by three quarters.
Cost per merged PR
$34.11 → $8.50
API list prices, not an invoice
Context per turn
197K → 87K
tokens re-read or written, average per model turn
Turns above 800K context
1,559 → 0
the turns that cost the most
Keavi's own work: summer (Jul 1 to Aug 21) before Stu, versus October.
Runaway sessions, which had taken two fifths of summer’s spend, had almost disappeared, and Stu runs were the cleanest of all.
By October, almost no money went to runaway sessions
Keavi. For each group, the share of its own Claude spend on turns with more than 300K tokens of context. Automated code review is left out. Lower is better.
Show the numbers
| Month | Sessions | Spend past 300K | Main sessions past 300K |
|---|---|---|---|
| Summer | Other sessions | 42% | 9% of 1293 |
| September | Stu runs | 8% | 1% of 152 |
| September | Other sessions | 18% | 4% of 562 |
| October | Stu runs | 0.5% | 0.3% of 346 |
| October | Other sessions | 3% | 3% of 264 |
Automated code review sessions are left out, which is why summer reads 42% here and 41% in the first chart.
I barely had to type, either. A run started with one line, copied from the end of the previous run (keavi-run was Stu’s name before it became a plugin):
/keavi-run ~/.keavi/runs/<run>/handoffs/campaign-handoff-3.md overnightMy median opening prompt for this kind of work fell from about 1,400 characters in July to about 100 in October. I was ready to write a post about how Stu cut my bill by three quarters.
What caused the drop
First I checked when it happened.
My cost per merged PR fell when the 5.5 models arrived
Keavi, by week: all Claude Code and Codex spend at API list prices, divided by the PRs merged that week.
Show the numbers
| Week of | PRs | Spend | Per PR |
|---|---|---|---|
| Jul 6 | 227 | $4,876 | $21.48 |
| Jul 13 | 151 | $6,342 | $42.00 |
| Jul 20 | 97 | $3,196 | $32.95 |
| Jul 27 | 122 | $4,858 | $39.82 |
| Aug 3 | 107 | $5,548 | $51.85 |
| Aug 10 | 115 | $5,281 | $45.93 |
| Aug 17 | 148 | $4,310 | $29.12 |
| Aug 24 | 187 | $4,530 | $24.23 |
| Aug 31 | 183 | $6,425 | $35.11 |
| Sep 7 | 125 | $5,570 | $44.56 |
| Sep 14 | 85 | $3,447 | $40.56 |
| Sep 21 | 187 | $4,405 | $23.56 |
| Sep 28 | 236 | $4,518 | $19.14 |
| Oct 5 | 211 | $2,425 | $11.50 |
| Oct 12 | 251 | $2,160 | $8.60 |
| Oct 19 | 262 | $2,668 | $10.18 |
| Oct 26 | 239 | $2,055 | $8.60 |
The first Stu run was on September 2. The drop came three weeks later, with Opus 5.5 on September 22 and Sonnet 5.5 on September 28.
The clearest evidence came from limits I had written into the agents’ own instructions. From August 10, every implementer subagent was told to hand off after roughly 80 tool calls. From August 17, every orchestrator had a hard budget of 120 turns. On the older models about half the implementers ran past their budget, one of them to 901 tool calls, and one orchestrator ran for 2,054 turns. Then the new models arrived.
Older models often ignored their written budgets. By October, almost no agent did.
Keavi. Share of agent sessions that ran past the length limit in their own instructions, by the model each mostly ran on. Lower is better.
Implementers, told to hand off past about 80 tool calls
Orchestrators, given a hard budget of 120 turns
Show the numbers
| Agents | Model | Days | Runs | Past the budget | Longest run | Peak context over 300K |
|---|---|---|---|---|---|---|
| Implementers | Older models | Aug 10 to Sep 27 | 2,336 | 1,131 (48%) | 901 tool calls | 164 |
| Implementers | Opus 5.5 | Sep 22 to 27 | 23 | 7 (30%) | 123 tool calls | 0 |
| Implementers | Older Sonnet | Sep 28 to 30 | 174 | 78 (45%) | 143 tool calls | 0 |
| Implementers | Sonnet 5.5 | Sep 28 to 30 | 375 | 25 (7%) | 111 tool calls | 0 |
| Implementers | 5.5 models | October | 2,299 | 92 (4%) | 201 tool calls | 0 |
| Orchestrators | Older models | Aug 17 to Sep 30 | 356 | 62 (17%) | 2,054 turns | 18 |
| Orchestrators | 5.5 models | Sep 22 to 30 | 178 | 2 (1%) | 224 turns | 1 |
| Orchestrators | 5.5 models | October | 881 | 0 (0%) | 109 turns | 0 |
Exploratory: this comparison was not planned before the study. Running past the budget can mean ignoring it or just needing longer, and this chart can't tell those apart. An implementer is past its budget when its session made more than 80 tool calls; an orchestrator, when it took more than 120 Claude turns. Implementers had the 80-call rule from August 10 and orchestrators the 120-turn rule from August 17; agents dispatched before those dates are left out. The implementer rule said "roughly", so a run at 85 calls counts as over. From October 1 the orchestrator rule also had a context ceiling and the run supervisor checked every orchestrator's report, so October is not the same instructions alone. Nearly every October orchestrator ran inside a supervised run, so the cleanest comparison is September 28, the day Sonnet 5.5 arrived: 68 of 148 implementers on the older Sonnet inside runs went past the budget, and 1 of 35 on Sonnet 5.5.
The first week of Opus 5.5 was mixed, but since Sonnet 5.5 arrived, agents have mostly kept their budgets.
Cost per PR can also fall because the PRs got smaller, so I scored the work itself. I took 120 merged PRs from each period, plus a matched sample of October Stu and non-Stu PRs. Claude Sonnet scored each one from its diff and a cleaned-up title, without being told when or how it was made, on a scale of 0, 1, 2, 3, 5 or 8. One point is one unit of work. Codex scored every PR again as a check. I published the method before scoring any PR and logged every change I made to it during scoring, with the reason.
October cost about a quarter of what summer did per unit of work
Keavi. Each merged PR is scored for value from 0 to 8; one unit of work is one point. Dots are all AI spend divided by the units merged; lines show the 95% range.
Show the numbers
| Period | Value per PR | Cost per value unit | Index (summer = 100) | Tokens per value unit, index | Second scorer, cost index |
|---|---|---|---|---|---|
| Summer | 3.66 | $9.50 ($8.88 to $10.25) | 100 | 100 | 100 |
| September | 3.09 | $9.21 ($8.55 to $9.97) | 97(87 to 108) | 112(101 to 125) | 93 |
| October | 3.60 | $2.40 ($2.26 to $2.55) | 25(23 to 28) | 30(27 to 32) | 25 |
September also used about 12 percent more tokens per unit of work than summer.
Separating Stu from the models is hard on Keavi, because they arrived together. Stu started as an in-repo skill on September 1, was rewritten on September 22, and began enforcing its rules on the evening of October 1. A second codebase, not Keavi’s, got them in a different order: Stu went in on September 14, eight days before the new models. Both codebases had a stretch of early Stu on the old models.
Early Stu on the old models didn't lower the cost per unit of work
Cost per unit of work at each stage, with each codebase's summer set to 100. Lower is cheaper; lines show the 95% range.
Show the numbers
| Stage | Keavi, per value unit | Keavi, index | Second codebase, per value unit | Second codebase, index |
|---|---|---|---|---|
| Old models, no Stu | $9.50 | 100 | $18.05 | 100 |
| Old models, early Stu | $10.36 | 109(95 to 124) | $17.61 | 98(67 to 155) |
| Opus 5.5, Sonnet older | $7.61 | 80(71 to 91) | $9.92 | 55(35 to 91) |
| Both on 5.5 | $2.40 | 25(23 to 28) | $4.89 | 27(21 to 37) |
Exploratory: these stages were not planned before the study. Each index is all AI spend in the window divided by the estimated value of the PRs merged in it. Keavi's September is split at Opus 5.5's arrival, a date from the release history, but chosen after the pre-registered results had been read; its intervals resample the scored PRs. The second codebase fixed its dates from its model-usage records before reading any value, scored every merged PR with a judge that saw titles and descriptions, and resampled days; it has no PRs scored between August 21 and September 13. Dollar figures differ in scope between the codebases, so compare the indexes, not the dollars.
What I can’t test is the finished Stu on the old models. On Keavi its rewrite landed the same day as Opus 5.5, and its checks started three days after Sonnet 5.5.
Within October, Stu runs were cheaper than other work or about even, depending on the codebase and the comparison. The comparison I planned before scoring set Stu runs against other work of the same kinds, in the same weeks, and Stu came out at about half the cost per unit. Counting the sessions that set up each run, it was 0.73, with a range that includes no saving at all. Most of the other work wasn’t hand-driven, either. It came from sessions that ran their own orchestrator and implementer agents, without Stu’s supervisor. Here it is split that way, on both codebases.
Against other work in the same weeks, Stu cost half as much or about the same
Stu's cost per unit of work divided by the other work's. Below 1 means Stu was cheaper; lines show the 95% range.
Show the numbers
| Codebase | Stu runs against | Other PRs | Ratio | 95% interval | Top three sessions' share of the other group's cost |
|---|---|---|---|---|---|
| Keavi, October: the planned comparison | vs all other work | 89 PRs, matched by kind | 0.53x | 0.37x to 0.75x | |
| Keavi, October: split by how it ran | vs sessions I drove by hand | 29 PRs | 0.95x | 0.71x to 1.31x | 44% |
| Keavi, October: split by how it ran | vs agent teams without Stu | 58 PRs | 0.45x | 0.31x to 0.66x | 42% |
| Second codebase, Sep 30 to Oct 31 | vs sessions I drove by hand | 11 PRs | 0.42x | 0.24x to 0.94x | 83% |
| Second codebase, Sep 30 to Oct 31 | vs agent teams without Stu | 19 PRs | 0.87x | 0.62x to 1.35x | 57% |
| Planned comparison, Keavi, October | PRs each | Ratio | 95% interval |
|---|---|---|---|
| Features | 36 | 0.75x | 0.61x to 0.94x |
| Fixes | 23 | 0.50x | 0.22x to 1.55x |
| Design and docs | 17 | 0.39x | 0.27x to 0.60x |
| All kinds, run setup sessions counted | 89 | 0.73x | 0.44x to 1.27x |
| All kinds, Codex scores instead of Sonnet | 89 | 0.56x | 0.39x to 0.79x |
In the planned comparison, Stu runs did as much per PR as the other work (3.4 value units against 3.3), except in design and docs (1.9 against 2.7). Only the kind of work is matched, not its size or difficulty, and the other work ran mostly on Opus while the runs ran mostly on Sonnet. Refactor and test work (13 PRs each) is too small to report alone.
Only the top row was planned before scoring; the split by how the other work ran is exploratory. "Agent teams without Stu" means a session that dispatched orchestrator or implementer agents, or invoked a run skill itself, without the run command being typed. Keavi's Stu runs are its 89 kind-matched October runs; the second codebase's are all 242 of its runs in those weeks. Resampling whole sessions instead of PRs widens the second codebase's plain-session interval to 0.25x to 1.13x. Measured in tokens rather than dollars, every gap shrinks, because Stu runs did more of their work on the cheaper Sonnet model: on the second codebase, 0.75x against plain sessions and 0.96x against orchestration.
Each result in Stu’s favor rests on a few expensive sessions, and part of every gap is the model mix: Stu sends more of its work to Sonnet, which costs less per token. Stu runs were never clearly more expensive per unit than other work in the same weeks.
So the evidence points to the models for most of the drop. That’s good news, even if it wasn’t the news I expected. Agents that stop when their instructions say to, the thing I built Stu to force, are now mostly what I get from the models on their own. By October neither codebase had a single turn past 800K, inside Stu runs or out.
How Stu works
Your main session becomes a supervisor that isn’t allowed to read the work. It holds a file path, a scoreboard of unit IDs, and a cycle count. Everything else happens in disposable subagents, and their context goes away with them. Each orchestrator returns a short, fixed-format report, and the supervisor checks it mechanically. It rejects any ID it never issued, and a run that can’t pass the checks stops rather than guessing.
No agent grades its own work:
- 1
Writes it
An implementer agent, in its own worktree.
- 2
Reviews it
A different agent, in its own checkout of the exact commit.
- 3
Re-checks it
A merge gate that gets only the PR number and collects its own evidence.
- 4
Merges that commit
Pinned to the reviewed SHA. Or, in report-only mode, hands it to you.
Anything only you can decide becomes a parked question with a recommendation, and the run moves on. You answer the questions later, in a short /stu:settle session that ends with the command to start the next supervisor.
Why I still believe in Stu
Stu’s effect on the bill is now small and uncertain. It does three things the models don’t.
1. It saves my attention
A Stu PR took far less typing than other work
Keavi. Characters I typed per merged PR, not counting the one-line run and settle commands. Shorter is better.
Summer, before Stu
September, mostly older models
October, 5.5 models
Show the numbers
| Period | Work | Merged PRs | Typed messages per PR | Characters per PR |
|---|---|---|---|---|
| Summer, before Stu | All work | 943 | 2.32 | 638 |
| September, mostly older models | Stu runs | 476 | 1.08 | 131 |
| September, mostly older models | Other work | 170 | 2.41 | 553 |
| October, 5.5 models | Stu runs | 928 | 0.76 | 59 |
| October, 5.5 models | Other work | 145 | 2.57 | 531 |
Exploratory: this measure was chosen after the planned results were in. A typed message is one I wrote into a top-level session; command output, notifications and the run and settle commands (pasted from the end of the previous session) are left out. A Stu run's typing includes its setup and settle sessions. Other work counts only the sessions that opened at least one merged PR, so leftover chatter is left out too. Other pasted text, including summer's manuals and the August paragraph, counts as typing.
Not counting the one-line commands I copy from the end of the previous session, a Stu PR in October took 0.8 typed messages against 2.6 for other work. The comparison flatters Stu a little: a run’s PRs share one opening session, and the other work was leftovers and odd-sized jobs. I wasn’t hands-off, either. I sent about a fifth more messages a day than in summer, and merged almost twice as many PRs. Reading Stu’s questions and PRs doesn’t show up in these counts.
2. It checks the rules
The 5.5 models keep their budgets most of the time, but 4 percent of October’s implementers still ran past theirs. A session that stops following its rules is the most expensive work there is, and that has included Stu’s own supervisor. On the second codebase, one ran past 300 turns and cost $21.50 per unit of work, against $5.54 for the other Stu runs in those days. Without it, Stu runs in those old-model weeks cost about half as much per unit as hand-driven work. With it, they cost about three quarters as much, with a range that includes no saving. Since October 1, Stu enforces its rules itself, whichever model is running.
3. It works for hours between short check-ins
Every PR is reviewed and merged by agents that didn’t write it, and anything only I can decide waits as a parked question while the run carries on. Each supervisor runs until it hands off, and the next starts when I settle its questions. The run below merged 48 PRs in about 32 hours of supervisor time, and I checked in 10 times, for 2 to 23 minutes each. The 29-hour gap in the middle is a Sunday when I didn’t check in.
One run
This run is from Keavi, which isn’t public yet, so I’ve swapped the feature for stand-ins. The counts and timings are real.
A pre-launch review found a dozen correctness bugs. One looked like a one-line fix: deleting a project left its auto-assign rules running. Stu parked it with three options, and I rejected all three. You shouldn’t be able to delete a project while rules still point at it. That turned a small fix into a feature across the server and both mobile apps.
One run: 48 PRs in 57 hours, with 10 short check-ins from me
Keavi, a server and two mobile apps. Hours run from the first merge. Each blue bar is a supervisor session, each tick a merged PR, each orange dot one of my check-ins.
Show the numbers
| Event | Hours from first merge | Detail |
|---|---|---|
| Supervisor 1 | -12.6 to -4.2 | 8.5 h |
| Supervisor 2 | -0.4 to 1.4 | 1.7 h |
| Supervisor 3 | 30.7 to 33.5 | 2.8 h |
| Supervisor 4 | 33.7 to 37.7 | 4.0 h |
| Supervisor 5 | 38.4 to 40.5 | 2.1 h |
| Supervisor 6 | 41.0 to 46.6 | 5.6 h |
| Supervisor 7 | 47.8 to 53.7 | 5.9 h |
| Supervisor 8 | 55.7 to 57.2 | 1.5 h |
| Check-in 1 | -15.9 | 18 min |
| Check-in 2 | -12.9 | 16 min |
| Check-in 3 | -0.5 | 2 min |
| Check-in 4 | 30.5 | 10 min |
| Check-in 5 | 33.5 | 9 min |
| Check-in 6 | 38.1 | 4 min |
| Check-in 7 | 40.6 | 23 min |
| Check-in 8 | 47.7 | 2 min |
| Check-in 9 | 55.3 | 11 min |
| Check-in 10 | 57.3 | 8 min |
| PR merged (1 of 48) | 0.0 | tooling or docs |
| PR merged (2 of 48) | 0.4 | server |
| PR merged (3 of 48) | 0.5 | tooling or docs |
| PR merged (4 of 48) | 0.6 | iOS |
| PR merged (5 of 48) | 0.8 | tooling or docs |
| PR merged (6 of 48) | 31.0 | iOS |
| PR merged (7 of 48) | 31.1 | server |
| PR merged (8 of 48) | 31.4 | Android |
| PR merged (9 of 48) | 31.7 | server |
| PR merged (10 of 48) | 31.9 | Android |
| PR merged (11 of 48) | 32.1 | server |
| PR merged (12 of 48) | 32.4 | server |
| PR merged (13 of 48) | 32.9 | server |
| PR merged (14 of 48) | 33.0 | server |
| PR merged (15 of 48) | 33.3 | server |
| PR merged (16 of 48) | 34.3 | server |
| PR merged (17 of 48) | 34.4 | server |
| PR merged (18 of 48) | 35.8 | iOS |
| PR merged (19 of 48) | 36.3 | server |
| PR merged (20 of 48) | 36.7 | server |
| PR merged (21 of 48) | 36.8 | server |
| PR merged (22 of 48) | 37.4 | server |
| PR merged (23 of 48) | 37.4 | server |
| PR merged (24 of 48) | 37.6 | server |
| PR merged (25 of 48) | 38.9 | server |
| PR merged (26 of 48) | 39.1 | server |
| PR merged (27 of 48) | 39.3 | server |
| PR merged (28 of 48) | 39.5 | server |
| PR merged (29 of 48) | 40.0 | Android |
| PR merged (30 of 48) | 40.1 | server |
| PR merged (31 of 48) | 40.2 | server |
| PR merged (32 of 48) | 40.3 | server |
| PR merged (33 of 48) | 40.5 | server |
| PR merged (34 of 48) | 41.5 | server |
| PR merged (35 of 48) | 42.1 | server |
| PR merged (36 of 48) | 42.9 | server |
| PR merged (37 of 48) | 43.6 | Android |
| PR merged (38 of 48) | 43.6 | server |
| PR merged (39 of 48) | 44.3 | iOS |
| PR merged (40 of 48) | 45.8 | Android |
| PR merged (41 of 48) | 45.9 | server |
| PR merged (42 of 48) | 48.6 | tooling or docs |
| PR merged (43 of 48) | 48.8 | Android |
| PR merged (44 of 48) | 50.0 | tooling or docs |
| PR merged (45 of 48) | 51.2 | Android |
| PR merged (46 of 48) | 53.4 | tooling or docs |
| PR merged (47 of 48) | 56.5 | Android |
| PR merged (48 of 48) | 57.1 | Android |
Three moments from that run:
- The merge gate said no. A PR had quietly declined two automated review findings. The gate failed it until the PR said so.
- The supervisor handed itself off. Twice its context grew more than 30K tokens past where it started, and both times it handed off at the end of its current cycle. That’s the rule I used to paste.
- A check fired on healthy work. Tracking the feature as one unit tripped the no-progress check, so I changed the plan and split it into one unit per PR.
What it doesn’t do
It doesn’t make CI faster, though waiting on CI costs less. It doesn’t remove you: questions that change what gets built still come to you, they just don’t block everything else. And it doesn’t work halfway. A session that starts /stu:run and then does the work itself costs more than every disciplined supervisor combined.
Every rule in Stu is listed with the incident or measurement behind it in docs/design-rationale.md.
Notes on the numbers
A value score is one model’s judgment, no person checked the scores, and each period rests on 120 PRs. The scale isn’t linear, but I add points up as if it were. Sonnet and Codex gave the same score 75 percent of the time and were never more than one step apart, and Sonnet agreed with itself on 87 percent of a 30-PR rescore. Before scoring, I had Sonnet guess each PR’s origin from a shortened copy of the same input. It couldn’t tell Stu runs from other work, but it could tell the months apart better than chance (58 percent against a 36 percent baseline), mostly from which features were in progress, so the month-to-month comparison may carry some of that. On the second codebase the scorer also saw titles and descriptions.
Before writing the method down, I had already looked at the cost side of the October comparison; the method says so. I chose the budget comparison, the splits by stage and by how the other work ran, and the typing measure after seeing the planned results. Every dollar figure is at API list prices, not what I paid.
These figures are provisional. The value figures count reverts and fix-ups within two weeks of each merge, checked through November 3, so October’s last PRs have had less than the full two weeks. Reverts were rare in every period. The final check runs on November 15, and if any figure here changes, I’ll correct this post and say so.
Try it
claude plugin marketplace add keavi-app/stu-claude
claude plugin install stu@keavi --scope userThen run /stu:init in your repo and start small: /stu:run "<one small fix>".
New posts by email
One email when we publish something new, and when Keavi opens up. No tracking, unsubscribe in one click.