Cori vs Agent Harnesses

Give an agent the same task three times and it may take three different paths to the same result.

The destination can be identical. The route rarely is. Execution time moves. Tool calls move. Token use moves. Each run is a new interpretation of work the agent may already have completed successfully.

That is the problem behind this benchmark. It gives us a broader way to test agent harnesses: ask each one to solve a realistic multi-tool task and capture the procedure as a Cori workflow, then compare the fresh Agent LLM runtime with the captured Cori code runtime. We grade the resulting Workspace state, not the confidence of the final answer.

Ten tasks across the operating week

The fixtures are synthetic. The work is ordinary: daily queues, weekly reviews, meeting prep, onboarding, leave, and finance reporting. These are the jobs people already hand to agent harnesses, with the same mix of reads, business rules, writes, and controlled side effects.

No task is difficult because of one action. The difficulty sits between them: several systems, exact output contracts, and side effects that must stay controlled.

Variance is an operating property

We ran fresh agents three times on each of ten multi-step Google Workspace tasks. The fixtures were equivalent, and outcomes were graded from Workspace state — not from the agent's final message.

On support inbox triage, one run made 9 tool calls. Another made 26. The 9-call run passed. The 26-call run missed a required digest and failed.

That's the part of agent reliability that averages hide.

Across those tasks, the median run-to-run coefficient of variation was:

• 23.4% for tool invocations
• 13.9% for tokens
• 11.3% for execution time

The tool path moved most.

Support triage ranged from 9 to 26 calls. Expense auditing, 10 to 21. Customer meeting prep burned between 254,000 and 671,000 tokens.

And a stable-looking run can hide the uncertainty somewhere else. Budget variance reporting used eight or nine tool calls and roughly 282,000 to 307,000 tokens every time — but wall time ranged from 91 to 160 seconds.

Three trials per task is a small sample. These are descriptive results, not a universal estimate of agent variance. But the production question shows up immediately:

Which version of the path are you provisioning for?

The average doesn't tell you how many calls the next run will make. It doesn't tell you where it will retry, what it will reread, or whether it will find a new detour through a system that has side effects.

Your tools feel the variance too

Tool calls aren't private thoughts. They're load — on Gmail, Sheets, Drive, Calendar, your database, your API, whatever internal system sits behind the agent.

A different path means a different number of network requests, auth checks, rate-limit opportunities, partial side effects, and places to stop halfway through. Your infrastructure feels every decision the model improvises.

The prompt becomes a patch log

When an agent misses a case, the natural fix is to improve the prompt.

--- process-prompt-v2.md
+++ process-prompt-v2-final-reviewed-OK.md
 [...]
+Remember to create the digest.
+Check that the counts match.
+Don't send the email.
+If the sheet is empty, look in the archive.
+Verify the draft before finishing.

Each line covers a failure you've already seen. None of them cover the failures you haven't.

The prompt slowly becomes operational code written in prose. But unlike code, it doesn't define one execution path. A new instruction changes the model's whole decision surface — it may fix the missing digest while adding more reads, a different tool order, or a new branch you now have to watch.

So the doubt compounds. When will this fail again? Which case have we not prompted for? How many calls will the repaired prompt make? What state gets left behind if it stops midway?

That isn't only a model-quality problem. It's a process that has never settled.

Worse: the prompt can disagree with itself

There is a failure mode beyond accumulation. The patch that fixes run twelve can contradict the rule added after run three — and both remain active.

This isn't hypothetical, or limited to prompts written in a hurry. Anthropic recently described finding conflicting guidance inside Claude Code itself: its system prompt, skills, and user requests could ask for documentation where appropriate while also saying not to add comments. Claude could usually infer the intent, but first had to decide which instruction should win.

That is worse than ordinary variance. The ambiguity is now part of the specification. A successful run doesn't prove the prompt is coherent; it only tells you which side the model chose that time.

Use the brain where the process changes

Cori treats the agent as a design-time collaborator — the same split I wrote about last week.

Let it inspect the live system, understand the task, try tools, recover from mistakes, and verify the result. That exploration is worth paying for. Then capture the procedure it found as an explicit workflow.

At runtime, the computer executes that workflow. Inputs are declared. Tool requirements are visible. Control flow is fixed. Retries are policies rather than improvisations. Every step shows up in a trace.

The agent comes back when the process changes — not because Tuesday arrived.

In the same benchmark, unchanged Cori workflows replayed all 30 task trials successfully. They executed 294 CLI and code activities, and every one completed on its first attempt. Replay averaged 13.5 seconds against 100.6 for fresh agents, and used no runtime LLM tokens.

Those speed and token numbers are nice, but they're downstream of the change that matters: runtime stopped being a new interpretation of the task.

Agents should help us use computers

The goal isn't to get agents out of the work. It's to stop spending general-purpose reasoning on settled mechanics.

We want agents to help us understand systems, design processes, and adapt when reality changes. We don't need an overqualified brain rediscovering the same six API calls every morning.

Variance is useful while you're still looking for the process.

Once you know the process, variance is debt.

Use the agent to decide what the computer should do. Then let the computer do it.

Try it on something you repeat

Give your agent the Cori skill:

npx skills add cori-do/cori

Then work with the agent as usual. When it has found a result worth repeating, say save_workflow. The skill turns the settled procedure into a workflow folder you can inspect, edit, and commit.

Run it directly from that folder:

cori run ./path/to/workflow

Start with a task you already repeat. The second run is where the difference becomes obvious.