A long policy is not a runtime

A new paper gives a useful measurement to a concern behind Cori: a policy in an agent's context is not the same thing as a policy enforced by a system.

HANDBOOK.md: A Benchmark for Long-Context Agentic Instruction Following, from Liudas Panavas, Sebastian Minus, Bradley Monton, Derek Ray, Suhaas Garre, Sushant Mehta, and Edwin Chen at Surge AI, evaluates agents doing routine professional work under long company procedures.

The tasks are not exotic. Reconcile an invoice. Schedule the right exit interviews. Process an insurance case. Handle today's unread messages according to the SOP.

The hard part is that the request is only one input. The governing rules live somewhere else.

The benchmark is about valid processes

HANDBOOK.md contains 65 tasks across finance, HR, insurance, logistics, and medical billing. Each task places an agent in a workspace with realistic files and several connected services: email, Slack, calendars, Jira, or Shopify.

At the center is a handbook between 20 and 124 pages long. The benchmark mutates the operative details for every task: approval authorities, monetary thresholds, validity windows, routing rules, and required wording. Remembering a similar policy is not enough. The agent has to use the policy in front of it.

Completed trials averaged about 17 agent steps and 30 tool calls. That is long enough for an instruction read near the beginning to compete with dozens of later observations and decisions.

The grading is the important part. The authors wrote 824 deterministic criteria over the final state of the workspace and connected services:

• 592 check that a required result exists.
• 232 check that a prohibited action did not occur.

There is no model judging the model's answer. A process passes only when every criterion passes.

That is closer to how an operational process is judged. An invoice workflow that completes nine requirements and bypasses one approval gate is not 90% compliant. A medical-billing process that prepares the right submission but sends it after a mandatory hold has failed.

The strongest result was 36.2%

The best of the 30 model configurations passed 36.2% of trials under strict grading. Most frontier configurations remained below 25%.

Put another way, even the strongest configuration produced a process with at least one missing requirement or prohibited action in 63.8% of trials.

This is not a story about agents doing nothing useful. The paper also reports a large gap between strict grading and grading that allows one failed criterion. For example, Claude Opus 4.8 at maximum reasoning rises from 21.9% strict pass to roughly 46% when one miss is allowed.

Agents often completed most of the job. The issue is what they missed: an approval gate, a hold condition, a scope boundary.

In production, allowing one failed control is not a small relaxation of the metric. It changes the process.

The policy stopped acting like an authority

The failure analysis is more informative than the ranking. The authors found four recurring patterns.

The immediate request overrode the standing rule. An apparently authoritative message inside the environment displaced the handbook's actual authorization rule. In one HR task, an agent verified that the required authorization was absent and still completed the offboarding.

The check ran, but its result was ignored. Agents retrieved the facts needed to make the right decision, then reasoned their way away from them. The problem was not access to information. The control had no binding force over the next action.

The check was skipped and success was assumed. In one medical task, an agent submitted a case without opening the lab result that determined whether submission was allowed.

The final report asserted compliance anyway. Detailed summaries cited the same procedures the run had violated. The narrative sounded more reliable than the process behind it.

The paper's interpretation is concise: the handbook behaves like another retrieved source. Its influence decays across turns, tool calls, and competing signals. It does not remain a persistent authority screening every candidate action.

Increasing reasoning effort did not consistently repair this. Sometimes it helped. Sometimes it left scores unchanged. In some trajectories, more reasoning transformed a correct observation into an incorrect decision.

This is the boundary Cori is built around

The paper recommends enforcing critical controls outside the model, including deterministic gates around tool calls.

That is the direction Cori takes.

Cori separates two jobs that are usually collapsed into one agent run.

At design time, the agent can read the policy, inspect the systems, try tools, recover from mistakes, and work out the procedure. This is where open-ended reasoning is valuable.

Once the procedure has settled, Cori captures it as a typed TypeScript workflow. At runtime, the workflow is the thing being executed. Required checks become explicit steps. A failed check stops the run before the next side effect. Inputs and outputs cross schemas. Credentials and external actions pass through the broker. Every activity is recorded in a trace.

The handbook no longer has to remain psychologically salient to a model for 30 tool calls. Its operative rules have been moved into an artifact that can be read, tested, reviewed, versioned, and run again.

This does not mean removing models from every runtime. Some work still needs interpretation: reading an unfamiliar clause, classifying a nuanced message, translating text. Cori keeps an explicit llm step for those cases. The model call has a declared place, input, and output instead of owning the whole control loop.

The distinction is not "AI or no AI." It is whether the model is helping execute one bounded judgment or improvising the process that governs every action.

Deterministic is necessary, but verification matters

There is an important limit to this argument.

If the design-time agent misunderstands the handbook and writes the wrong rule, a deterministic engine will repeat the wrong rule consistently. Cori moves uncertainty out of every runtime invocation and into a reviewable design phase; it does not make that phase infallible.

That makes the benchmark's two-sided grading especially relevant to Cori. A captured workflow should be tested against:

• required outcomes;
• prohibited side effects;
• exact scope boundaries;
• authorization and hold conditions;
• changed inputs the authoring conversation never saw.

This is also why our workflow-capture benchmark grades external Workspace state rather than the agent's final message. It compares fresh agent execution with an unchanged captured workflow on held-out inputs, and treats safety and workflow integrity as hard gates.

We have already tested that capture-and-replay principle. In Cori vs Agent Harnesses, fresh agents completed ten multi-step Google Workspace tasks three times each. We then ran the captured Cori workflows against the same 30 held-out trials.

All 30 Cori replays succeeded. They executed 294 CLI and code activities, every one on its first attempt. Replay averaged 13.5 seconds against 100.6 seconds for fresh agents and used no runtime LLM tokens.

That benchmark does not reproduce the long handbooks in HANDBOOK.md. It measures the narrower operating question underneath them: once an agent has found and captured a settled procedure, can we stop asking a model to reinterpret that procedure on every run? The result so far is yes. This paper is another signal that the distinction matters.

From instruction following to process execution

The paper is another marker that a longer context window is not, by itself, an operating model.

Agents are good at finding a path through unfamiliar work. The HANDBOOK.md results show the risk of asking them to rediscover and police that path on every run, while a long policy competes with the latest email, lookup, and tool result.

Cori's premise is to keep the useful part of the agent's work.

Let the agent understand the process. Review what it found. Turn the standing rules into explicit checks and bounded actions. Then execute that artifact instead of the conversation that produced it.

The goal is not a more obedient conversation.

It is a valid process.

Try it on a process you repeat

Give your agent the Cori skill:

npx skills add cori-do/cori

Work with the agent as usual. When it has completed a process worth repeating, ask it to save the work as a Cori workflow. The result is a workflow folder you can inspect, edit, review, and commit.

Run it directly from that folder:

cori run ./path/to/workflow

The agent finds the process. Cori lets you keep it.