Held-Out vs. Train-Time API Fixtures in Agent Evals

Teams usually discover fixture contamination the way they discover any other regression: in production, after a customer has already been affected and someone is writing the postmortem. An agent that scored well in the eval suite starts failing on tasks that look, on paper, like the ones it was supposed to have mastered. The postmortem traces the failure back to the harness, and the harness turns out to have been built from the same recordings used to train or fine-tune the model. When API fixtures used during training or reinforcement fine-tuning get recycled into the evaluation suite, the agent is being scored against its own training distribution. The resulting benchmark gains look like capability. They are closer to memorization, and the gap between the two appears once the agent meets a request the fixture store never anticipated.

A held-out set is supposed to mean something specific: the agent has not seen this data, this distribution, this shape of task, before being asked to perform on it. Labeling a set "held-out" does not make it so. Fixture contamination in agent evals is the same gap, one layer down. Instead of a language model being scored on general knowledge it memorized during pretraining, a tool-using agent is scored on API call-and-response patterns it memorized during fine-tuning, and the distance between that score and real tool-use competence is just as wide.

The four-set fine-tune evaluation framework sharpens this further by distinguishing between a model that changed and a model that got better. A win on a set drawn from the same distribution as the training data tells you the training run did something. It does not tell you the something was good: that is what a genuinely held-out set, a capability-drift set, and a safety set are for. Fixture contamination corrupts even the weakest of these claims. If the "held-out" set shares fixtures, a recording pipeline, or a synthetic generator with the training data, the eval cannot even establish that the model changed in the direction intended. The objection that naturally follows, that a team's fixtures are labeled held-out and therefore are held-out, mistakes a filing convention for an architectural guarantee. If the same recording pipeline, the same fixture store, or the same synthetic generator produced both sets, the leakage is structural, and no amount of careful labeling closes it.

How Stateless Fixtures Create the Contamination Surface

A stateless mock is a configuration file wearing the costume of a service. It maps request shapes to canned responses with no memory of what came before. Every fixture it produces is a decontextualized artifact that can be lifted out of its original purpose and dropped somewhere else without anyone noticing. That portability is precisely what makes stateless fixtures useful for fast unit testing and dangerous for eval design.

Consider the "create-then-read" failure that appears constantly in agent test suites. A mock returns a hardcoded customer object on a POST request, then returns a hardcoded subscription object on a later GET request against a related endpoint. It does not learn to use the service the fixture is standing in for, and a held-out eval is supposed to measure whether an agent has learned the underlying service or merely the fixture's pattern.

Stateless fixtures compound this because they are so easy to move. None of this makes stateless mocking wrong as a tool. For unit-level regression testing, a stateless fixture is exactly the right instrument: fast, deterministic, cheap to run on every commit. The argument here is narrower: stateless fixtures are the wrong instrument for an eval environment meant to measure genuine capability, because their portability is the same property that erodes the boundary between train and test.

Drift adds a second failure mode on top of portability. A fixture recorded six months before an eval run may no longer represent the API it was recorded from, and a static recording has no way to flag that it has gone stale. The Kalshi SDK illustrates this concretely. On October 9, 2026, a nightly spec-drift run against that SDK found that the contract suite no longer matched the upstream OpenAPI and AsyncAPI specifications. The replay system was well designed and it still drifted, because a recording, however well captured, has no internal mechanism for noticing that the world it describes has moved on.

What stateful evals catch that stateless fixtures miss

An agent evaluated against fixtures it has already encountered during training is demonstrating fixture recognition rather than tool-use skill, and the failure modes this conceals are the same ones that appear in production once the fixture's canned world is replaced by a real one.

The Stripe integration benchmark makes this concrete. One agent's own reasoning captured the failure directly: "Good, the endpoint is working, it's returning a proper Stripe error for an invalid customer ID." That is a plausible-sounding sentence attached to a completely wrong conclusion, and it is exactly the kind of failure a stateless fixture would never expose, because a stateless fixture configured for that call would simply return whatever canned response someone wrote for it. A live API, or a simulator that behaves like one, returns the actual error and forces the agent to recognize that an error is not success.

AppWorld, presented at ACL 2024, offers the cleaner contrast. Because the simulation tracks its own state, held-out test tasks remain genuinely unseen even to an agent that has been heavily trained on the environment's general shape. Under that design, GPT-4o solves roughly half of normal tasks and about a third of challenge tasks. Those numbers mean something specific because the test tasks were not available to be memorized in advance; a contaminated fixture-based version of the same benchmark would likely produce very different, and far less meaningful, numbers.

The same logic applies to how success is measured as well as to how tasks are generated. Work on agent evaluation in 2026 points out that judging task completion from the agent's final text response alone will pass an agent that reports booking a flight without actually booking it. tau-bench closes that gap by checking the state of an underlying database as the record of what actually happened. Verifying outcome against actual state requires an evaluation environment that has state to check. A stateless fixture has no database behind it to query, so there is nothing to verify against, and the agent's self-report becomes the only available signal.

This also changes what an eval can say about an agent's trajectory, not just its final answer. Whether an agent called the right tools in the right sequence, whether it looped unnecessarily, whether it recovered from an early mistake: all of that is invisible to a fixture that checks only the shape of the final response. If the agent has already seen the expected trajectory during training, its trajectory score during eval tells you it remembers the path, not that it can find a new one.

The fault-injection dimension that contaminated fixtures cannot test

A fixture set recorded against a cooperative API, one that always behaves the way it is supposed to, cannot test whether an agent recovers when the API stops cooperating. That gap sits at the center of why agents fail in production in ways their eval scores never predicted, because recovery from degraded conditions is where real-world tool use differs most sharply from scripted tool use.

The faults that do the most damage are not the loud ones. An HTTP 500 or a timeout tells an agent, unambiguously, that something went wrong and a retry is in order. OccuBench formalizes this distinction with an E1/E2 taxonomy. An agent facing an E2 fault has to notice, on its own, that something is wrong with data that looks structurally fine.

A stateless fixture recorded against a cooperative API is, by its nature, incapable of producing an E2 fault. It was built to return a clean, correct-looking response every time, because that is what a cooperative API returns, and the fixture simply replays that behavior. An agent evaluated only against such fixtures has never once been asked to detect the failure mode most likely to hurt it in production.

Fault injection also has to run inside a stateful environment to mean anything. Recovery testing of this kind is still an emerging discipline, and frameworks built around axes like consistency, robustness, and fault tolerance are starting to formalize what "recovering well" even means for an agent. Teams building eval harnesses now have the chance to build this in from the start, rather than retrofitting it once it becomes the obvious gap in their numbers.

Why eval environments that reconstruct state rather than replay recordings stay clean

The fix for fixture contamination is replacing recorded fixtures with environments that generate their own state dynamically, so that the state an agent encounters during evaluation did not exist at the time the agent was trained, because it had not been generated yet.

A stateful simulator persists state across requests, enforces the transitions between states that a real API would enforce, and fires events, like webhooks, when state changes. An agent's action in step one determines what it will encounter in step three. Every eval run constructs a genuinely new world even when the underlying task definition is held constant. Platforms built this way, Rystic among them, bake that isolation into the simulation layer itself, rather than asking engineers to remember which fixture store belongs to which phase of the pipeline.

The same structural argument holds outside the API-simulation context. The Agentic Time Machine paper proposes an infrastructure for web-based forecasting agents that filters post-cutoff content dynamically rather than replaying a frozen snapshot of the internet. Dynamic reconstruction preserves that realism because it rebuilds the relevant state at the moment it is needed, rather than serving up a snapshot from whenever the recording happened to be made.

None of this works without continuous verification sitting underneath it. A stateful simulator that nobody checks against the live API will drift exactly the way a static fixture drifts, just on a longer and less visible timescale. Stateful API simulators that scale horizontally across eval containers and support fault injection, latency spikes, rate-limit errors, service degradation, expose agents to the same failure conditions they would meet in production, which is what separates genuine tool-use capability from fixture memorization in a way a static recording never could.

Structuring a Fixture Hygiene Discipline

Fixture separation is an ongoing discipline that has to be built into pipeline design, environment architecture, and CI enforcement, because the agent, the API it calls, and the distribution of tasks it is asked to perform all keep changing after the initial split was made.

The four-set structure from the fine-tune evaluation framework, a task-specific holdout, a capability-drift set, a refusal and safety set, and a paired arena against the base model, gives a workable template for fixture hygiene specifically. Each of those four sets needs its own environment, with no shared fixture store and no shared recording pipeline between them, because sharing either one reintroduces the exact leakage the four-set structure was designed to prevent.

Isolation has to happen at the level of the environment, not just the file. The fine-tune playbook's own rule applies without modification here: no overlap, no near-duplicates, and no leakage through a synthetic generator that saw the eval set at any stage.

Where each kind of fixture belongs in a CI pipeline follows naturally from what each one is good at. Each layer answers a different question, and none of the three can substitute for either of the other two.

Fault injection needs to be treated as a core part of this hygiene from the start, not bolted on afterward. The held-out eval set should include E2 fault scenarios, truncated lists, stale values, responses that look superficially valid but carry degraded content, that the agent has never encountered anywhere in its training fixtures. Those scenarios need to be generated by the stateful simulator at the moment of evaluation rather than recorded in advance, for the same reason any other held-out fixture needs to be unseen: a recorded fault is just another fixture waiting to be memorized.

Version and drift tracking closes the loop on all of this. Every held-out fixture set should be pinned to the specific API spec version it was verified against at the time it was built. A nightly drift run, of the kind that caught the Kalshi SDK's contract suite falling out of sync with its upstream specification, is what appears the moment a simulator's state model has drifted from the live API it is supposed to represent. Once that drift is found, the held-out set tied to the old spec has to be regenerated before the next evaluation cycle runs against it.

Offline evaluation is reproducible and belongs in CI, but it is blind to inputs it has not been built to anticipate, to multi-turn conversations that drift in ways no fixture scripted, and to tool failures that only occur against systems that are actually live. Fixture hygiene earns the offline layer the right to be trusted. A team that treats fixture hygiene as infrastructure, with its own architecture, its own verification cadence, and its own place in the CI pipeline, has earned the right to believe its eval numbers mean what they claim to mean. A team that has not done this work is still shipping confidence intervals computed on contaminated data, whether or not anyone has noticed yet.