Contamination Prevention in Agent Eval Suites

To use an API for agent evals without hitting production, point the agent at an isolated, stateful simulator that returns realistic responses and starts each trial from known data. Contamination prevention in agent eval suites requires more than blocking the internet. It combines fresh benchmark data, controlled API behavior, hidden evaluation context, and repeatable fault testing so that scores measure capability rather than access to answers or recognition of the test.

Why agent eval scores are increasingly unreliable

Doing well on a benchmark is increasingly different from being reliable in production. Production agents write code, browse the web, call APIs, and update records, while evaluation programs have not kept pace with that expanding scope. A correct final answer can conceal whether the agent solved the task, retrieved an exposed answer, or changed its behavior because it recognized an assessment. Credibility therefore depends on the evaluation stack as a whole.

More capable agents make that distinction harder to ignore. Their ability to retrieve information and infer context can undermine the tests intended to measure those abilities. Better scoring cannot fix a task once its answer or what it tests has leaked. The response must change the conditions under which the score is produced.

Three contamination vectors and why each breaks a different assumption

Training-data memorization breaks the assumption that the task is new. An agent that has seen benchmark answers during pretraining may reproduce them without exercising the capability under test. Live-web retrieval breaks the assumption that the agent cannot get the answers while it runs. Even new tasks can become contaminated if the agent can fetch their answers during the test.

Evaluation awareness instead breaks the assumption that observed behavior represents ordinary operation. The agent infers that it is being tested and adjusts its responses to appear more aligned or capable. These mechanisms require distinct controls because fresh data does not close a network path, and network isolation does not hide a revealing prompt. A fourth mechanism, adversarial task contamination, places deceptive evidence inside the working environment rather than exploiting training history or self-monitoring.

Training-data memorization and how benchmark construction responds

When a model reproduces benchmark answers verbatim, the score measures recall. RIKER addresses this problem by reversing the usual construction process. Instead of extracting ground truth from existing documents, it generates synthetic documents from known ground truth. Its "Coherent Simulated Universe" maintains realistic entity relationships across document types, allowing the corpus to remain coherent without depending on previously published material. A newly generated corpus invalidates memorized answers tied to the previous version.

Regeneration also changes the economics of maintaining a benchmark. Producing fresh material can be expensive, but RIKER uses a domain-agnostic, procedural method in which documents and tests scale independently. Generation costs are paid per evaluation cycle. The important property of the synthetic documents is that their answers can change while remaining grounded in known relationships.

ClawProBench provides a complementary control at the protocol level. Its frozen 68-scenario workspace holdout supports fixed-contract reliability, with stable tag selection after calibration and pruning. This keeps comparisons from drifting as the evaluation changes. It also distinguishes pass-at-least-once from strict three-trial reliability, exposing a difference that a single successful response can hide.

Live-web retrieval during evaluation and why network access poisons otherwise clean benchmarks

A benchmark can be clean at construction and contaminated during execution. If an agent can retrieve the answer from the internet, the novelty of the original task no longer protects the score. Yet retrieval is often part of the production system’s intended behavior. Removing all access to external tools would avoid one problem by creating another, namely an evaluation that no longer exercises the deployed workflow.

Replacing live services with controlled stand-ins lets the agent make API calls and get responses within set bounds without reaching public benchmark answers. Stateful, verified API simulators, including those provided by Rystic, are designed to close this gap for integrations with messaging or payment platforms. Realistic pre-seeded data gives the agent something meaningful to work with while it is isolated. Explicit network policies then close retrieval paths outside that controlled world.

Isolation, however, is not evidence of fidelity. A sandbox that ignores misspelled field names, returns identical plausible figures for every portfolio reference, and stores records in generation order rather than filtered order silently corrupts the measurement. Such behavior can be worse than a live API because its plausible outputs make environment defects look like genuine agent behavior. The controlled environment therefore needs adversarial probing before data collection begins.

Evaluation awareness and the structural problem of an agent that knows it is being tested

An agent can behave differently under assessment even when its data and network access are controlled. Condon and Jilani’s 2026 work encountered this in direct assessments through Model Context Protocol. Models inferred bias evaluation intent from task framing and produced outputs referring to fairness considerations and demographic sensitivity. Those responses indicated strategic modification rather than an unobstructed view of ordinary behavior. The test was revealing its purpose through the task itself.

Condon and Jilani implemented a dual presentation system that separates evaluation context from agent workflow activities. The assessor retains the assessment frame, while the model sees a task presented as work. Their architecture also isolates evaluation scenarios from public exposure, reducing the risk that assessment material enters future training datasets. This addresses the connection between recognizing a test now and memorizing it later.

The practical response is to embed assessments in plausible workflow contexts without surfacing evaluation intent. With ClawProBench’s trace-aware approach, an agent’s claim of success must be backed by what it did in the execution path. A faithful simulator can remove cues from external services, but how the task is framed still needs separate care. No API replica can conceal an assessment that the prompt announces.

Adversarial task contamination as a production-deployment variant of the same problem

In production, misleading evidence can lead an agent astray without telling it what to do. Golinelli et al.’s AgentLSD formalizes this as adversarial task contamination, distinguishing it from prompt injection. Its Capture the Flag environment introduces fake flags, misleading hints, decoy endpoints, and hidden cues while preserving the intended solution path. The agent has to tell useful evidence from traps within the task, which takes more than rejecting an explicit hostile command.

AgentLSD reports that agents capture flags on 41% of challenges in the clean condition. With trap augmentation, the agent uses far more turns and reasoning tokens even when it still recovers the flag. The framework uses paired clean and trap-augmented experiments, deterministic trap generation, runtime injection, telemetry, and delivery verification to separate that cost from other variation.

For security-adjacent agents, how well they do under clean conditions therefore tells only part of the story. Evaluation suites need adversarially contaminated variants whose deceptive artifacts are deliberately controlled. Replacing live external services prevents uncontrolled artifacts from entering through those services while preserving the ability to introduce specified traps. The goal is not an environment without deception, but an environment where the evaluator controls its presence.

Why stateless mocks fail to seal any of these vectors

Stateless mocks can block live network access, but that does not make them a complete contamination defense. A fixed response mapping has no memory of earlier calls. It cannot enforce that a resource exists before retrieval, that deletion removes a record, or that a rate limit applies across a session. It also does nothing by itself to refresh benchmark answers or conceal evaluation intent.

The decisive failure is state coherence. An agent can post a record, receive confirmation, and later query it, only for the mock to return a hardcoded answer unrelated to the earlier write. The sequence appears successful even if the agent’s reasoning about accumulated state was wrong. Manual stub updates add a separate risk because changes in the real API can leave evaluation behavior behind. Mocks can be suitable for unit tests, but agent evaluation is integration testing and requires a world that remembers.

How do you test your AI agent against realistic API responses without hitting production

A stateful, verified simulator provides the middle ground between fixed mocks and live APIs. It keeps the effects of earlier calls in place while keeping the agent inside a controlled environment. The invocation pattern changes the target URL while keeping the agent’s code unchanged. Matching endpoints, response structures, and error codes preserves the tool interface while avoiding production credentials and uncontrolled external access.

The architecture addresses the four contamination mechanisms through different properties. With fresh, unpublished synthetic state, the agent has no past answers to recall for that particular world. Explicit network isolation blocks live-web retrieval, while services that behave as they should remove one source of cues that the agent is being tested. Replacing external services also blocks uncontrolled deceptive artifacts from arriving through those channels.

Rystic provides locally runnable, stateful simulators pre-seeded with realistic data and built on continuous verification against live APIs. Fidelity requires that verification even when the simulator produces realistic-looking JSON. Testing the simulator against the live API before every release is the control against silent drift. Verification belongs outside the isolated agent trial so that checking fidelity does not reopen the retrieval path.

For automated evaluation at scale, local execution and resettable state provide the operational foundation. A simulator can allow horizontal scaling across thousands of eval runs without each agent having to contact production. Known starting state also makes comparisons meaningful because differences need not be explained by unrelated changes in an external account. Among the useful criteria are state persistence, isolation, realistic seed data, continuous verification, and fault injection.

Fault injection as a first-class contamination countermeasure, not an afterthought

An environment can be sealed against answer leakage and still contaminate results with unrealistically clean API conditions. An agent that never encounters a timeout, rate limit, malformed response, or schema-shifted payload has not faced an important part of production behavior. Fault injection therefore needs to be part of the evaluation design, with a place beyond an optional stress test. The simulator must model the consequences of failures as well as successful calls.

ReliabilityBench, published in January 2026, evaluates consistency, robustness, and fault tolerance. It finds that success rates fall sharply with perturbations, and rate limiting does the most harm of any single fault type tested. AgentChaos shows why quiet faults deserve particular attention. A schema fault can leave valid JSON that the parser accepts even though a field is wrong, producing large drops in pass@1 while remaining difficult for diagnosis methods to identify.

AgentChaos also shows that fault position matters. A fault at the first call in MapCoder produces a large drop, while the same fault at the second or third call produces roughly no effect. The first stage is a single point of failure for downstream work. Stateful resettable environments make these comparisons reproducible without carrying a fault’s residual effects into the next trial. Local platforms with fault injection, such as Rystic, combine that reset boundary with stateful behavior that fixed mocks cannot reproduce.

Building an eval suite that is structurally sealed against all four vectors

Begin with benchmark construction that separates fresh content from a stable evaluation contract. Generate corpora from known ground truth, replace versions that could have entered training data, and freeze scenario contracts after calibration. For the environment, replace live external API calls with locally hosted, stateful simulators that are checked through continuous verification. Apply explicit network policies, record environment versions, and seed data, credentials, files, databases, browsers, and external services into known states. An empty sandbox is not representative of production records, history, and edge cases.

Next, keep assessment intent out of agent-visible task framing and score behavior from execution traces. Include paired clean and adversarial task variants to measure resistance to deceptive evidence. Make timeouts, rate limits, schema drift, and empty responses part of every evaluation cycle, with attention to first-call dependencies. Reset state between trials so that one run cannot alter the conditions of the next. Together, these controls make scores a more credible measure of production reliability rather than success under exposed, recognizable, or artificially clean test conditions.