Minimal Reproducible Task Environments for Agent Benchmarking

Agent benchmarks have become the default way to claim capability, yet a score on one benchmark rarely transfers cleanly to another. The reason is structural rather than incidental: the environment beneath the benchmark shapes what the score actually measures. Three design problems, stateful reset, interface fidelity, and deterministic external dependencies, sit underneath every result, and a failure in any one of them corrupts the number silently. This article lays out each requirement, shows how they compound, and gives a construction path for environments that can actually stand under scrutiny.

Why agent benchmark results are harder to trust than they look

The number that gets quoted from an agent benchmark is often less stable than it looks. Research on efficient benchmarking found that fully standardized evaluation runs can cost about $40,000 across nine benchmarks, and that absolute score prediction degrades under scaffold and temporal shift even when rankings hold. A leaderboard position can survive a change in evaluation harness, but the reported score itself drifts. Anyone comparing two models by absolute score is comparing numbers that may not mean the same thing from one run to the next.

That instability is structural. Agent performance depends not just on the model but on the scaffold, the harness governing tool use, memory, retry logic, and execution flow. Two runs of the same benchmark with different scaffolds are not measuring the same thing, because the scaffold changes what the agent can do at each step. The same model can get different scores under different harnesses with no change in its underlying capability.

Most benchmark environments bundle together three distinct design problems without distinguishing them. Resetting state, matching the real interface, and controlling external dependencies are treated as one undifferentiated engineering task, so a failure in any one layer corrupts the results silently. When a layer breaks, the benchmark still produces a number. Nothing announces that the number is noise, and that invisibility is what makes the design choices consequential.

What "reproducible" actually requires in an agent environment

Reproducibility in agent benchmarking means that two runs of the same task, by the same or different agents, start from an identical world and are evaluated against the same ground truth. Anything preventing this is an environment design failure, not a measurement artifact. That standard is stricter than reproducibility in static NLP benchmarks, where a prompt and a reference answer are sufficient. Agents act over multiple steps, and each step changes the world the next step sees.

Gaia2 makes the requirement concrete. Its 1,120 scenarios run inside asynchronous environments that evolve independently of agent actions, and evaluation is done by a write-action verifier that checks every state-changing action against oracle annotations. Reproducibility is therefore not just about the starting state but about the verifiability of every intermediate action. The same requirement surfaces in efficient benchmarking research: ranking fidelity holds across scaffold and temporal shifts, but absolute score prediction degrades, which means that unless the environment is genuinely controlled, leaderboard positions are the only defensible output, and even those require care.

The word minimal is a design virtue here, not a shortcut. The smallest environment that exercises the relevant behaviors is the easiest to reset, the easiest to verify, and the hardest to game accidentally. Gaia2's ARE platform, the Agents Research Environments framework, provides abstractions for creating asynchronous, event-driven environments and supports continuous extension of benchmarks, enabling the write-action verifier to compare against a known oracle state. OccuBench treats the environment simulator as stateful and interactive, a Language Environment Simulator configured with a system prompt, tool schema, initial state, and state description, which makes the initial state a first-class design artifact rather than an implicit assumption.

Why stateful reset is the first thing to get right

An environment that cannot be cleanly reset to a known starting state turns sequential agent runs into a contaminated sequence, where earlier runs alter the world later runs measure. A stateless mock cannot exercise even a mundane multi-step flow. Create a resource, poll its status, cancel it, get a conflict response: a stateless fixture has no memory of the creation and returns the same response on every call. The polling loop either never terminates or terminates immediately, and the conflict path is unreachable entirely.

A true stateful environment maintains a persistent world across calls. A created record exists, contributes to aggregates, and can be retrieved or deleted, and when the task resets, all of that is wound back to the starting snapshot. Stateful API simulators with fault injection support make this reset cycle reliable and repeatable at scale, so the environment can serve as the authoritative ground truth for every evaluation run. Rystic's stateful simulators maintain world state across sequential API calls, and the per-environment isolation that makes CI pipelines safe also makes each simulator instance independently resettable for RL rollouts.

Reinforcement learning sharpens the requirement further. RLVR needs not just a starting state but thousands of independently resettable instances, because gradient updates depend on variance across rollouts. A single shared environment that leaks state across rollouts corrupts the reward signal. The same infrastructure that serves a developer's integration test can scale horizontally across training containers, because the isolation that makes one instance trustworthy is what makes a thousand instances independent.

How interface fidelity determines whether the benchmark tests the agent or the mock

When the simulated interface diverges from the real one, the agent learns to satisfy the simulator, not the API. Benchmark scores become a measure of how well the agent adapted to the mock's quirks. Spec-derived mocks inherit the accuracy of the OpenAPI document, while handwritten mocks inherit the accuracy of somebody's memory, and neither approach includes a mechanism for detecting when it has drifted from the real API.

Gaia2 addresses fidelity by construction. It is built on a consumer environment with the open-source Agents Research Environments platform, featuring realistic apps for email, messaging, calendar, contacts, and more, specifically to ground scenarios in interfaces that behave as deployed systems do. The asynchronous event model is not a stylistic choice but a fidelity requirement, because real apps emit events independently of agent actions. OccuBench identifies the same concern from a different direction: strong agents are not necessarily strong environment simulators, and simulator quality is critical for LES-based evaluation reliability. The tool generating the environment responses must itself be validated before the agent scores it produces can be trusted.

Rystic publishes fidelity scores for each simulator, verified against the live API before every release, so the gap between simulator and real API behavior is a tracked, auditable number rather than an assumption. An environment built on a simulator with a known, high fidelity score gives benchmark results a quality floor that handwritten mocks cannot provide. Tasks built on well-known APIs carry the opposite risk: a model that has seen an API's documentation in training may score highly through recall, not competence, meaning high benchmark scores reflect memorization rather than generalizable tool-use. Fidelity runs in both directions.

What deterministic external dependencies actually require

Any external dependency that is not fully controlled introduces a source of variance indistinguishable from agent error, making it impossible to know whether a task failure is the agent's fault or the environment's. Live API calls introduce rate limits, credential management, network latency, and upstream state changes, none of which are under the benchmark operator's control. One analysis from 2026 found a 42% higher build failure rate for teams relying on live third-party endpoints compared to those using isolated mocks.

Asynchronous events make this worse. A standard checkout flow initiates a payment synchronously but depends on an asynchronous webhook delivery seconds later to complete. If that delivery is delayed or dropped by a live service, the task fails for reasons the agent cannot observe or address. Gaia2 resolves this by running all interactions within the ARE platform's fully simulated environment, so asynchronous events are generated by the benchmark infrastructure, not by upstream services. The environment evolves on a schedule the benchmark controls, not a schedule a third-party service determines.

OccuBench addresses the same problem through the LES architecture. Because the simulator generates tool responses directly, there is no live network dependency to introduce variance, but this trades the live-API variance problem for the simulator fidelity problem. That is why fidelity and determinism are interdependent, not separable. Fault injection belongs in this layer: injecting controlled faults such as timeouts, rate limits, partial responses, and schema drift is only meaningful when the non-fault baseline is itself deterministic. Otherwise the injected fault cannot be distinguished from ambient noise. OccuBench's systematic evaluation found that fault type matters, with implicit faults such as truncated data and missing fields harder than explicit errors like timeouts and 500s, because they lack overt error signals and require the agent to independently detect data degradation.

For RL environments, determinism is load-bearing. The reward signal for a given action sequence must be identical across rollouts for gradient updates to be meaningful. A live API that returns different state on two calls to the same endpoint corrupts training in ways that may not surface until production. Rystic's simulators eliminate live network dependencies entirely. They run locally hosted with no rate limits, are offline-capable and credential-free on the Enterprise plan, and support fault injection as a first-class feature, so both the clean baseline and the fault scenarios are deterministic and repeatable. The same infrastructure scales horizontally across thousands of RL training containers without the coordination overhead that live sandbox environments impose.

How the three layers compound and where most environments fail

The three layers are not a checklist to complete in sequence. Each one depends on the others, so a partial solution at any layer degrades the value of work done at the other two. High fidelity without stateful reset means agents can discover the simulator's real behavior, but two runs of the same task start from different worlds, so scores cannot be compared across runs or agents. Stateful reset without interface fidelity means runs are perfectly comparable, but they are measuring the agent's ability to satisfy a mock, not an API, and the score does not transfer to production.

Both reset and fidelity without deterministic external dependencies mean the environment is correct and stable except when it isn't, and the benchmark cannot tell the difference. Research on efficient benchmarking shows that rank-order prediction remains stable under scaffold and temporal shifts even as absolute score prediction degrades. Even well-designed benchmarks lose score validity under distributional shift. Poorly designed ones lose it in the baseline condition.

The practical failure mode is that teams treat these as separate engineering problems. The infrastructure team controls the reset mechanism, the API team owns the mock, the platform team manages live service dependencies, and no one owns the interaction between them. That organizational structure guarantees that the compounding failure escapes attention, because each team can truthfully report that its own layer is functioning.

A diagnostic question for each layer exposes the gaps. Can the complete world state be snapshotted and restored in under a second? Does the simulator have a published, continuously verified fidelity score? Can the full benchmark suite run without any live network calls? If the answer to any of these is no, the benchmark has a known validity gap at that layer.

Building a minimal reproducible environment in practice

A minimal reproducible task environment is the smallest system that satisfies all three layers at once. The design choices that achieve minimality at each layer also tend to be the choices that make the environment easier to extend, verify, and reuse. Start with the task specification: define success as a state diff, what changed in the environment, not what the agent said, so that the verifier is independent of the agent's trajectory. Gaia2's write-action verifier takes this approach, checking every state-changing action against oracle annotations, which makes it directly usable for RLVR as well as for benchmarking.

Containerize the environment boundary so that all API calls are intercepted and routed to the simulator at the network level. The agent code needs no modification. The same SDK call that would hit a live service hits the local simulator instead. This is what makes the environment transparent to the agent and eliminates a class of accidental live-service dependencies. Pre-seed with realistic data, because an empty environment produces unrealistic agent behavior. Agents that list resources and find nothing behave differently from agents operating in a populated world. Realistic pre-seeded data is a fidelity requirement for the task, not just for the API.

Make fault injection explicit and scheduled. Define which faults fire on which calls, at what rate, and in what order, so fault scenarios are as reproducible as happy-path scenarios. OccuBench's taxonomy of explicit errors, implicit data degradation, and mixed faults is a useful starting point for deciding which faults to cover. Verify the simulator continuously, because fidelity is not a property established at build time and assumed thereafter. APIs change, and a simulator that was accurate at release drifts silently. Rystic's approach, running verification probes against the live API before every release and publishing the resulting fidelity scores, gives benchmark operators an auditable signal for when a simulator has drifted, and scores from before and after the drift are not comparable.

For RL environments specifically, the same infrastructure should scale horizontally across training containers without coordination overhead. Each container gets its own isolated simulator instance with its own starting state, and the reward signal is computed from the state diff after the rollout, not from a live API response. Efficient benchmarking research offers one additional practical shortcut: once the environment is correctly built, not every task needs to run on every evaluation. A mid-range difficulty filter can reduce evaluation tasks by 44–70% while preserving rank fidelity, which means tasks with intermediate historical pass rates carry the most discriminative information.

The investment compounds. The same simulator infrastructure that serves a developer's CI pipeline also serves the benchmark, which also serves the RL training loop. Build the environment once, at the smallest size that satisfies all three layers, and the same stateful, high-fidelity, deterministic foundation carries the work across testing, evaluation, and training.