Parallelizing Agent Eval Runs Across CI Workers

Parallelizing agent eval runs across CI workers requires more than distributing scenarios. The reliable pattern is to plan the work, fan out into isolated, stateful lanes, and gather evidence through one authoritative gate. Its central engineering requirement is isolation. Without it, shorter runs can conceal shared-state corruption, missing coverage, and results that cannot be reproduced.

Why serial agent eval runs become an engineering problem as suites grow

AMD GAIA PR #4097 describes a serial eval job that took up to 9.2 hours and held the single shared Lemonade slot the whole time it ran. Every other eval on the box queued behind it. A long run occupied the resource needed to evaluate other work, so adding scenarios increased both execution time and contention. The suite’s wall-clock duration became a constraint on the surrounding engineering workflow.

Microsoft/hve-core issue #2939 identifies a similar problem in PR evaluations that serialize ordinary agent work. Existing hosted runs establish a 119.4-minute serial median, against an operational requirement of a 55-minute median target for full-topology runs. As agent and skill coverage grows, the serial job must finish more work before it can deliver a decision. The delay makes the capacity to run evaluations part of release engineering as well as a concern for the test suite.

The plan-then-fan pattern: how to structure a parallelizable eval job

A lightweight planning job runs beside lint and emits the matrix of agents, skills, and lanes that downstream jobs consume. The hve-core issue specifies a canonical, digest-covered plan that emits only nonempty ordinary agent shards and conditionally required baseline-model producers. Ownership is settled before runners are provisioned. In GAIA, eval/ci_lanes.json serves as the single source of truth for category assignments, and a hosted lanes job validates that map before work reaches the scarce AMD pool.

The fan-out then executes those assignments under a bounded concurrency limit. GAIA uses four independent lanes on the ephemeral devlab-dispatch pool, each taking approximately 2 hours and finishing together. The hve-core design permits at most four deterministic, artifact-cohesive ordinary agent shards, plus up to two isolated baseline-model lanes when baseline equivalence is required. Every artifact and run key has exactly one owner.

The gather phase turns those independent executions back into one gate. It is not merely a place to average scores. GAIA uploads transcripts, judge rationales, failure analysis, backend logs, and Lemonade logs from each lane because ephemeral machines are destroyed when jobs end. The former advice to reproduce a failure locally cannot recover a machine that no longer exists. Evidence preservation is therefore part of the execution design.

Why shared state across parallel lanes produces silent, unreproducible failures

Concurrent execution exposes dependencies that serialization can hide. If an API test behaves differently alone than it does beside other tests, state is already crossing worker boundaries. In a ParallelAgent pattern, two branches writing the same session key produce last-writer-wins corruption. Timing determines the apparent result, so it cannot be taken as a measure of agent quality. Reproducing it requires the race to resolve the same way again, which makes a quality regression difficult to distinguish from an environmental failure.

Memory makes this problem hard to spot because remembered state can change what an agent does later without showing up as a resource collision. GAIA’s memory lane opts back in to memory only after receiving its own ephemeral machine, while every other lane keeps memory off. A named preflight step rejects a lane whose memory environment does not match its categories. Coverage can fail just as quietly. A category assigned to no lane disappears from measurement even while every executing lane reports green, so a unit test rejects orphaned categories and the lanes job validates the map before scarce capacity is used.

Lanes can still depend on shared credentials even when each has its own world state. GAIA’s Anthropic organization behind the CI judge credential exhausted its monthly spend, causing every eval to die in preflight. More runner capacity could not resolve that failure. Judge budget management belongs in the same engineering scope as worker provisioning because a shared spending limit can stop all lanes together.

Giving each lane its own isolated, stateful environment

A separate worker is insufficient if it still talks to a shared sandbox. Each lane needs its own world state, independent of every other lane. A worker creating a Stripe payment intent should not encounter payment intents created by a concurrent worker. Yet separation alone is not enough. The environment must remember what happened within its own lane so that later tool calls test the consequences of earlier actions.

A stateless mock can return a hardcoded customer object in step 3 independently of whether the customer was created in step 1. Consistency then comes from the mock’s configuration. Agent evals need the stronger test. Flows that create a record, retrieve it, update it, and verify deletion reveal whether the agent reasons correctly across changing state.

That stateful environment needs realistic pre-seeded data, a way to reset it between runs, and an image that can scale across containers without the overhead of coordinating them. Starting from an empty sandbox does not exercise plausible responses from the first call, while reset capability restores a known baseline without reprovisioning infrastructure. MCP adds a separate session boundary. An MCP server exposes a stateful JSON-RPC 2.0 session beginning with an initialize handshake, after which the client discovers available capabilities at runtime. Each worker therefore needs its own MCP session as well as its own world state.

How the fan-in step validates evidence without masking lane failures

The fan-in must distrust incomplete evidence by design. The hve-core issue requires its authoritative gather step to reject missing, duplicate, unexpected, stale, malformed, and schema-incompatible evidence while preserving current PR gate semantics. Those are distinct failure categories. When evidence is missing, there is no way to account for the planned work, while duplicate or unexpected evidence breaks the plan’s guarantees about who owns it. Stale or incompatible inputs cannot show the result of the evaluation that was meant to run, even if their scores look convincing.

An agent registration guard asks the backend which agents registered and refuses to start when the expected agent is absent, naming the install line to fix. A job must fail if its simulator did not start rather than silently falling back to a live endpoint or to nothing. Otherwise, the fan-in may get evidence that looks like a success from a run that never tested the planned environment.

Mean scores compared with a frozen floor do not distinguish a real change from judge noise. Statistical gating should use Welch’s t-test for continuous rubrics and a z-test for binary rubrics, with both p < 0.05 and a minimum effect floor required to reject a regression. Tail rubrics need p95 evaluation.

Fault injection as a first-class concern within each isolated lane

An isolated lane can test failure behavior without disturbing neighboring runs or the real third-party service. Its owned environment makes latency, overload, unavailability responses, and schema drift controllable per lane. This belongs inside the CI pipeline. Clean-condition evaluation cannot establish whether an agent handles damaged responses, and a separate offline exercise leaves that behavior outside the recurring gate.

The AgentChaos paper, dated August 2026, demonstrates HTTP-layer injection that forwards each LLM request and mutates a named response field before the agent reads it. Its code monkey-patches httpx.AsyncClient without needing changes to the agent source. Across five agent systems and four backbone models, pass@1 fell by up to 49.66 points. Two systems that were close under clean conditions separated by 32 points under faults. Their clean scores hid a large difference in how the systems were built that model choice alone could not explain.

ReliabilityBench, dated January 2026, found rate limiting the most damaging of the compared rate-limit, timeout, and partial-response faults. Truncation presents a different risk because it can remain invisible. Rule-based diagnosis found Timeout and Empty faults at high rates but rarely got Truncate right. The agent can therefore proceed on corrupted context without raising an error. A pipeline that tests only clean responses misses this class of failure, whereas a lane-owned fault configuration can exercise it without contaminating other evaluations.

Simulator fidelity drift as a silent threat to eval validity

The paper identified as arXiv:2609.05975 describes a sandbox that ignored misspelled field names, causing a query intended to filter results to return the entire database while the agent believed filtering had succeeded. Evaluation could then misclassify the environment defect as an agent outcome. The paper concludes that tools should be adversarially probed before data collection. This is the strongest objection to trusting a simulator merely because its interface looks correct.

Ongoing checks with probes address that objection without claiming to be perfect. Structured probes run against the real API, simulator behavior is compared with those responses, and the match rate is published. When real behavior changes, falling probe scores expose the discrepancy before engineers rely on a stale environment. Hand-maintained mocks lack that automatic warning because API changes require manual stub updates. Lane isolation protects runs from one another, but only fidelity verification tests whether their shared environment image still represents the service they intend to evaluate.

Environment choice also limits what an evaluation can claim. A 2026 taxonomy paper identified as arXiv:2605.16282 distinguishes LLM-emulated tools from sandboxed-interactive, containerized, and live environments, documenting a 31.2% false positive rate for the emulated tier. It also notes the absence of consensus about which tier supports which claim. The environment tier should therefore be explicit and documented. Rapidly generated tool responses are not interchangeable with a stateful environment whose behavior is probed against the real API.

Putting the architecture together: what a working parallel eval pipeline looks like

The plan emits the lane matrix, rejects orphaned categories, checks agent registration, and confirms simulator availability before expensive workers are provisioned. Each lane that passes gets an ephemeral machine and its own simulator instance, and shares no credentials, world state, or LLM judge budget pool with other lanes. Inside each lane, realistic seed data and memory matched to its categories set the starting conditions. Route-scoped cases can use in-process parallelism, while fault toggles remain under the lane’s control.

Transcripts, judge rationales, and failure analysis are uploaded before the machine is destroyed, and the fan-in validates them against the planned assignments before applying statistical gates. Cheap deterministic checks and a classifier cascade run on every PR, while a full LLM-judge sweep runs nightly against a versioned dataset. That nightly trigger can catch environment drift from dependency bumps that move nothing in the repository.

GAIA implements its nightly as a schedule trigger on the same workflow, replacing weekly_eval.yml GAIA implements its nightly as a schedule trigger on the same workflow, replacing weekly_eval.yml. That choice avoids the phantom-job bug that filed issues about jobs that never started. Beyond PR gates, isolated stateful environments with reset capability and controlled faults provide a foundation for reinforcement-learning training pipelines. The same architecture can support repeated agent interaction without letting one run redefine the world seen by another.