Deterministic Agent Replay for Regression Testing
Recording agent calls deterministically turns debugging into testable regression cases.
Setting the model's temperature to 0 does not make an agent deterministic, and that single fact is why agent debugging resists the tools engineers already trust. An agent failure is not one wrong output but a trajectory: a sequence of model calls, tool calls, and routing decisions, and any one of those steps can shift on re-run even with sampling locked down.
Why agent failures resist existing debugging tools
Three sources of non-determinism stack on top of each other in a typical agent run. Model inference adds a third layer on top of both: even pinned model versions and fixed sampling parameters don't guarantee bitwise-identical output, so the first link in the chain is already unstable before the tool calls or the routing logic get involved. Tracing systems and evaluation frameworks were built for adjacent problems, not this one. Neither lets a developer take one recorded run, change a single component, like a routing rule or a tool-calling guard, and check whether that change fixes the failure while every other part of the run is held exactly fixed. Chronicle, a 2026 paper from Microsoft researchers Tisha Chawla and Susheem Koul, names this gap directly: existing agent tooling records runs only to trace or score them, not to test a code change against them.
How record-and-replay closes the gap: recording at non-deterministic boundaries
The engineering response to this problem doesn't try to eliminate non-determinism. It records every source of it and substitutes those recordings during replay, which turns a run nobody can reproduce into something a developer can step through exactly, one boundary at a time. Four categories of information need capture at each boundary crossing. LLM interactions require the full prompt, the sampling parameters, the model version, and the exact response tokens returned, not a paraphrase of them. Tool calls require the function name, the arguments passed, and the complete response, including errors, timeouts, and partial results, since a silently dropped error is itself part of the behavior under test. Decision metadata requires the inputs and outputs of whatever routing logic, planner, or classifier steered the agent at each branch point. During replay, the agent doesn't call the model or the tools again. Deterministic stubs return the recorded response verbatim, and if the agent attempts a call that isn't in the recording, the replay engine fails loudly instead of quietly falling through to a live system and introducing the very non-determinism replay exists to remove. None of this is new to software engineering broadly. What's specific to agents is that they treat model calls and tool calls as the meaningful semantic boundaries to record, rather than low-level system calls that don't map onto how an agent actually reasons and acts. Docker's Cagent applies this pattern with a proxy-and-cassette model: in recording mode it captures real provider interactions into a YAML cassette, and in replay mode it blocks external calls entirely and matches incoming requests against that cassette, failing the run deterministically if execution diverges. Docker's own repository describes Cagent as under active development, with breaking changes expected, marking the tool as an early signal of where agent tooling is heading.
Chronicle's cut-point replay: turning a recorded incident into a CI regression test
Chronicle sharpens the replay pattern into something earlier systems didn't offer: it replays a recorded trace after the run has already finished, and it runs new code at a chosen subset of boundaries within that replay, so a candidate fix gets evaluated against the actual past incident. The central operation, which the paper calls cut-point replay, works by letting a developer designate a chosen subset of boundaries, the cut points, to run live with new code, while the complementary subset is served from the record as immutable envelopes. Each stubbed boundary is guarded by a per-name call-count check, so if the new code's behavior diverges structurally from what was recorded, for example by calling a tool a different number of times, the test catches that divergence. The outcome is a committed regression test: it fails against the faulty code that caused the original incident and passes against both the guarded fix and any benign, unrelated change. When every boundary gets stubbed, the tool gate or router change under test never actually runs against the recorded trajectory. Other systems use selective replay for different ends. Temporal's replay tests re-run changed workflow code against a recorded event history, and Chronicle adapts that same selective-replay idea to agent boundaries, with the live set chosen freely. Chronicle operates after the run has already concluded, which makes it a tool for regression testing specifically, not for steering a run in progress.
Tool calls: the boundary that most replay systems leave live
Model call boundaries get most of the attention in replay design because they're the obvious target: a model call returns a sampled value, and recording that value makes the run reproducible. Tool call boundaries carry a harder problem, because a tool call doesn't just return a value. It reads and writes external state that keeps changing independently of the replay run. A recorded tool response is only valid if the world state that produced it still holds at replay time. If it doesn't, the recording is stale, and the replay quietly misrepresents what actually happens. Three distinct things can break a recorded tool response. The compounding effect across a multi-step agent run is severe: if each tool call carries even a small individual failure probability, the compound success rate across many sequential steps drops sharply, and a stale recorded response fails silently. This is the specific gap that tracing, evaluation frameworks, and cassette-style replay all leave open. Each one treats the tool boundary as opaque, recording the response and playing it back, without ensuring the state model behind that response stays coherent across the full recorded trajectory. The question that follows is concrete: what does a tool environment need to look like for a recorded tool call to replay faithfully?
Stateless mocks versus stateful simulators at the tool boundary
A stateless mock can't answer that question, because it can't enforce the cumulative state that a recorded sequence of calls implies. The definitional gap is simple to state: a stateless mock returns the same response for the same request every time, with no memory of the previous request, so it has no way to tell whether the resource posted in step 1 of a trajectory actually exists by step 3. A create-then-read test against a stateless mock proves nothing about how the real API behaves. A stateful simulator closes this gap because it remembers previous requests and updates its internal model of the world accordingly. A replay that creates a resource, reads it back, and then cancels it sees a coherent progression of state, matching what the original live run actually observed, rather than three disconnected, pre-baked answers.
The second failure mode: simulator drift makes recorded tool responses go stale
Statefulness alone doesn't finish the job. A stateful simulator that has drifted from the real API it's meant to stand in for is just as dangerous for replay as a stateless mock, because it accepts calls the real API would now reject, returns fields the real API has since removed, and gives developers false confidence that their recorded trajectories still hold up. Fidelity matters more for agent replay because agents tend to call APIs with complex, chained argument structures, passed from one tool's output into another tool's input. A 2025 paper testing REST APIs as tools for LLM agents, across a range of enterprise applications, found that parameter type and value mismatches were a leading source of error, appearing in a meaningful share of test cases. A replay built on a simulator that accepts a malformed argument the real API would reject can pass a trajectory that would actually fail in production, hiding the gap and producing a worse outcome than an obvious crash. The remedy is continuous verification: testing the simulator against the real API before every release, so that discrepancies surface as simulator test failures rather than as production incidents discovered weeks later.
What a verified stateful simulator provides
Put the requirements together: the right tool environment is a simulator that is stateful, pre-seeded with realistic data, supports fault injection, and gets verified continuously against the real API. No combination of cassette replay or live API access delivers all four at once. Cassette replay, the approach Docker's Cagent takes, solves model call reproducibility but leaves tool state untouched. Live API access in CI has the opposite set of problems. A verified stateful simulator sits between those two failure modes. It's pre-seeded with realistic data, so if a replay reads before it writes, it finds valid records. Rystic builds this class of simulator for APIs that agents commonly call in practice, among them Stripe, GitHub, Slack, Kalshi, and PayPal, with each simulator stateful, pre-seeded with realistic records, and checked against the real service on an ongoing basis. A Chronicle-style cut-point test that runs a tool call live against a simulator of this kind exercises real API behavior without needing real credentials, without hitting rate limits, and without exposure to state that might change underneath the test between one run and the next.
Building the CI pipeline: two execution modes
A working CI setup for agent regression testing needs two distinct execution modes, and the tool environment has to support both without requiring a code change in the agent itself. Live mode, with real tools and a real network, fits canary deployments and production smoke tests, where the point is to confirm the agent still works against actual external state. Replay mode serves the opposite purpose. The simulator stands in for the real API, providing stateful, verified responses that match what the real API would actually return, the agent's tool calls hit the simulator instead of the network, the simulator's internal state evolves correctly across the call sequence, and the recording's envelopes cover the model call boundaries. This yields a bit-stable, zero-credential, offline-capable regression test that can run on every single pull request. A tiered task suite maps cleanly onto this two-mode structure. Smoke tasks run fast on every PR, in replay mode against the simulator, and they cover the most common tool trajectories a team cares about. CI gates for agents need to cover more than output accuracy: step budget, tool error rate, latency, cost caps, and safety constraints all belong in the gate criteria, and a simulator that supports fault injection lets a team gate on resilience behavior, not correctness alone. A/B testing against live traffic answers a different question than replay testing does. A/B testing shows which version users prefer on new traffic; replay testing shows whether the agent still handles known failure inputs correctly, and you can answer that second question through replay without costing a single real user session.
Fault injection as a first-class dimension of replay, not an afterthought
Many recorded production failures come from the tool environment behaving badly rather than from the agent reasoning badly, and a replay environment that can't reproduce adverse tool responses can't reproduce those failures. Rate limiting is one concrete case: an agent hits a 429 response, with a Stripe-Rate-Limited-Reason header, partway through a trajectory, and the test needs to show whether its retry logic fires correctly or instead loops, skips a step, or falls back to an unsafe action. Timeouts and partial failures round out the set: a tool call returns a 503 after a long wait, and the test needs to confirm the agent's timeout handling still matches what was recorded, rather than taking a different recovery path because a route changed. Chaos engineering practice has already established that fault injection belongs in continuous integration, not only in dedicated production stress tests. Capital One has embedded chaos engineering into its payment network SRE practices, and LinkedIn runs request-level failure injection across its microservices, and both demonstrate the value of finding cascading failure paths before they reach users. For agent replay specifically, fault injection needs to be switchable by configuration rather than by code change: the same golden task should run against a happy-path simulator state and, separately, against a rate-limited or auth-failed state, with the scenario chosen at the configuration level. Rystic's Pro-tier simulators support this kind of scenario switching for fault conditions including rate limiting (429 responses), service unavailability (503 responses), and latency, conditions a stateless mock simply has no mechanism to reach.
The data problem inside the simulator: why pre-seeded realistic records matter for replay
A simulator can be fully stateful and fully verified against the real API and still fail at replay if its data environment is empty or toy. The problem compounds across multi-tool flows for a structural reason: tools require input values that come from the outputs of other tools, so a replay that finds no realistic records in the simulator at step 1 can't produce the realistic output that step 2 needs to consume, and the whole trajectory diverges from the one originally recorded. The simulator's starting data is itself part of the non-deterministic boundary that replay is supposed to hold fixed. Get that data wrong, and the replay is wrong, no matter how stateful or how well-verified the simulator is otherwise.
Putting it together: from a recorded production failure to a committed CI regression test
The full path from a production incident to a regression test running in CI depends on every property this piece has laid out, working together rather than in isolation; missing any one of them leaves the replay either non-reproducible or quietly misleading. A production agent failure starts the sequence: a trajectory of model calls, tool calls, and routing decisions that together produced a wrong or unsafe action. Each piece contributes something distinct to that outcome. Deterministic agent replay is only as faithful as the tool environment standing behind it, and a stateful, verified simulator is what makes that faithfulness hold up at the scale a CI pipeline demands.
