Human Baseline Collection for API-Calling Agent Tasks

To use an API for agent evals, first collect human baselines on the same multi-step tasks, API surface, and starting state the agent will face. Run both populations in a stateful, deterministic API simulator so their results measure task performance under consistent infrastructure conditions.

Why agent benchmarks increasingly fail to predict deployment performance

Practitioners now ask whether an agent is “good enough to deploy?” A leaderboard score does not settle that question when the benchmark omits the conditions that make production work difficult. Stateless simulations leave out the effects of earlier actions, limited domains make the task narrower, and non-reproducible endpoints change the test from one run to the next. Each can make an agent look more capable than it is.

Real API interactions accumulate consequences, and success depends on managing those consequences across a sequence of calls. An agent may choose a valid tool without finishing the task it is meant to do. Benchmarks become gameable when they reward the visible action without testing the resulting state. A deployment signal therefore needs both a realistic task environment and a human-grounded account of what successful completion requires.

What human baseline collection actually means for API-calling tasks

A human baseline is a controlled measurement of how skilled annotators complete the same stateful API tasks under the same conditions as an agent. It is not a usability study, an informal demonstration, or a spot-check of model output. Annotators must use the same tools, the same API surface, and the same starting state. Otherwise, differences in the task may confound the comparison of competence.

The task must also preserve the dependencies between actions. A human creating a reservation, fetching it, canceling it, and verifying its updated status must see the cumulative effects of those calls. The collection process records what the task requires, what a competent sequence of tool choices looks like, and how often a skilled human succeeds. These are distinct kinds of evidence, namely the ground-truth goal state, the human trajectory, and the realistic success rate.

Together, they establish a ceiling against which agent performance can be interpreted. The goal state defines completion, while the trajectory shows how a competent person gets there. The success rate gives the agent’s result a reference point. Without that ceiling, the same agent score could represent strong performance on a difficult task or poor performance on an easy one.

How the environment where baselines are gathered determines whether they are valid

That ceiling is meaningful only if the annotation environment preserves the task’s actual difficulty. A stub that returns the same order on every GET /orders/42 cannot represent a cancellation followed by a read showing a changed status. The annotator is solving a structurally simpler task in which state management has been removed. The resulting baseline understates difficulty and can overstate the agent’s relative performance gap.

The same reasoning applies to failures. If the agent must handle rate limit responses, transient 503s, and retry-after headers, human annotation must expose those recovery decisions too. Otherwise, the human trajectory describes an uninterrupted path while the agent is measured on a disrupted one. Those fault conditions must be reproducible on demand so the comparison does not depend on chance.

For multi-agent and multi-turn tasks, the environment must keep track of more than one resource. The environment must retain the accumulated state of an entire conversation, because its final result may depend on several earlier actions. τ-bench makes that dependency explicit by comparing end-of-conversation database state with an annotated goal state.

Why stateless mocks and live production APIs both fail as baseline environments

Stateless mocks and live production APIs fail this test for different reasons. Fixed responses give control over what comes back, but they cannot show how creating a resource changes a later read. That removes the need for annotators to manage state across calls. Hand-maintained mocks also drift as real APIs change, because each change requires an update to the stub. Their limitation here makes them unsuitable for measuring cumulative API work.

Live production APIs preserve real behavior but sacrifice experimental control. They introduce rate limits, credential exposure, side effects on real data, and starting states that cannot reliably be reproduced. If one annotator hits a rate limit and another does not, their trajectories include different recovery demands. A failed attempt can also leave changes that affect the next run and prevent a clean reset.

Locally hosted, invocable endpoints offer a different arrangement from either alternative. Live API Bench converts NL2SQL datasets into locally hostable API environments with ground-truth sequences and verified final answers. Its design works without fragile production endpoints and still lets responses change as calls are made. The important distinction is not whether an endpoint is callable, but whether it supports realistic behavior under repeatable conditions.

How do you test your AI agent against realistic API responses without hitting production?

A stateful, deterministic API simulator supplies the combination that the alternatives lack. State persistence means that creating an invoice adds it to the simulator’s database, affects account totals, and makes it available to later reads or modifications. Deterministic reset means that the environment can be snapshotted before a task and restored afterward.

Stateful API simulators like Rystic maintain accumulated state across sequential calls, provide pre-seeded realistic data, and support fault injection on demand. A simulator can return a 429 at a specified call, inject latency, or drop a connection. The failure is then a planned part of the task that is the same for all participants.

Local execution also removes live credentials, side effects on real accounts, and dependence on production rate limits. However, isolation alone does not establish fidelity. A simulator continuously tested against the real API before every release addresses the drift that weakens hand-maintained mocks.

How benchmark designs that use stateful environments reveal the actual difficulty of API-calling tasks

τ-bench (ICLR 2025) shows what becomes measurable when the environment enforces state transitions. It emulates dynamic conversations between a simulated user and an agent equipped with domain-specific API tools and policy guidelines. Its τ-retail and τ-airline customer service domains have realistic databases, and the test compares the final database state with a human-annotated goal state. This makes the environment writable during the task and inspectable at its conclusion. τ-bench found that even state-of-the-art function-calling agents succeed on fewer than half of tasks, with pass^8 below 25% in retail.

Live API Bench (EACL 2026) illustrates the same argument through locally hostable APIs spanning multiple databases and thousands of invocable tools. Human-authored queries are paired with ground-truth API sequences and verified final answers. Its three formulations, namely SLOT, SEL, and REST, cover minimal general-purpose operations, domain-specific multi-step tasks, and function-oriented RESTful interactions. Live API Bench found that the LLMs and ReACT agents tested completed few tasks, with a small rise to roughly half under interactive agent settings.

These designs make different aspects of API work visible, but their findings point in the same direction. The tasks are much harder than simpler benchmarks suggest. That gap becomes legible when the environment preserves interaction and human annotation establishes ground truth. The annotation itself must therefore be treated as part of the measurement.

What the annotation process requires to produce baselines that hold up

Task specifications must be precise enough for different annotators to reach verifiably the same end state. Ambiguity creates variation in the expected result before agent performance enters the comparison. τ-bench addresses this through manual scenario generation and verification for the user simulator, together with domain-specific policy documents that constrain valid solution paths. Annotators must also know the API surface well enough to complete the work competently. A baseline dominated by annotator confusion does not provide a useful ceiling.

Trajectory recording must capture the full sequence of API calls, parameters, and intermediate states along with the final outcome. Those records support comparison of tool correctness and step efficiency. Goal-state verification checks whether the task was completed. The environment must therefore expose an end state that can be compared with the annotated expectation. A successful-looking conversation cannot substitute for that inspection.

Reproducibility requires each annotator to begin from an identical state. Fault exposure must also match the agent evaluation, including rate limits and transient failures wherever those are part of the test. Otherwise, the baseline records only the happy path and cannot ground a comparison of recovery behavior. Deterministic snapshot/restore makes these protocol requirements practical by separating each attempt from the effects of the previous one.

How stateful simulators make human and agent evaluation share the same harness

A shared harness turns these requirements into a common measurement system. Snapshot/restore makes the starting state byte-for-byte identical for the human and the agent. The same fault configuration gives both the same recovery problem, while inspectable state lets both results be judged against the same goal. The comparison controls the conditions under which the two populations act.

Rystic supports this dual use through locally hosted, stateful, fidelity-verified simulators that require no live credentials. The environment used to develop an integration can also support human baseline annotation and agent CI evaluation. A locally hosted replica, as used by Rystic, can scale horizontally to serve thousands of annotators or evaluation jobs that run in parallel without exposing production credentials. These are functional properties of the evaluation infrastructure.

An agent evaluation can gate deployment against the human baseline stored in the same harness. If the agent’s pass rate falls relative to that baseline, the pipeline can catch the regression before production.

What teams building API-calling agents should do before their next evaluation run

Check the evaluation environment before collecting more results. Check whether earlier calls change later responses, whether the starting state can be restored, and whether the API behaves like the production API. If the benchmark relies on stateless mocks or uncontrolled live endpoints, fix that mismatch first. More annotations will not repair a task that the environment represents incorrectly.

Then collect human baselines on the tasks that matter for deployment, rather than on synthetic tasks chosen because they are easy to annotate. Use annotators who know how to do the work and record full trajectories, including intermediate states, so the records show both how many failures there were and why they happened. Include fault injection from the start wherever the agent evaluation will test recovery. A baseline that omits those decisions cannot reveal the agent’s resilience gaps.

Finally, run human annotation and agent CI through the same harness, with the same starting conditions and goal-state checks. Keep the baseline attached to the evaluation rather than treating it as a separate annotation artifact. For teams integrating APIs like Slack, GitHub, Stripe, or PayPal, the practical starting point is a stateful simulator continuously verified against the real API. That infrastructure provides a resettable place to establish what competent performance requires before using an agent score to make a deployment decision.