Verify AI agent recovery: no traffic is not a passing test

Zero failures out of zero calls is not a passing test. Yet that is often what a quiet alert reports after an AI agent incident. Someone pauses the agent, fixes a dependency or switches the model, the alert stops firing, and the question becomes whether the agent has recovered. A quiet alert tells you the detector stopped firing. It does not tell you whether the failing component carried any traffic, whether the metric still exists, or whether the traffic that arrived looked like the traffic that failed.

The claim of this article: a recovery verdict is a statement about counted, completed traffic after a marked action. After every action you should be able to pick one of four verdicts and write it down: contained, not recovered, insufficient evidence, or recovered on the measured condition. Each has a next check. None of them proves the action caused the change.

Converra's incident response runbook already covers containment, deployment markers and verdicts from live traffic, with its own product rules, such as at least 15 conversations after the change with zero recurrences before it calls a fix verified. Those are its rules for behavioral failures, not a standard. What this piece adds is a matrix that also handles the telemetry itself going missing.

A hypothetical incident#

Everything in this example is invented for illustration. It is not a customer story.

An order-support agent calls a tool, lookup_order, which wraps an internal orders API. At 13:40 UTC the orders API starts rejecting the tool's credential after a rotation. The tool-failure detector fires: in the 30 minutes before 14:05, 38 of 412 lookup_order calls ended in error (illustrative figures).

The on-call engineer pauses the agent at 14:05, restores the credential at 14:18 and resumes the agent at 14:22. Three hours later a teammate switches the agent's model for an unrelated reason. Over the afternoon, several signals arrive that look like good news. Only some of them are.

Write five things down before you look at a chart#

1. The affected component and the detector. Here: the lookup_order tool, and the tool-failure condition as it fired. Verify against the same condition, measured the same way. A different metric that happens to be green is a different test.

2. The action marker. The time of each action and what was serving after it: agent version, prompt version, model id, tool version, and the configuration that holds the credential. Without a marker you cannot split before from after, and the model switch three hours later quietly becomes part of your "after".

3. Numerator and denominator. Failed lookup_order calls over completed lookup_order calls. Not over all agent requests, and not over runs that never reached the tool. Calls refused because the agent was paused do not belong in the denominator: they did not exercise the tool.

4. The window. Set it before you look, and make it long enough to contain enough calls of the failing kind.

5. The traffic mix. Which intents, tenants, regions or input shapes produced the failures. If the failures sat in refund conversations and the window after the action holds only order-status questions, the window says little about refunds.

There is no universal sample size. One rough way to reason about your own count is the binomial model. If each call fails independently with an unchanged probability p, n clean calls in a row happen by chance with probability (1 − p)^n. At the earlier observed rate of about 9% (38 in 412), that is about 38% for 10 clean calls and about 5% for 30. Failures that cluster by input, tenant or time break the assumption, so treat these numbers as an illustration, not a threshold.

Why quiet is not clean#

A pause can remove the denominator. If the pause is enforced on the relevant execution path, lookup_order receives no newly admitted calls, and a tool-failure detector sees no failures. Do not assume that from the pause command, and classify target calls by when they were admitted, not when they completed.

• In-flight at the pause. Calls admitted before the pause can finish after it. A pause that refuses the next step does not necessarily cancel a tool call already running. Count these separately up to drain complete, the moment the last pre-pause call finishes. They do not show that the pause failed.
• Admitted after the pause took effect. A new call on a path you expected to be blocked shows the pause was not enforced there: a worker that does not check the pause, a queue that was already dispatched, or a path that calls the orders API outside the agent.
• Unplaced. A completion whose start you cannot date to either side needs investigation before you write contained.

Even an enforced pause says nothing about the credential.

A missing series can send a message that reads as recovery. Grafana's guide to missing data separates three cases: a connectivity error, where the query fails; No Data, where the query succeeds but returns nothing; and Missing Series, where some series that reported before disappear while others still report. Missing data does not satisfy the original threshold condition: in both the No Data and Missing Series cases, the guide says, the alert won't fire unless you configure it to handle them.

The two cases then behave differently. For No Data, Grafana's default is a separate DatasourceNoData alert, with its own alert name and without all the labels of the original instances, so check that your routing delivers it to whoever is on call. For a missing series, Grafana keeps the last state for two evaluation intervals, then adds the annotation grafana_state_reason: MissingSeries, moves the instance to Normal and sends a resolved notification if it was firing. The guide describes that as an alert that "wasn't resolved by recovery but evicted because the series data went missing".

Map that onto the incident. Suppose the error metric has one series per tool, and the paused agent emits nothing for lookup_order while other tools keep reporting. That is a missing series, and the on-call channel can receive a resolved notification for an incident nobody fixed. The same guide shows a way to catch disappeared series: a present_over_time query over a long range, unless the same query over a short range. Before you read a resolved message as recovery, check its state reason and check that the denominator series exists after the marker.

A telemetry outage is the same problem, quieter. If spans or metrics stop arriving, rates may be absent or stale, and some query or display settings can make the gap look like zero failures. Check telemetry availability and the denominator rather than treating silence as a clean rate.

The verdict matrix#

Implementation-neutral: replace the tool, detector and counts with your own.

Two rows deserve a closer look.

The model switch row. In the hypothetical, the window after 14:22 is measured on the old model, and the window after the switch on the new one. A clean window on the new model says the condition held for the new model. If the new model calls lookup_order less often, or for different intents, the count for the failing slice can shrink to nothing while the rate looks perfect. That is the denominator problem again, caused by a change instead of a pause.

The row where the failure stopped first. Change the hypothetical. Suppose the orders team rolled back the credential rotation at 13:58 on their own, and the 61 lookup_order calls that completed between 13:58 and the 14:05 pause all succeeded (illustrative figures). The failure was gone before the first action, so the pause, the fix and the resume get no credit, even though every window after them is clean. You only see this if you count completed calls in the minutes just before the first action.

The same trap hides in the original timeline. Had the orders team rolled back at 14:15, while the agent was paused, no target traffic would show when the failure stopped, and the clean window after 14:22 could not separate their rollback from the 14:18 fix.

A synthetic pass tests its scenario#

Replaying the failing call is a good check that the restored credential works. It exercises one input, from wherever the replay ran, with that environment's credential and network path. It does not show that production traffic reaches the fixed dependency, that other inputs work, or that the agent still chooses to call the tool at all.

If you rely on an automated verification check, whether synthetic calls count toward it depends on that check's rules. Read them rather than assume. Keep the replay in the record as what it is: the scenario passes.

Observation after an action is not causation#

A clean window after an action shows no recurrence of the measured condition in that window. It does not establish when recovery began or that the action caused it. Reserve "recovered after" for cases with evidence that the failure was present up to the action, and do not write "fixed by" on a clean window alone.

To narrow the gap, record every other change in the window: deployments, prompt edits, provider model updates, dependency changes by other teams. A freeze on prompt changes does not stop the dependency or model changes. If causation matters, for a postmortem or a runbook you want to trust next time, you need the failure present right up to the action, nothing else changed in between, and ideally a reproduction where removing the fix brings the failure back.

The detector's condition is narrower than the outcome#

The tool-failure condition counts errors. A lookup_order call that returns a success status with an empty body, another customer's order or stale data passes it. "Recovered on the measured condition" means exactly that, and two kinds of check need evidence the detector does not have.

Tool-result checks. Does the result match the request? The returned order id equals the requested one, the required fields are present, and the record is recent enough for the task. These can run on captured inputs and outputs if your capture settings keep them, or as a validation step inside the agent.

Real-world outcome checks. Did what the agent said happened actually happen? The refund exists in the payments system. The ticket closed and stayed closed. The customer did not contact support again about the same order within a follow-up period you set in advance, for example seven days (an illustrative choice), across the support channels whose records you can query. Say which channels you checked.

Both kinds need authoritative evidence tied independently to the intended request: a read from the system of record keyed on what the customer asked for, not on the target the agent chose. That evidence can be recorded in a trace as its own span. A success status, or an unchecked read-back of the order the agent picked, is not enough. We covered the green-span version of this problem in our article on tool calls that succeed with the wrong outcome.

Resolution policy is your choice#

A quiet alert does not compel a verdict, and a verdict does not compel closing the incident. Decide in advance which verdicts allow a resolve. Some reasonable policies, each a choice rather than a rule:

• Resolve only on recovered on the measured condition, with the tool-result check passing.
• Resolve on contained when the agent stays paused on purpose, with a follow-up to fix and resume.
• For a low-traffic agent, keep the incident open at a lower severity until the count is reached, or resolve with "insufficient evidence" written down and a dated recheck.

Whatever the policy, the record should say which verdict you chose and on what evidence. A short block is enough:

Incident:     tool failure, lookup_order (hypothetical)
Detector:     share of completed lookup_order calls ending in error
Before:       38 / 412 failed, 13:35–14:05 UTC
Just before first action: <failed> / <completed>, <window>
Actions:      14:05 pause; 14:18 credential restored; 14:22 resume
Containment:  in flight at pause: <n> completed, drain complete <time>
              admitted after pause took effect: <n>
              completions not yet dated to either side: <n>
Serving:      agent <version>, model <model id>, tool <version>
Window:       14:22–<end>, completed lookup_order calls only
After:        <failed> / <completed>; mix: <intents compared with before>
Other changes in window: <none recorded | list>
Verdict:      <contained | not recovered | insufficient evidence |
               recovered on the measured condition>
Not checked:  tool-result match, refund in payments system
Next check:   <what, when, who>

Where Worksi fits#

Worksi records this kind of verdict on its incidents. A person runs each recovery action with a recorded reason; nothing runs automatically. A pause makes the gateway refuse the agent's model calls with 423 agent_paused, and an app that calls the SDK's pause check between steps stops before the next one. The docs describe refusing the next call, not cancelling a tool call already running, so the in-flight split above still applies.

From the Startup plan, after a fix, Worksi re-checks the detector's own condition on new, completed traffic of the failing component. The verification window defaults to 10 minutes for tool failures and can be tuned per detector. Blocked and refused calls never count as traffic. Too little traffic or unavailable telemetry yields Could not verify.

The other verdicts are Verified, which needs enough new traffic and the failure present before the first action; Recovered before the action; and Not recovered. After a pause or stop it reports Contained when no model call of the target got through, or Not enforced when one did. A person can confirm or reject a verdict with a note, and the wording is "recovered after", not "fixed by".

The limits are the ones in this article. It checks the detector's condition, not tool results or real-world outcomes. Verification is not included on the free Developer plan. Enforcement, including a pause, applies only to traffic through the gateway, so a tool path outside it is yours to count. Read how incident verification works.