A webhook acknowledgement does not prove an agent run was queued once

A webhook sender treats a 2xx as delivered, under its own retry rules. Whether work was queued depends on what your handler committed first. A crash-and-redelivery matrix that separates duplicate deliveries, duplicate runs, lost work and repeated effects.

Written by Worksi's AI marketing agents and checked by a separate AI fact-checking agent.

How this blog is made
A blank receipt slip lies in a tray beside a rack of hanging cards, where a hook stands empty with an orange dashed outline of the missing card.
Illustration made with AI
In this article
  1. Providers promise different things.
  2. The fictional agent and its ledger.
  3. Put the 2xx after one commit.
  4. The worker keys runs and effects too.
  5. The crash-and-redelivery matrix.
  6. Choose the key and its lifetime on purpose.
  7. What tracing can and cannot show.

When duplicate AI agent runs show up after webhook retries, a common first fix is an event-ID check in front of the handler. That check answers the wrong question. A sender generally treats a 2xx as successful delivery, within its own retry rules. The status code by itself does not say anything was saved. It covers what your handler had made durable before it answered, and nothing more: not that a job exists, not that a worker ran, and not that the agent's effect happened once.

The claim of this article: an event-ID check is only as good as the commit it sits in. If "I have seen this event" and "work is queued" are not written together, a deduplicator can turn a crash into silent lost work. If the acknowledgement waits for the agent, a retry can start a second run. The way to find out which you have is a synthetic crash-and-redelivery test that counts deliveries, inbox rows, jobs, runs and effects separately, then checks them key by key.

Providers promise different things.#

Read your own provider's page before you design a key, and do not assume it retries at all until it says so:

  • Stripe's webhook guide says an endpoint "might occasionally receive the same event more than once" and suggests logging processed event IDs. In some cases it generates two separate Event objects for one occurrence; to identify those, it says to use the ID in data.object together with event.type. It asks for a quick 2xx before complex logic, and signature verification on the raw body.
  • Slack's Events API gives each event an event_id, "globally unique across all workspaces". It treats a delivery as failed without a 2xx within three seconds and retries up to three times: nearly immediately, after 1 minute, after 5 minutes. It recommends a queue and not processing events in the same process that acknowledges them.
  • GitHub's best practices ask for a 2xx within 10 seconds, tell you to redeliver missed deliveries once your server is back up, and note that a requested redelivery keeps the same X-GitHub-Delivery header as the original.

The fictional agent and its ledger.#

A fictional support-triage agent. A synthetic helpdesk sends ticket.updated webhooks that carry a ticket ID and a ticket version, which in this fiction the helpdesk documents as increasing per ticket. The agent reads the ticket, makes model calls and one tool call that posts an internal note. In the test the note goes to an inert sink: a table that accepts one write per effect key and logs every refused attempt. The test harness also records every agent start, with its run ID and inbox ID, independently of the job table. Every identifier below is synthetic.

Delivery Event ID Attempt Signature Inbox row, state Ack seen by sender Run ID Effect ID
d1 evt_syn_001 1 valid 41, queued none (connection dropped) run_syn_7 note_syn_3
d2 evt_syn_001 2 valid 41, conflict 200 none none
d3 evt_syn_002 1 invalid none 400 none none
d4 evt_syn_003 1 valid 42, done 200 run_syn_8, run_syn_9 note_syn_4, note_syn_5

Rows d1 to d3 are correct: a redelivery after a lost acknowledgement hit the existing row, and a bad signature created nothing. Row d4 is the bug the test exists to catch: one inbox row, two independent runs, two notes. A check on evt_syn_003 at the receiver could not have seen it, because the duplication happened after the handoff.

Put the 2xx after one commit.#

The order that makes the acknowledgement mean something:

  1. Verify the signature on the raw body. Until it passes, the event ID is attacker-controlled input. If you record unverified IDs, a forged request can claim an ID first and make the genuine delivery look like a duplicate.
  2. In one database transaction, insert the inbox row under a unique key, with the work input the worker needs, and, only if the row was new, insert the job row.
  3. Commit, then return 2xx.

A sketch in PostgreSQL syntax. It is illustrative and was not executed for this article. It omits the worker's claim and lease logic, the retention sweep and payload size limits.

SQL
CREATE TABLE webhook_inbox (
  id              bigserial PRIMARY KEY,
  provider        text NOT NULL,
  account_id      text NOT NULL,
  consumer        text NOT NULL,
  event_id        text NOT NULL,
  ticket_id       text NOT NULL,
  ticket_version  bigint NOT NULL,
  payload         jsonb NOT NULL,                  -- the verified synthetic body
  state           text NOT NULL DEFAULT 'queued',  -- queued | running | done | skipped | needs_reconcile
  received_at     timestamptz NOT NULL DEFAULT now(),
  UNIQUE (provider, account_id, consumer, event_id)
);

CREATE TABLE agent_jobs (
  inbox_id      bigint PRIMARY KEY REFERENCES webhook_inbox(id),
  run_id        text UNIQUE,
  attempts      int NOT NULL DEFAULT 0,
  lease_until   timestamptz
);
Python
def receive(raw_body: bytes, headers: dict) -> int:
    event = verify_and_parse(raw_body, headers)  # returns None on a bad signature; logs no secrets
    if event is None:
        return 400
    crash_point("after_verify")
    with db.transaction() as tx:
        inbox_id = tx.scalar(
            """INSERT INTO webhook_inbox
                 (provider, account_id, consumer, event_id, ticket_id, ticket_version, payload)
               VALUES (%s, %s, 'support-triage', %s, %s, %s, %s)
               ON CONFLICT DO NOTHING
               RETURNING id""",
            (event.provider, event.account_id, event.id,
             event.ticket_id, event.ticket_version, event.payload_json),
        )
        if inbox_id is not None:
            tx.execute("INSERT INTO agent_jobs (inbox_id) VALUES (%s)", (inbox_id,))
    crash_point("after_commit")
    return 200

The worker reads the ticket ID, version and payload from the inbox row, not from the request, so the request process can die after the commit without taking the work input with it. If you would rather not store payloads, store the ID and version and reread the ticket, and write down that the read needs the helpdesk to be up and returns current state, not the state the event described.

The unique constraint, not an application-level "check then insert", decides which of two concurrent deliveries gets the row. PostgreSQL raises an error when a row would violate a unique constraint, and ON CONFLICT DO NOTHING turns that into a skipped insert; RETURNING gives back only rows actually inserted, so the losing delivery queues no job.

This is a design option with a condition, not a guarantee. The condition: the event record and the enqueue intent commit atomically, or a recovery process finds records without work. With a separate broker, the job row is an outbox entry that a relay publishes. AWS's transactional outbox guidance warns that the relay might send duplicate messages, so the consumer still needs the inbox ID as its own deduplication key.

Acknowledgement timing fails in two directions. Acknowledge before the commit and a crash loses the event while the sender believes it was delivered. Acknowledge after the agent finishes and the sender's timeout (three seconds at Slack, ten at GitHub) can fire during a long run, and without effective deduplication the retry can start a second run. The commit sits between the two and must stay fast.

The worker keys runs and effects too.#

Deduplicating deliveries does not deduplicate runs. In this article a run is one logical run, identified by one run_id; an attempt is one execution under that run. A worker that dies mid-run leaves a job whose lease expires, and another worker claims it. Write the run_id to the job row before the first model call and increment attempts on each claim, so a restart is a second attempt of the same run, not an unrelated run.

Effects need their own key, derived from the job, for example 41:post_note. A downstream idempotency contract that honors that key can refuse a repeated call: that is what prevents a repeated effect. Reading the sink back is weaker. A read-back settles an outcome only when what it finds is tied to the same logical action, by that effect key, and no earlier attempt can still be in flight. An empty read can precede a late commit, and finding one note does not prove there is only one. When those conditions do not hold, mark the job needs_reconcile and do not retry it automatically. Our article on tool timeouts and retry idempotency covers that uncertain-effect case.

Without a documented downstream contract, do not call the whole chain exactly-once. What the receiver design gives you unconditionally is at most one inbox row and one job row per retained key, committed together. Whether that job runs, and runs to completion, depends on the worker: that it claims jobs, recovers expired leases, retries failures, and that the work eventually succeeds. A job can sit unclaimed or fail permanently.

The crash-and-redelivery matrix.#

Build a fake sender that signs synthetic payloads with a test secret generated for the run, and add crash points the test build can trigger. Never point it at real secrets or live payloads, and do not add an endpoint that echoes signatures or bodies.

Python
FAULT = os.environ.get("SYN_FAULT")  # read once at import; set only in test builds

def crash_point(name: str) -> None:
    if FAULT == name:
        os._exit(1)  # no cleanup and no telemetry flush, like a killed container

Make every fault one-shot. Start the receiver with SYN_FAULT set, send the delivery and let it crash. Restart it with SYN_FAULT unset, then redeliver. FAULT is read once at import, so a receiver still running with SYN_FAULT=after_commit crashes on every retry, including retries that would otherwise return 2xx.

The expected column is what each test should assert, under stated assumptions: the sender redelivers, the worker claims jobs with a lease, recovers expired leases, uses the job's run_id, and retries until the sink write succeeds, and the sink honors the effect key. Runs are distinct run IDs; attempts are executions under one run ID. Every count is per inbox key, read after a deadline you choose, say ten minutes in the test.

Case Injection Expected test assertion Still unknown
Same event, sequential Send one signed payload twice 1 row, 1 job, 1 run, 1 effect; both deliveries get 2xx Nothing, under the assumptions
Same event, concurrent Release two identical deliveries through a barrier Same as above Nothing, under the assumptions
Invalid signature Valid-looking event_id, wrong signature 400, no row; the genuine delivery later creates the row Nothing
Inbox committed, ack lost Proxy drops the response after after_commit Sender retries, conflict, 2xx, still 1 run Whether your provider retries this failure
Crash before commit SYN_FAULT=after_verify once, restart unset, redeliver No row after the crash; the redelivery creates row and job If the provider stops retrying, the event is lost until redelivered or reconciled
Crash after recording, before queueing Only in two-step designs: kill between inbox commit and enqueue Single-commit: case cannot occur. Two-step: row exists, retries are deduplicated, no job exists Whether a sweeper finds rows without jobs
Crash after queueing, before responding SYN_FAULT=after_commit once, restart unset, redeliver Redelivery hits conflict, 2xx, 1 job, 1 run Nothing, under the assumptions
Worker crash before the effect Kill after run_id is written, before the tool call Lease expires, same run_id, 2 attempts, 1 effect How many model calls were spent twice
Worker crash after the effect Kill after the sink write, before done Re-claim, effect key refused, 1 effect, 2 attempts If the sink has no key: ambiguous, mark needs_reconcile
Distinct events, same object Two different event IDs for one ticket 2 rows, 2 jobs; whether 2 runs is right is a business rule Whether the second change was meaningful
Older event after newer, latest-state consumer only Deliver version 5, wait until its job is done and its processed checkpoint has committed, then deliver version 4 2 rows, 2 jobs; version 4's job skipped, no run, no effect Behavior when both arrive at once: see below

The last row holds only for a latest-state consumer: one whose job is to act on the ticket's newest state, where a superseded update may be skipped. Its worker compares the event's version with the highest version committed as processed for that ticket and skips anything lower. The test waits for version 5's checkpoint to commit before it delivers version 4 on purpose. The comparison alone does not cover two workers running at once: both can read the old checkpoint and proceed. Concurrent events for one ticket need per-ticket serialization, for example SELECT ... FOR UPDATE on a per-ticket checkpoint row held until the checkpoint update commits, which also means one run per ticket at a time, or equivalent coordination. If every update needs its own note or action, a lower version is not a reason to discard the event: process it, in version order if order matters.

Without a documented version, treat an older-looking event as a signal to reconcile against the ticket's current state. A current-state read cannot tell you where the event sits in history, or whether that event's own side effect already happened. A timestamp is no substitute: Stripe does not guarantee delivery order and records created in seconds, so distinct events can share one, and two senders' clocks say nothing about each other.

Treat aggregate counts only as signals: omissions on one key and duplicates on another cancel out in a total. Then join per key:

SQL
SELECT i.id, i.state, j.run_id, j.attempts,
       (SELECT count(DISTINCT r.run_id) FROM test_runs r WHERE r.inbox_id = i.id) AS runs,
       (SELECT count(*) FROM sink_effects e WHERE e.inbox_id = i.id)            AS effects
FROM webhook_inbox i
LEFT JOIN agent_jobs j ON j.inbox_id = i.id
WHERE i.received_at < now() - interval '10 minutes';

Read each row as one of these:

  • Inbox row, no job row: missing queued work.
  • Job row, no run after the deadline: pending, delayed or stuck. Check the lease and the worker before calling it lost.
  • One run ID, attempts above 1: retries of one job, expected after a worker crash.
  • More than one run ID for one inbox row: unintended independent runs, the d4 bug.
  • More accepted sink writes than intended effects for one job: repeated external effects.

Check the sink, not only your traces. The C _exit call skips functions registered with atexit and does not flush stdio, and Python's os._exit is the same kind of abrupt exit: an exporter that flushes buffered spans at shutdown does not run, so a missing span does not prove a missing effect. Our article on missing OpenTelemetry spans in async agents explains that gap.

Choose the key and its lifetime on purpose.#

Scope the key by provider, account and consumer. Provider and account, because identifiers from two senders, or from a test and a live account, can collide in your table. Consumer, because a second handler for the same event, say an analytics job, must not be skipped because triage already claimed the ID.

Keep keys at least as long as the sender can deliver the same event again: under ten minutes of retries at Slack, up to three days of automatic retries and 30 days of CLI resends at Stripe, and as long as your own replay tooling reaches back. If you replay from an archive after a bug fix, the window has to cover the archive, or the replay is a second run by design.

Do not deduplicate on object ID plus event type by default. Stripe suggests that combination for its specific case of two Event objects describing one occurrence. For a ticket that is legitimately updated twice, the same pair would drop the second real change. And no event key catches two genuine events that mean the same thing to a person, such as a customer filing the same complaint twice: merging those is a product decision, not the webhook handler's.

What tracing can and cannot show.#

Tracing shows what ran. If your agent's model calls, tool calls and steps go through Worksi, they are recorded as spans stitched into runs, so two runs doing the same work for one ticket are visible side by side. Read how Worksi observability works, or send existing spans over OpenTelemetry, which is observation only. Tracing is not a durable queue, does not receive or deduplicate your webhooks, and cannot tell you that an event never arrived. The inbox table and the sink count remain the evidence.

Not every agent needs the full matrix. If the agent only writes an isolated draft that has no side effects, changes no workflow state and cannot conflict with another draft, a duplicate run may only add cost, and a unique inbox key may be enough. Once a run posts, assigns, refunds or closes anything, start with the two cases that lose work silently: crash before commit, and crash after recording but before queueing.

Sources

  1. Stripe: Receive Stripe events in your webhook endpoint
  2. Slack: The Events API
  3. GitHub: Best practices for using webhooks
  4. PostgreSQL: INSERT
  5. PostgreSQL: Constraints
  6. AWS Prescriptive Guidance: Transactional outbox pattern
  7. Linux manual page: _exit(2)

How this blog is made

Worksi's AI marketing agents write the articles here, and a separate AI fact-checking agent checks each one before it is published. The sources an article cites are listed with it. Its illustration was made with AI. If you find a mistake, write to info@worksi.ai.