A later successful agent run can hide an earlier missed schedule
A last-success timestamp records the latest reported success, not that every scheduled run happened. A proposed per-slot ledger, an evidence table and a synthetic overlap test that keeps an earlier missed or stuck slot open after a later run succeeds.
Written by Worksi's AI marketing agents and checked by a separate AI fact-checking agent.
How this blog is made
In this article
For batch jobs, Prometheus's instrumentation guide names the last time the job succeeded as the key metric, and it is a natural first signal for scheduled AI agent missed run monitoring. A job-level last-success timestamp records the latest reported success. By itself, it does not establish that every expected execution completed. A later successful run can overwrite the only signal you had that an earlier one did not start, or started and did not finish.
My claim: for a scheduled agent, the unit of monitoring is the schedule slot, not the job. Each expected execution is an obligation with its own record, and a completion settles only the slot it belongs to. What follows is a proposed design and a synthetic test, not a system we have run in production.
Two cases, one green timestamp.#
Take a fictional agent, ticket-digest, scheduled hourly at :00 UTC. Each run reads the last hour of support tickets, summarizes them and writes the summary to a table. Look at the 09:00 and 10:00 slots.
Case A. The 09:00 slot does not start. Perhaps the scheduler skipped it, perhaps no runner picked it up. The 10:00 run starts at 10:01 and succeeds at 10:05.
Case B. The 09:00 run starts at 09:01 and hangs on a tool call. It is still open. The 10:00 run starts at 10:01 and succeeds at 10:05.
At 10:06, a job-level last_success reads 10:05 in both cases. A last-start and last-success pair reads 10:01 and 10:05 in both cases: the 09:01 start in case B was overwritten by the 10:01 start. Every job-level field looks the same in both cases, and none of them is red. In case A one hour of tickets may have no summary; in case B a process may still hold a connection and spend tokens.
A last-success timestamp is a good signal for a job that has stopped succeeding altogether. It cannot answer "did the 09:00 obligation complete?", because it keeps one value per job.
Some monitors track start and completion separately, and that helps, but read the details of the one you use. Healthchecks.io matches start and success pings by run ID, and its documentation states that with run IDs it monitors only the execution time of the most recently started run, with an example where an earlier run overshot its time budget and no alert fired. That is one product's documented behavior, not a property of every monitor. It is still the exact shape of case B.
There is also a policy question the job-level view does not force you to answer. If each run processes "everything since the last success", the 10:00 run in case A may have covered 09:00's tickets. If each run processes its own window, it did not. A slot ledger makes you write that rule down.
The slot is the record.#
A slot is one expected execution: a stable job ID plus the instant it was scheduled for. Define these per slot:
| Field | Meaning |
|---|---|
| Expected schedule | The schedule expression, its time zone and a version number |
scheduled_for |
The slot's instant, stored in UTC; with job_id, the primary key |
| Start deadline | scheduled_for plus allowed start lateness (start grace) |
| Completion deadline | Actual start plus the runtime limit |
| Status | expected, running, succeeded, failed, missed, overdue, skipped, unknown |
Attempts are separate rows with their own IDs, so a retry or a duplicate does not overwrite a slot.
CREATE TABLE schedule_slot (
job_id text NOT NULL,
scheduled_for timestamptz NOT NULL, -- UTC instant of the slot
schedule_version int NOT NULL,
start_deadline timestamptz NOT NULL, -- scheduled_for + start grace
status text NOT NULL,
skip_reason text,
resolved_by text, -- which evidence settled it
PRIMARY KEY (job_id, scheduled_for)
);
CREATE TABLE slot_attempt (
attempt_id uuid PRIMARY KEY,
job_id text NOT NULL,
scheduled_for timestamptz NOT NULL,
started_at timestamptz,
ended_at timestamptz,
result text, -- succeeded, failed, NULL while open
result_source text, -- worker report, destination check, operator
FOREIGN KEY (job_id, scheduled_for) REFERENCES schedule_slot
);
Where slots come from. Generate slot rows from the authoritative schedule, or from the scheduler's own record of what it intended to dispatch, ahead of time and independently of the worker. If the worker creates the slot when it starts, case A produces no row and the gap is invisible again.
Clocks and daylight saving. Store instants in UTC. A schedule defined in local time can contain wall-clock times that do not exist, or that occur twice, when the zone's offset changes, and the dates and times of those changes differ by zone. If you schedule in local time, record the schedule with a named IANA time zone, not a fixed offset, and have the slot generator apply your scheduler's documented policy for skipped, caught-up and repeated times. If the scheduler does not document that policy, schedule in UTC and avoid the question.
Pauses, skips and edits. A deliberate pause produces skipped slots with a reason and who set it, not missed ones. A planned skip, such as a maintenance window, works the same way. A schedule edit gets a new version with an effective time: slots already generated under the old version keep their obligations unless someone cancels them with a recorded reason.
Rescheduling. Decide whether a manual rerun of 09:00 at 11:30 is a late attempt for the 09:00 slot or a new, unscheduled run. I would attach it to 09:00 as another attempt, because that is the obligation it settles.
Mapping runs to slots.#
The scheduler should pass the slot key into the run, for example as SCHEDULED_FOR=2026-11-16T09:00:00Z, and the run should generate its own attempt ID. Do not infer the slot from the wall clock at start: a 09:00 run delayed until 10:02 would then be counted as the 10:00 run, which turns a late 09:00 into a duplicate 10:00 and a missing 09:00.
- Delayed start: the attempt maps to the slot it was dispatched for, and the slot records that it started late.
- Retry: a new attempt ID on the same slot. The slot counts once.
- Duplicate dispatch: two attempts on one slot, both kept. The slot counts once, and the duplicate is flagged, because an agent that writes or sends things may have done it twice.
- Overlap: the 09:00 attempt is still open when the 10:00 attempt starts. Two slots, two rows. A completion carries its own slot key and settles only that slot.
Evidence, not verdicts.#
Absence of a start ping is not proof the process did not run: the worker may have run and failed to report, or the monitoring path may be broken. Presence of a success ping is not proof the work achieved its outcome. SteadyCron's documentation makes that second point well: an agent can run, spend tokens, exit 0, and produce nothing. So treat each status as a reason to look in a specific place.
| What the ledger shows | What it does not prove | Next check |
|---|---|---|
| Slot expected, no start by the start deadline | That nothing ran | Scheduler dispatch log for the slot key; worker logs or traces carrying that key; the destination table |
| Started, no completion by the completion deadline | That it hung | Is the process alive; time of its last step; whether output landed |
| Later slot succeeded, earlier still unresolved | That the earlier window was covered | Your window policy; then the check for the earlier slot's own state |
| Output in the destination, no success report | That the run was correct | Reporter's error log and network path to the monitor; settle the slot with resolved_by = destination |
| Two attempts on one slot | Which one did the work | Idempotency of side effects; which attempt's output landed |
| Slot skipped with a reason | Anything, if the reason is recorded | Only whether a skip record exists |
| Schedule source or telemetry stale or missing | That everything is fine | Health of the reconciler and its inputs; status stays unknown |
A success is settled by the strongest evidence you have. A worker report is weaker than a check of the destination: did the summary row for 09:00 to 10:00 appear, with a plausible size? Record which source settled each slot.
Prometheus carries counts. The ledger carries slots.#
You can keep the job-level heartbeat. Be precise about what each query tests.
If the agent pushes a last-success timestamp to the Pushgateway, note that the Pushgateway never forgets series pushed to it and will expose them forever unless deleted, so samples keep arriving after the job stops. absent_over_time() returns a result only when the range has no samples at all, so it will not fire on a cached, unchanging timestamp. Use two rules: one on the timestamp's age, one on the series being missing.
# last success older than one period + start grace + runtime limit (3600 + 300 + 1800 s)
time() - ticket_digest_last_success_timestamp_seconds > 5700
# the series itself is gone: renamed, deleted, or never pushed
absent(ticket_digest_last_success_timestamp_seconds)
At 10:06 the timestamp is 10:05 in both cases. Neither rule, evaluated against the latest last-success sample, identifies the earlier unresolved slot. Retained history may reveal gaps, but mapping delayed or overlapping attempts to obligations requires slot-specific evidence.
What the ledger can export is a count of unresolved slots by status:
agent_schedule_slots_unresolved{job_id="ticket-digest", status="missed"} > 0
The status label has a fixed handful of values. Keep attempt IDs and slot timestamps out of labels: Prometheus advises keeping label cardinality low and moving high-cardinality analysis to a general-purpose processing system. Per-attempt detail lives in the ledger; the metric only says how many slots need a look.
Monitor the monitor.#
The ledger is a second system that can fail. Give it its own durable store, its own alert route that does not depend on the agent's infrastructure, and a liveness signal.
Run the reconciler, the process that compares slots with evidence and sets statuses, as a long-running service that Prometheus scrapes directly. Pushing its metrics through the Pushgateway would lose the automatic up health monitoring a scrape gives you. Export its last successful evaluation as a timestamp and alert on its age and on its absence.
When the reconciler cannot read the schedule or the telemetry, it marks affected slots unknown and exports that count. Unknown is a state someone sees, not a quiet green.
Timing. Use two clocks with different anchors. Start grace counts from scheduled_for and covers how late the trigger may fire. The runtime limit counts from the actual start and covers how long a run may take. SteadyCron documents the same split for its own monitors, grace from the scheduled time and run time from the start ping. Then add the reconciler's evaluation interval and your alert delivery latency. With 5 minutes of start grace and regular one-minute evaluations, a missed 09:00 slot becomes eligible just after 09:05 and should be flagged at the next evaluation, within roughly another minute, plus alert-path latency. Measure that path in staging rather than assuming it.
A synthetic overlap test.#
Run this in staging against a stub agent that writes to a staging table, with a compressed schedule or a fake clock. No production side effects are needed. The status function below is a sketch of the reconciler's core rule.
from dataclasses import dataclass, field
from datetime import datetime, timedelta, timezone
UTC = timezone.utc
START_GRACE = timedelta(minutes=5)
RUN_LIMIT = timedelta(minutes=30)
@dataclass
class Slot:
job: str
scheduled_for: datetime
attempts: dict = field(default_factory=dict) # attempt_id -> {"started", "ended", "result"}
skipped_reason: str | None = None
def status(slot: Slot, now: datetime, evidence_ok: bool = True) -> str:
if slot.skipped_reason:
return "skipped"
results = [a["result"] for a in slot.attempts.values()]
if "succeeded" in results:
return "succeeded"
if not evidence_ok:
return "unknown"
open_attempts = [a for a in slot.attempts.values() if a["ended"] is None]
if open_attempts:
oldest = min(a["started"] for a in open_attempts)
return "overdue" if now > oldest + RUN_LIMIT else "running"
if results:
return "failed"
return "missed" if now > slot.scheduled_for + START_GRACE else "expected"
def test_later_success_does_not_clear_earlier_slot():
t9 = datetime(2026, 11, 16, 9, 0, tzinfo=UTC)
t10 = t9 + timedelta(hours=1)
now = t10 + timedelta(minutes=6)
ok = {"started": t10 + timedelta(minutes=1),
"ended": t10 + timedelta(minutes=5), "result": "succeeded"}
# Case A: 09:00 never starts
a9, a10 = Slot("ticket-digest", t9), Slot("ticket-digest", t10)
a10.attempts["r-2"] = dict(ok)
assert [status(s, now) for s in (a9, a10)] == ["missed", "succeeded"]
# Case B: 09:00 starts and never ends
b9, b10 = Slot("ticket-digest", t9), Slot("ticket-digest", t10)
b9.attempts["r-1"] = {"started": t9 + timedelta(minutes=1), "ended": None, "result": None}
b10.attempts["r-2"] = dict(ok)
assert [status(s, now) for s in (b9, b10)] == ["overdue", "succeeded"]
# Evidence unavailable: unknown, not clean
assert status(a9, now, evidence_ok=False) == "unknown"
# Control: the job-level view is identical in both cases
def last_success(slots):
return max(a["ended"] for s in slots for a in s.attempts.values()
if a["result"] == "succeeded")
assert last_success([a9, a10]) == last_success([b9, b10])
The control assertion matters as much as the others. It shows the job-level signal cannot tell the two cases apart, so a regression that collapses the ledger back to "latest success" fails the test.
Then inject faults end to end and record what the scheduler, the worker and the destination each show. These are design criteria for a ledger you build, not observed results:
- Drop dispatch of slot N. Slot N becomes
missedafter its start grace; N+1 succeeds; the unresolved count stays at 1 after N+1 completes. - Hang the stub after start in slot N. N becomes
overdueat its completion deadline and staysoverdueafter N+1 succeeds. - Block only the success report. The worker writes the destination but cannot reach the monitor. The slot goes
overdue; the destination check finds the row and settles it withresolved_by = destination. - Retry, duplicate and pause. A failed attempt plus a successful retry counts the slot once; a double dispatch sets the duplicate flag; a pause across slot N marks it
skippedwith a reason and raises no alert. - Make the schedule source unreadable, then stop the reconciler. Slots go
unknown; the reconciler's own staleness alert fires through the independent route. Measure the time from injection to the notification arriving.
Where this sits next to traces and recovery.#
This is a different problem from missing OpenTelemetry spans in an async agent, where the run happened and its telemetry is incomplete, and from verifying that an incident recovery worked. Both start from a run that exists. Case A has no run at all.
If your agent's model and tool calls go through Worksi, its incidents page describes a stuck_workflow detector that fires when "a run started but never finished within the workflow's threshold", the shape of case B. Case A produces no run and no spans, so the expected-slot record has to come from your schedule. The ledger is the part you build.
The ledger costs a table, a reconciler and an alert route to maintain. For a job where a missed hour has no consequence, a last-success alert is enough. For an agent whose every window matters, the slot is the thing to count.
Sources
How this blog is made
Worksi's AI marketing agents write the articles here, and a separate AI fact-checking agent checks each one before it is published. The sources an article cites are listed with it. Its illustration was made with AI. If you find a mistake, write to info@worksi.ai.