The pipeline that reported success while doing nothing
An identification outage that lasted thirteen days, three failed fixes, and what we changed about how we watch things.
The symptom
A customer's dashboard said identification had stopped yesterday. The truth was thirteen days, and every system involved was reporting healthy.
tracking_events 8,191 in 24h, most recent seconds ago ← pixel fine
liveintent_resolutions ~1,200/day, most recent minutes ago ← vendor fine
signal_account_visitors last created 2026-08-29 13:43 ← nobody new
unified_profiles last created 2026-08-29 13:43 ← nobody new
Traffic arriving, resolutions arriving, and not one new identified person in thirteen days.
Three things that were all green
The schedule. The hourly job was not paused, was firing on time, had nothing stuck, and showed ten recent actions. Looking at the scheduler told you everything was fine.
The updated_at column. Rows were being written every hour, so any monitor
watching row freshness saw a live system. Those writes were a different job —
an hourly intent recompute touching existing rows. updated_at moved;
created_at had not moved in thirteen days.
The workflow status. In an earlier round of the same outage, the workflow
caught its own exception and returned a result object with
status='failed'. Temporal saw a COMPLETED workflow. A caught failure
returned as a value is a completion.
The actual cause
Every hourly run failed in two seconds. The error surfaced as a circuit-breaker
trip, which was itself a mask — the first attempt raised the real error, the
second returned CircuitOpenError, and only the second was reported:
CheckViolationError: new row for relation "liveintent_ingestion_runs"
violates check constraint "liveintent_ingestion_runs_status_check"
DETAIL: Failing row contains (…, no_data, 0, 0,
discovery found no dated partitions under the configured prefix).
The job had a branch for "the vendor delivered nothing", which recorded
status='no_data'. The CHECK constraint permitted running | success | partial | failed. Every one of those writes was rejected, the retries exhausted, the
breaker opened, and the workflow died on its bookkeeping call — before the
step that turns resolutions into people.
SELECT DISTINCT status returned {success, failed}. The value no_data had
never once landed.
Each fix planted the next one
| What broke | Duration | |
|---|---|---|
| 1 | Discovery hit an S3 AccessDenied; the workflow caught it and returned a value, so the scheduler saw COMPLETED |
3d 7h |
| 2 | The fix for #1 recorded the failure with an empty date; the recorder died on date.fromisoformat('') and tripped the breaker |
4 days |
| 3 | The fix for #2 added the no_data branch, and nobody widened the constraint |
13 days |
All three were in the same recorder. The common factor was not carelessness: the status vocabulary had two homes — string literals in the job, a CHECK constraint in SQL — and no test read both.
The design error underneath
Recording what happened must not be able to prevent what happens.
The write that logged the run was awaited on the critical path, ahead of the step that produced the product's entire value. A logging concern was a hard dependency of the business. It took identification down twice before anyone named it.
What we changed
- One vocabulary, one home. The statuses are declared once in code, and a test asserts that set equals the CHECK constraint as written in the migration. Adding a status without a migration now fails a test instead of production.
- The recorder cannot fail the run. It logs loudly and returns. Safety comes from the absence of the row: a monitor watches run freshness with a three-hour budget and goes stale precisely when this breaks.
- The banner measures the output, not the job. It had read "when did our ingest last run", which is a different fact from "when did we last identify someone" — and a crashed run refreshed it, which is how thirteen days displayed as one. It now reads the last actual identification, on a threshold derived from the measured gap distribution (p50 1.0h, p90 2.0h) rather than chosen.
What to take from it
Ask what would make the number move. A monitor that cannot go red is
decoration. Before trusting one, work out which specific failure it would catch
— and check that a failure cannot refresh the thing it measures. Our banner,
our updated_at, and our workflow status all had that defect in different
forms.
Distinguish "the job ran" from "the job did something." They are different questions with different right answers, and fusing them lets a healthy job speak for a dead pipeline.
A test that reads one side of a contract is not a test of the contract. The constraint and the code each said something true about themselves. Nothing read both.