Restarts are not an edge case
Any pipeline that runs long enough will be killed mid-flight — a deploy, an OOM, a reclaimed spot instance. The Email Classifier reads a stream of messages, asks an LLM whether each one matters, and pushes the important ones to a live dashboard. The question that shaped the whole design: what happens when the worker dies between "classified" and "delivered"?
Get it wrong one way and you replay the stream on restart, double-classifying and double-alerting. Get it wrong the other way and you skip the in-flight message and lose it. Neither is acceptable for something whose job is to surface billing failures and security alerts.
A guard table instead of a distributed transaction
The fix is deliberately boring: a ProcessedEmail guard table that records a message ID once it's been handled. Layer one of the pipeline checks that table before doing anything else; an ID it has already seen is silently skipped — no AI call, no write, no broadcast.
The ordering is the clever part. The ID is marked processed after classification succeeds but before the alert is broadcast:
- Crash after the mark → the restart sees the ID and skips it. No duplicate.
- Crash before the mark → the message is simply reprocessed, safely.
No queue to reason about, no two-phase commit. Just a row that says "already handled."
Determinism is what makes reprocessing safe
That second case only works because reprocessing the same email always yields the same decision. So the classifier runs at temperature 0.0 — identical inputs produce identical outputs. Determinism here isn't only an accuracy choice; it's the property that lets "just run it again" be a valid recovery strategy. A wandering temperature would turn every retry into a coin flip.
Push, don't poll
Once a message is classified and stored, the dashboard needs to know now. Polling would mean latency, wasted bandwidth, and a server answering "anything new?" thousands of times for mostly-no. Instead, an ASGI WebSocket group broadcasts each alert the instant it's persisted. The REST API still serves history on first load; the socket carries everything after. Real-time observability, zero polling.
Filter at the source
The last decision is restraint: unimportant mail is dropped at classification — important: false means it is never stored and never reaches the UI. Filtering noise at the front keeps the database lean and the dashboard honest. Everything on screen is, by construction, worth looking at.
The lesson
"Exactly once" sounds like it demands heavy machinery. It didn't. A guard table, a deterministic model, and careful ordering bought restart-safety without a single distributed transaction. The best reliability features are the ones nobody notices, because nothing ever breaks.