- TL;DRwhat you say if they cut you off
- Exactly-once delivery is not something you can guarantee, so engineer exactly-once effects through idempotency: turn every write into a deterministic upsert keyed on a stable business/event key (MERGE or insert-overwrite-by-partition), so reprocessing identical data lands on the same result. Deal with late data using event-time windows plus a watermark that caps how long you wait, and checkpoint progress so a retry p…
- HOW TO APPROACH ITthe order, and the clarifying questions
- Fix the framing up front: distributed systems deliver at-least-once, so the aim is idempotent processing, not some magical exactly-once. Then walk the three mechanisms in sequence: idempotent writes, partition/window scoping, and watermarking for late events.
- THE FOLLOW-UPwhat they probe next, with the strong reply
- "Why not rely on exactly-once delivery?" It is fragile and usually holds only within one system; idempotent writes survive retries, replays, and backfills regardless.
- THE MISTAKEwhat sounds right and loses the offer
- Blind INSERT/append, so a retry double-counts. Processing on wall-clock time, so late events land in the wrong window or are lost. No watermark, so windowed state grows unbounded. Advancing the source offset before the write commits, losing data on a crash between the two.