Overview
Delivery guarantees describe how many times a message may be processed. At-most-once: messages may be lost but never duplicated (ack before processing). At-least-once: messages are never lost but may be duplicated (ack after processing). Exactly-once: each message affects the result exactly one time.
True exactly-once delivery over a network is impossible in general, because a lost acknowledgment makes the sender retry. What systems actually provide is exactly-once processing: at-least-once delivery combined with idempotent or transactional processing, so duplicates have no extra effect. Kafka's transactions offer this within Kafka (read-process-write).
At-most-once is dropping it in a mailbox and hoping. At-least-once is sending it again until you get a receipt, so the recipient might get two copies. Exactly-once is sending until you get a receipt, with the recipient throwing away duplicates by checking a reference number.
When to use it
- Designing any messaging pipeline.
- Payments, billing, and inventory where duplicates matter.
- Metrics and logs where occasional loss may be acceptable.
- Interview questions about reliability semantics.
Where it shows up in interviews
Recognize it when: what happens if a message is lost or duplicated?
- Design a payment event pipeline
- Design a metrics collection system
Recognize it when: count or charge exactly once.
- Design ad click billing
- Design a ledger from events
Where it is used in real software
Idempotent producers plus transactions make consume-transform-produce pipelines exactly-once within Kafka.
Standard is at-least-once; FIFO provides exactly-once processing within a deduplication window.
Consistent snapshots of state and offsets give exactly-once state updates; sinks need transactions or idempotence.
Key terms
- At-most-once
- Zero or one time; possible loss.
- At-least-once
- One or more times; possible duplicates.
- Exactly-once processing
- Effect applied once, via dedupe or transactions.
- Acknowledgment timing
- Before processing = at-most-once; after = at-least-once.
- Deduplication
- Recognizing repeated messages by ID.
How it works, step by step
- 1Pick the requirement
Can you lose messages? Can you tolerate duplicates?
- 2Producer side
Retries with idempotent producers or dedupe IDs.
- 3Broker side
Replicate and acknowledge durably (acks=all).
- 4Consumer side
Process, then commit or ack.
- 5Make effects idempotent
Dedupe table or transactional writes with offsets.
Where ack timing leads
Consumer crashes at different points
| Strategy | Crash after ack, before processing | Crash after processing, before ack | Guarantee |
|---|---|---|---|
| Ack then process | Message lost | N/A | At-most-once |
| Process then ack | N/A | Message processed twice | At-least-once |
| Process + record ID atomically, then ack | N/A | Duplicate detected and skipped | Exactly-once effect |
NOWStrategy: Ack then process | Crash after ack, before processing: Message lost | Crash after processing, before ack: N/A | Guarantee: At-most-once
Exactly-once is at-least-once delivery plus idempotent processing.
Implementation
// Exactly-once effect: store the processed message ID in the same transaction as the side effectexport async function handlePaymentEvent(msg: { id: string; accountId: string; amountCents: number }) { await db.transaction(async (tx) => { const inserted = await tx.query( "INSERT INTO processed_messages (message_id) VALUES ($1) ON CONFLICT DO NOTHING RETURNING message_id", [msg.id], ); if (inserted.rowCount === 0) return; // duplicate delivery: already applied await tx.query("INSERT INTO ledger (account_id, amount_cents, source_message) VALUES ($1, $2, $3)", [msg.accountId, msg.amountCents, msg.id]); }); // ack/commit offset only after the transaction commits (at-least-once delivery)}Complexity and performance
No retries or dedupe.
Per message.
Trade-offs
Stronger guarantees add storage, latency, and complexity; choose per data flow.
Kafka EOS covers Kafka-to-Kafka flows; external side effects (email, payments) still need idempotency.
Variants and related techniques
Producer-side guarantee that DB changes and events are published together.
Remember IDs for a bounded time (SQS FIFO 5 minutes).
Common mistakes
- Believing a broker's 'exactly-once' covers external side effects.
Fix: Emails and API calls still need idempotency keys.
- Auto-commit offsets.
Fix: Offsets may be committed before processing, losing messages on crash.
Interview questions
Is exactly-once delivery possible?
Not strictly over an unreliable network, because the sender cannot distinguish a lost message from a lost ack. Systems achieve exactly-once processing by combining at-least-once delivery with idempotent or transactional consumers.
Which guarantee would you choose for metrics vs payments?
Metrics can often use at-most-once or at-least-once with approximate dedupe; payments need at-least-once delivery with strict idempotent processing so each payment is applied exactly once.
Practice problems
| Problem | Difficulty | What it trains |
|---|---|---|
| Classify 5 pipelines by delivery guarantee | Easy | Requirements. |
| Implement exactly-once ledger updates from a queue | Medium | Dedupe in transaction. |