Quick reference for delivery semantics, ordering, DLQs, retries, and broker comparison.
KafkaDeliveryIdempotencyDLQ
Topology
Producer --msg--> [ Broker: queue/topic/log ] --> Consumer(s)
durable buffer, retries, fan-out
| Model | Delivery |
|---|
| Queue | Each msg to ONE consumer |
| Pub/Sub | Each msg to EVERY subscriber |
| Log | Retained; consumers track offset, replay |
Delivery Semantics
| Guarantee | Risk |
|---|
| At-most-once | May lose |
| At-least-once | Duplicates (default) |
| Exactly-once | Hard; use idempotency |
# idempotent consume
if seen.has(msg.id): return
with tx: apply(msg); seen.add(msg.id) # UPSERT, not INSERT
Ordering & Backpressure
| Ordering | Per-partition/key only; global is costly |
| Parallelism | Capped at partition count |
| Backpressure | Bounded queues, pull, autoscale on lag |
| Key metric | Consumer lag |
Retries & DLQ
delay = min(cap, base * 2^attempt) + jitter
1s -> 2s -> 4s -> 8s -> DLQ (alert, inspect, replay)
poison message = permanent failure -> DLQ, don't block queue
Patterns
Competing consumers: [queue]->C1/C2/C3 (scale throughput)
Fan-out: [event] -> emailSub / indexSub / analyticsSub
Outbox (no dual-write bug):
BEGIN; update state; INSERT outbox; COMMIT;
relay/CDC tails outbox -> broker (at-least-once)
Kafka Terms
| Topic | Named stream |
| Partition | Ordered shard; unit of parallelism |
| Offset | Consumer position; committed |
| Consumer group | 1 partition -> 1 member |
| Compaction | Keep latest value per key |
| ISR | In-sync replica set |
Broker Pick
| System | Use when | Throughput |
|---|
| Kafka | Streams, replay, high volume | millions/s |
| RabbitMQ | Rich routing, priorities, TTL | tens of k/s |
| SQS/SNS | Zero-ops managed AWS | near-unbounded |
| NATS | Ultra-low latency, tiny | millions/s |
