Message Queues

Why Async Messaging?

A message queue (or broker) sits between producers and consumers so they don't call each other directly. Instead of a synchronous request that blocks until the callee responds, the producer hands a message to the broker and moves on; consumers process it later. This buys three things: decoupling (services don't know about each other), buffering (traffic spikes are absorbed rather than dropped), and resilience (a consumer can be down and catch up later).

Queue vs Pub/Sub vs Log

The key difference of a log: messages are not deleted on consume. They persist for a retention window, so multiple independent consumers can read the same stream at their own pace and you can replay from any offset. A classic queue deletes a message once acknowledged.

Delivery Semantics

At-least-once is the practical default. True end-to-end exactly-once across a network is impossible in general (the ack can be lost after the work is done), so systems approximate it with idempotency or transactional processing. Kafka offers exactly-once within Kafka (idempotent producer + transactions), but the moment a consumer writes to an external system, you are back to needing idempotency.

Idempotency — the real fix

Since duplicates are unavoidable with at-least-once, make processing idempotent: applying the same message twice has the same effect as applying it once. Use a unique message/idempotency key and dedupe.

def handle(msg):
    if seen.contains(msg.id):     # dedupe store (Redis/DB w/ TTL)
        return                    # already processed -> skip
    with db.transaction():
        apply_effect(msg)         # e.g. UPSERT, not blind INSERT
        seen.add(msg.id)          # record processed id atomically

Ordering Guarantees

Global total ordering across a distributed queue is expensive and limits throughput. Most systems offer partition-level (or per-key) ordering: messages with the same key go to the same partition and are delivered in order there, while different partitions are processed in parallel. Choose a partition key (e.g. user_id) so related events stay ordered.

Backpressure & Flow Control

If producers outpace consumers, the queue grows without bound — latency balloons and the broker may run out of disk. Backpressure is the feedback that slows producers or sheds load. Tactics: bounded queues that reject/block on full, consumer-driven pull (Kafka) so consumers set the pace, autoscaling consumers on queue depth/lag, and rate limiting producers. Always alarm on consumer lag (unprocessed backlog), the single most important queue metric.

Retries, DLQs, and Poison Messages

Transient failures (a downstream timeout) deserve a retry; permanent failures (malformed payload — a poison message) will fail forever and can block the queue if retried endlessly. The pattern: retry a bounded number of times with exponential backoff + jitter, then route the message to a dead-letter queue (DLQ) for inspection.

Consumption Patterns

Competing consumers parallelize a workload: many consumers pull from one queue, each message handled once, throughput scales with consumer count (up to partition count for logs). Fan-out delivers each event to multiple independent subscribers — the backbone of event-driven architecture, where services react to events instead of being commanded.

The Outbox Pattern

A common bug: a service writes to its DB and publishes an event as two separate steps. If it crashes between them, state and event diverge (the "dual-write" problem). The transactional outbox fixes this: write the event to an outbox table in the same DB transaction as the state change, then a relay (or CDC) reads the outbox and publishes to the broker.

BEGIN;
  UPDATE orders SET status = 'PAID' WHERE id = 42;
  INSERT INTO outbox (id, topic, payload) VALUES (...);  -- same txn
COMMIT;                          -- both or neither

-- relay / CDC (Debezium) tails the outbox table -> Kafka
-- at-least-once publish; consumers dedupe by event id

Kafka Internals

Kafka is a distributed, replicated commit log. Key concepts:

Because a consumer group assigns each partition to exactly one member, maximum consumer parallelism equals the partition count — choose partitions with future scale in mind. Offsets let a slow or restarted consumer resume exactly where it left off, and let you replay history.

Broker Comparison