Backpressure, queue lag and dead-letter queues

Queue lag dynamics and backlog accumulation

Let $\lambda$ be the inbound message arrival rate (messages/sec), $N$ be the active consumer worker count, and $\mu$ be the average processing capacity per worker (messages/sec).

The net queue accumulation rate $\Delta Q$ is defined as:

1. Equilibrium ($\lambda \le N \cdot \mu$): Consumers keep pace with ingestion; queue depth remains near zero.
2. Deficit ($\lambda > N \cdot \mu$): Queue lag accumulates at a rate of $\lambda - N \cdot \mu$ messages/second.
3. Drain Time ($T_{\text{drain}}$): If an initial backlog $L_0$ accumulates during a traffic spike, the time required to restore the queue to zero once arrival drops to $\lambda_{\text{post}}$ is:

This only works when $N \cdot \mu > \lambda_{\text{post}}$; otherwise the backlog never drains and keeps growing.

Back pressure: An ever-growing queue can outgrow memory, causing cache misses, disk reads and even slower processing. Limiting the queue size keeps throughput and response times good for the jobs already in it. Once the queue is full, new requests are refused (for example with HTTP 503), and clients retry later, ideally with exponential backoff.

Poison pills and Dead-Letter Queue (DLQ) routing

A poison pill is a corrupted, unparseable, or schema-invalid message that causes worker code to throw an unhandled exception every time it is read:

How much damage it does depends on the broker. In an ordered log such as a Kafka partition, a consumer that keeps retrying the bad message never moves past it, so everything behind it waits (head-of-line blocking). In a queue such as SQS, a message that is not deleted is received again later (SQS moves standard-queue messages received three or more times to the back of the queue), so other messages keep flowing, but the bad one keeps coming back and wasting work.

The DLQ pattern:

1. Capped exponential backoff with jitter: Retry transient failures with waits that grow exponentially up to a cap, and add randomness so many clients do not retry in lockstep. Marc Brooker's "full jitter" picks the wait at random between zero and the capped exponential value:
   $$t_{\text{sleep}} = \text{random}(0, \min(M, B \cdot 2^i))$$
   where $B$ is the base delay (e.g. $100\text{ ms}$), $M$ the cap (e.g. $30\text{ s}$), and $i$ the attempt number.
2. Max retry threshold: After a set number of failed attempts, move the message to a dead-letter queue (DLQ) with its error details. In SQS this is the redrive policy's maxReceiveCount: set it high enough to allow genuine retries. (A DLQ breaks strict ordering, so avoid one where the exact order of messages matters.)
3. Unblocked pipeline: Healthy messages keep flowing. Engineers inspect the DLQ, fix the bug, and move the messages back for reprocessing (SQS calls this redrive).