Lesson 7.1• 9 min read

Queues, task queues and commit logs

Point-to-point task queues (RabbitMQ, SQS) versus distributed append-only logs (Kafka, Pulsar).

What you will learn

Use Little's law to see why slow synchronous work starves a thread pool.
Take slow work off the request path with a queue and a quick acknowledgment.
Compare task queues with a distributed commit log such as Kafka, and scale Kafka consumers by partition.

The synchronous bottleneck and thread pool starvation

In a traditional synchronous request-response architecture, an incoming HTTP request blocks an operating system or runtime worker thread for the entire duration of downstream processing:

T_{\text{sync}} = t_{\text{network}} + t_{\text{auth}} + t_{\text{transcode}} + t_{\text{disk\_io}} + t_{\text{notification}}

If video transcoding or PDF rendering takes $3\text{ to }15\text{ seconds}$ , a server with $200$ worker threads is fully saturated at just:

\text{Max Concurrency} = \frac{200\text{ threads}}{5\text{ seconds}} = 40\text{ requests/sec}

By Little's law ( $L = \lambda W$ ), 200 busy threads at 5 seconds each can sustain at most $200 / 5 = 40$ requests per second. Beyond that, requests queue up waiting for a thread, latency climbs, and clients (or proxies in front) start timing out.

Asynchronous decoupling: fast acknowledgment, deferred work

Asynchronous workflows take expensive operations off the request path. With a message queue:

The application publishes a job to the queue and tells the user the job's status straight away (for example, HTTP 202 Accepted with a status URL). The user is not blocked.
A worker picks up the job, processes it in the background at its own pace, and signals when it is done.

The client can even make the result look immediate: a posted tweet appears on your own timeline at once, though delivering it to all your followers takes longer. The cost: queues add delay and complexity, so cheap calculations and real-time workflows may be better done synchronously.

Common choices differ: Redis is a simple broker but can lose messages; RabbitMQ is popular but uses the AMQP protocol and you run the nodes; Amazon SQS is hosted but can have high latency and may deliver a message twice. Task queues such as Celery go further: they run the task and return its result, and can schedule jobs.

Task queues versus distributed commit logs

Dimension	Message / task queue (RabbitMQ, SQS, Celery)	Distributed commit log (Kafka)
Storage	A buffer from which messages are removed once processed	An append-only log; events are not deleted when consumed
Consumption	Competing workers take messages; each message goes to one worker	Each consumer group tracks its own position (offset) in each partition
Ordering	Usually best effort, especially across retries (SQS FIFO queues are the exception)	Events with the same key go to the same partition and are read in exactly the order written
Replay	Not normally possible once a message is consumed	Rewind a consumer group's offset, for as long as the topic's retention keeps the events
Typical use	Background jobs: emails, transcoding, scheduled tasks	Event streams read by many consumers, change data capture, audit trails

Kafka is built for throughput: it relies on sequential disk writes and the operating system's page cache, and uses sendfile to send data from the page cache to the network without copying it through the application.

Partitioning and consumer group scaling in Kafka

A Kafka topic is divided into $P$ partitions; by default an event's key is hashed to choose its partition:

\text{Partition} = \text{hash}(\text{key}) \bmod P

Ordering: Events with the same key (e.g. user_id_492) land on the same partition, and every consumer reads a partition's events in the same order (as long as the number of partitions does not change).
Consumer groups: Each partition is consumed by exactly one consumer within a consumer group at any given time, so a group can use at most $P$ consumers in parallel ( $N_{\text{consumers}} \le P$ ); extra consumers sit idle.

Task queue versus commit log

Task queue model: Messages are transient units of work. Competing workers pull items from a shared buffer. Once acknowledged (ACK), the message is deleted permanently. Designed for fine-grained task distribution and arbitrary scale-out.

Message Buffer (2 Pending)

FIFO order

msg_101

video_transcode_1080p

ACKNOWLEDGED

msg_102

push_notification_order_paid

PROCESSING

msg_103

email_invoice_pdf_gen

PENDING

msg_104

fraud_score_evaluation

PENDING

Competing worker pool (consumer fleet)

Worker-A

Status: Idle (Polling)Processed: 14 tasks

Worker-B

Status: Processing msg_102Processed: 11 tasks

Worker-C

Status: Idle (Polling)Processed: 9 tasks

Figure 1: Synchronous web ingestion tier decoupled from asynchronous worker fleet via Kafka message log.

Key takeaways

200 threads holding requests for 5 seconds each serve at most 40 requests a second; past that, requests queue and time out.
Publish a job, answer at once (for example 202 Accepted) and let workers do the work; queues add delay and complexity, so keep cheap work synchronous.
A task queue removes each message once one worker processes it; a commit log keeps events so every consumer group reads at its own offset and can replay.
Kafka keeps order per partition, and a consumer group can use at most one consumer per partition.

Sources

Adapted from The System Design Primer by Donne Martin, licensed under CC BY 4.0. Changes were made.