Lesson 1.2• 5 min read

Latency versus throughput

Understanding Little's law, response time percentiles, and tail latency amplification.

What you will learn

Define latency and throughput, and the units each is measured in.
Use $W = \frac{1/\mu}{1 - \rho}$ to see how waiting grows with utilization.
Size a pool with Little's law, and explain why p99 matters more than the average.

Precise definitions

Latency: The elapsed time required to complete a single operation or round-trip, measured in milliseconds (ms).
Throughput: The rate at which the system successfully processes operations or data, measured in requests per second (RPS) or bytes per second (GB/s).

You should aim for maximal throughput with acceptable latency. As utilization approaches the capacity of a server, requests spend longer waiting in its queue, and the wait grows without bound. In the simplest queueing model (M/M/1: random arrivals, one server), the average time in the system is the bare service time divided by $(1 - \rho)$ , where $\rho$ is utilization:

W = \frac{1/\mu}{1 - \rho}

At $50\%$ utilization a request takes on average $2\times$ its service time, at $90\%$ it takes $10\times$ , and at $99\%$ it takes $100\times$ . The growth is not exponential, but it is steep enough that systems are run well below full utilization.

Little's law of queueing

In any stable queuing system, the long-run average number of requests in the system ( $L$ ) equals the average arrival rate ( $\lambda$ ) multiplied by the average time each request spends in the system ( $W$ ):

L = \lambda \times W

For example, if your application receives $10,000\text{ RPS}$ ( $\lambda$ ) and each request spends $50\text{ ms}$ in the system ( $W = 0.05\text{ s}$ ), then on average $500$ requests are in flight at once. Size thread pools and connection pools for at least that, plus headroom for bursts.

Why averages lie: tail latency (p99)

Averages obscure critical degradation. If $99\%$ of requests take $10\text{ ms}$ , but $1\%$ take $2,000\text{ ms}$ , the average looks healthy ( $\approx 30\text{ ms}$ ), but $1$ in every $100$ customers experiences an unacceptable delay.

When a single user request fans out to many servers in parallel, it is only as fast as the slowest one. If a page load fans out to $50$ backend services and each independently has a $1\%$ chance of being slow:

P(\text{at least one slow response}) = 1 - (1 - 0.01)^{50} \approx 39.5\%

Nearly $40\%$ of page loads are slow, even though each service is slow only $1\%$ of the time. Dean and Barroso's "The Tail at Scale" gives the same effect at Google: with $100$ servers per request, $63\%$ of user requests take over one second.

Figure 2: Traffic queuing through application workers under heavy concurrent load.

Key takeaways

Aim for the most throughput at acceptable latency: time in the system is the service time divided by $(1 - \rho)$ , so 2× at 50% busy and 10× at 90%.
Little's law, $L = \lambda \times W$ , gives the number of requests in flight; size thread and connection pools for at least that, plus headroom.
Averages hide slow requests. A request that fans out is as slow as its slowest call: 50 calls, each slow 1% of the time, make about 40% of requests slow.

Sources

Adapted from The System Design Primer by Donne Martin, licensed under CC BY 4.0. Changes were made.