Serving reads: caching, hot keys and sharding

Scaling reads (the primer)

The primer's scaled design:

• A memory cache handles "traffic for popular content (and their sales rank)", which also absorbs uneven traffic and spikes. With so many reads, "the SQL Read Replicas might not be able to handle the cache misses", so further SQL scaling patterns may be needed.
• 400 writes per second on average "might be tough for a single SQL Write Master-Slave". The primer points to federation, sharding, denormalization and SQL tuning, and to moving some data to NoSQL.
• An analytics database (a warehouse such as Redshift or BigQuery) for the computation. Raw data beyond a limited period stays in the object store, which "can comfortably handle" 40 GB a month.

Layers of caching

Rankings change at most hourly in the primer's design, so cached copies stay valid for a long time. You can cache them at several layers:

1. CDN or shared HTTP cache: send Cache-Control: public, max-age=60 on the category page (RFC 9111, chapter 9).
2. In-process cache on each app server, with a short TTL.
3. Redis holding the rankings.
4. The database with read replicas.

How much does each layer absorb? Suppose a page is requested λ times a second, and each miss caches it for $T$ seconds. Then each miss is followed by about $\lambda T$ hits, so the hit ratio is

Popular categories are requested often, so they nearly always hit. Rarely viewed ones mostly miss. The visual applies this with Zipf-distributed popularity over the primer's 1,000 categories, so you can see how skew, TTLs and the number of app servers change the load that reaches Redis. Every cache layer adds staleness of up to its TTL. With hourly rankings, a TTL of a minute costs little.

Hot keys and sharding

A single very popular category is a hot key: all its reads land on the one Redis node that holds it. Caches in front (above) take most of that load, and Redis replicas can share the reads.

Writes can also concentrate: a purchase updates every category the product belongs to. If one category's updates are too many for one node, split them, but think about what each shard holds:

• Split by product: each product lives in exactly one shard, holding its full count. A category's top $N$ is then the top $N$ of the merged per-shard top-$N$ lists.
• Split writes for the same product across shards: each shard holds only partial counts. Merging the shards' top lists is now wrong. A product could be just outside the top $N$ in every shard yet first overall. You must sum each product's counts first (ZUNIONSTORE does this) and then rank. The visual's tests include a case where the shortcut picks the wrong winner.