Lesson 6.1• 8 min read

Caching tiers and cache placement

Browser, CDN edge, reverse proxies, distributed Redis, and DB buffer pools.

What you will learn

Place caches along the request path, from the client to the database, and say what each tier holds.
Compute effective latency from the hit ratio, and how the hit ratio changes database load.
Size an application cache for its working set.

The memory hierarchy and the cost of disk I/O

Data access times span many orders of magnitude (chapter 2's latency numbers, from about 2012): a main memory reference takes about $100\text{ ns}$ , a random 4 KB SSD read about $150\,\mu\text{s}$ , and a round trip within a datacenter about $500\,\mu\text{s}$ . Data held in RAM is much faster to reach than data on disk, which is why in-memory caches such as Memcached and Redis sit between applications and their databases.

Caching works because accesses are not uniform: recently used data tends to be used again, and popular items take a large share of requests. A cache in front of a database also absorbs uneven load: databases do best when reads and writes spread evenly across partitions, and popular items skew that distribution.

The caching hierarchy

Caches can sit at every stage of the request path:

Tier	Where	Examples	What it typically holds
1. Client	The OS or browser	HTTP cache (`Cache-Control`, `ETag`)	Static files, recently fetched responses
2. CDN	Edge locations near users	CloudFront	Static media and other cacheable responses
3. Web server	Reverse proxy or web cache	NGINX, Varnish	Static and dynamic responses, without contacting the application servers
4. Application cache	In-memory key-value stores between the app and its storage	Memcached, Redis	Sessions, fully rendered pages, activity streams, user graph data
5. Database	The database's own memory	PostgreSQL shared buffers, InnoDB buffer pool	Recently used pages; usually some caching is on by default, and tuning it for your workload helps further

Within the application cache you can cache at two levels: query results (hash the query as the key; hard to invalidate, because one changed cell can affect many cached queries) or objects (assemble the data into an object and cache that; remove it when its underlying data changes). Avoid file-based caching, which makes cloning and autoscaling servers harder.

Cache hit ratio mathematics: The effective latency equation

The performance of any cached system is strictly governed by its Hit Ratio ( $h$ ):

h = \frac{\text{Cache Hits}}{\text{Cache Hits} + \text{Cache Misses}}

The Effective Average Latency ( $L_{\text{effective}}$ ) experienced by clients is:

L_{\text{effective}} = h \cdot L_{\text{cache}} + (1 - h) \cdot (L_{\text{cache}} + L_{\text{db}})

Why 95% vs. 99% hit ratio matters

Consider a system where $L_{\text{cache}} = 2\text{ ms}$ and $L_{\text{db}} = 40\text{ ms}$ :

At $h = 80\%$ : $L_{\text{effective}} = 0.80(2) + 0.20(42) = 1.6 + 8.4 = \mathbf{10.0\text{ ms}}$ (Database absorbs 20% of traffic).
At $h = 95\%$ : $L_{\text{effective}} = 0.95(2) + 0.05(42) = 1.9 + 2.1 = \mathbf{4.0\text{ ms}}$ (Database absorbs 5% of traffic).
At $h = 99\%$ : $L_{\text{effective}} = 0.99(2) + 0.01(42) = 1.98 + 0.42 = \mathbf{2.4\text{ ms}}$ (Database absorbs only 1% of traffic).

A drop from $99\%$ to $95\%$ hit ratio multiplies database query load by $5\times$ ( $1\% \to 5\%$ ).

Working set estimation & sizing formula

The working set is the subset of data actively requested over a time window (e.g. 24 hours). Size the cache to hold it:

M_{\text{RAM}} = \text{Working set keys} \times \text{Average item size} \times (1 + \text{Overhead margin})

Assume, for illustration, that 20% of a catalog's items receive most of its reads (an 80/20 split; measure your own traffic to find the real share):

\text{Example: } 10\text{M products} \times 20\% \times 2\text{ KB} \times 1.25 = 2\text{M} \times 2.5\text{ KB} = \mathbf{5.0\text{ GB RAM}}

Figure 1: Multi-tier caching hierarchy across client, edge, compute, and data tiers.

Key takeaways

Caching works because accesses are skewed: recent and popular items take most requests.
Cache objects rather than query results where you can: an object is easy to invalidate when its data changes.
$L_{\text{effective}} = h \cdot L_{\text{cache}} + (1 - h)(L_{\text{cache}} + L_{\text{db}})$ ; falling from 99% to 95% hits sends five times the queries to the database.
Size RAM as working-set keys × item size × (1 + overhead), using the share of items your own traffic shows to be hot.

Sources

Adapted from The System Design Primer by Donne Martin, licensed under CC BY 4.0. Changes were made.