Lesson 9.4• 9 min read

Redirects, caching and visit analytics

HTTP 301 vs 302/307 redirects, caching at the browser, CDN and application, and counting visits from logs or event streams.

What you will learn

Choose between a 301 and a 302 redirect by what each does to load, counting and control.
Cache redirects in the browser, at the CDN and in an application cache.
Count visits with batch MapReduce over logs, or near real time with a stream.

301 or 302?

A shortener answers a click with a redirect and a Location header. Which status code it uses decides whether later clicks reach it at all. From RFC 9110:

Status	Meaning	Caching	Method on redirect
301 Moved Permanently	New permanent URI; "future references … ought to use" it	Heuristically cacheable: a cache may reuse it without asking again, unless told otherwise	A client may change POST to GET
308 Permanent Redirect	Like 301	Heuristically cacheable	Method preserved
302 Found	Resource "resides temporarily under a different URI"; keep using the original	Not cached unless the response gives explicit freshness	A client may change POST to GET
307 Temporary Redirect	Like 302	Like 302	Method preserved

The trade-off:

With a 301, a browser may answer repeat clicks from its own cache. That saves the round trip and server load, but those clicks never reach you. You cannot count them, and you cannot change or expire the link for anyone who has it cached.
With a 302 or 307, every click comes back to you unless you send explicit cache headers. Counting is accurate and links stay editable, at the cost of serving every click.

Services choose differently, depending on whether they care more about load or about counting and control. Ask which one the interviewer wants.

Caching redirects and content

The primer's scaled design puts the popular content in a memory cache, which also absorbs uneven traffic and spikes. Read replicas handle the misses, and a CDN serves static content.

A redirect can be cached at three levels:

Browser: a 301, or any redirect sent with explicit freshness such as Cache-Control: max-age=60 (RFC 9111).
CDN: a 301 (RFC 9110 lets any cache store it heuristically, not only the browser), or any redirect that allows shared caching, for example Cache-Control: public, max-age=60. Clicks the CDN answers never reach your servers. They appear only in the CDN's access logs.
Application cache (Redis or Memcached): key shortlink, value the target URL or paste location, evicted by LRU or LFU (chapter 6). Your servers still see every click it answers.

A short max-age is a compromise: most of the load goes to the edge, and changes take effect within a minute.

Counting visits

The primer's analytics are deliberately simple. The requirement is monthly visit statistics, not real-time ones. So it runs MapReduce over the web server logs, counting hits per shortlink per month:

def mapper(self, _, line):
    url = self.extract_url(line)
    period = self.extract_year_month(line)
    yield (period, url), 1

def reducer(self, key, values):
    yield key, sum(values)

The results go to an analytics database. The primer suggests a data warehouse such as Amazon Redshift or Google BigQuery.

If analytics must be near real time, publish a click event to a log such as Kafka after sending the redirect, so the click never waits on it. Then aggregate it with a stream processor (chapters 7 and 15). Either way, clicks answered by a browser cache are invisible, and clicks answered by a CDN appear only in the CDN's logs. Count those logs too, or choose redirect headers that send clicks to you.

Who sees a click?

Any cache, the browser or a CDN, may reuse a 301; a 302 is cached only if the response says so. A redirect that a cache answers never reaches your servers. Here every click falls within the cached copy’s lifetime.

Redirect status

Send Cache-Control: public, max-age=60

Clicks

Seen by your servers

Only in CDN logs

Never seen

Click	Visitor	Answered by	Click event

Figure 4: Production architecture with CDN edge caching and an optional click-event stream for near-real-time analytics.

Key takeaways

A 301 can be cached by browsers and CDNs, saving load but losing the clicks and the power to change the link; a 302 brings every click back to you.
A short max-age sends most of the load to the edge while changes still take effect within a minute.
Monthly statistics need only MapReduce over web server logs into a warehouse.
For near-real-time counts, publish a click event after sending the redirect, then aggregate the stream; clicks answered by a browser cache are invisible, and clicks answered by a CDN appear only in the CDN's logs.

Sources

Adapted from The System Design Primer by Donne Martin, licensed under CC BY 4.0. Changes were made.