Lesson 16.1• 9 min read

From one user to millions on AWS

The primer’s steps: one box, separate storage and database, horizontal web tier and CDN, cache and read replicas, autoscaling, then warehouse, NoSQL and queues.

What you will learn

Grow the primer's AWS design step by step, from one box to tens of millions of users.
Name the bottleneck behind each step and the AWS service that answers it.
Say what each step costs in complexity, security and money.

Scale in steps, driven by measurement

The primer's AWS exercise starts with one user and grows to tens of millions. It does not build the final design on day one. Each step follows the same loop:

\text{Benchmark / load test} \longrightarrow \text{Profile} \longrightarrow \text{Fix the bottleneck, weighing trade-offs} \longrightarrow \text{Repeat}

The primer marks growth with symbols (Users+, Users++, …) instead of user counts, because the point at which each step is needed depends on the workload. It also says that at each step "you might only need to implement one or two of these techniques".

The constraints

The primer's assumptions:

Traffic is not evenly distributed, and the data is relational.
10 million users; 1 billion writes and 100 billion reads a month (100:1).
1 KB of content per write.

Using its rule of thumb of 2.5 million seconds in a month:

Quantity	Calculation	Result
Writes per second	$10^9 / (2.5 \times 10^6)$	400 on average
Reads per second	$10^{11} / (2.5 \times 10^6)$	40,000 on average
New content per month	$1\text{ KB} \times 10^9$	1 TB
New content in 3 years	$36 \times 1\text{ TB}$	36 TB

With an exact 30-day month (2,592,000 s) the rates are 386 writes and 38,580 reads per second. Peaks are higher, since traffic is uneven.

The steps

Start with one box. A web server on EC2 with MySQL on the same instance. Give it an Elastic IP, a public address that does not change on reboot, and point a Route 53 DNS record at it. Open only the ports needed: 80 and 443 to everyone, 22 only to allowed IPs. Scale vertically (a bigger instance) while monitoring CPU, memory, I/O and network. This setup has no redundancy and no failover, and vertical scaling gets expensive.

Users+: separate the parts. Profiling shows MySQL taking more memory and CPU, and user content filling the disk.

Move static content (user files, JS, CSS, images, videos) to an object store such as S3.
Move MySQL to its own managed instance, such as RDS, which offers multiple Availability Zones and encryption at rest.
Put everything in a VPC: a public subnet for the web server, a private subnet for everything else.

Now the web server and the database can scale independently. In exchange there is more to secure, and the web server must change to use the new components.

Users++: scale the web tier horizontally. The single web server is the bottleneck at peak.

Add a load balancer such as ELB, and terminate TLS on it.
Run several web servers across several Availability Zones.
Run MySQL as primary and standby across zones, for failover.
Separate web servers (acting as reverse proxies) from application servers, so each layer scales on its own.
Serve static, and some dynamic, content from a CDN such as CloudFront.

Users+++: relieve the database. The workload is read-heavy (100:1) and reads are overloading MySQL.

First tune MySQL's own cache. If that is not enough, add a memory cache such as ElastiCache for frequently read content.
Move session data into the cache too. The web servers become stateless, which makes autoscaling possible.
Add MySQL read replicas, with application logic that sends writes to the primary and reads to replicas.

Users++++: follow the daily curve. Traffic peaks in U.S. business hours and drops afterward. Paying for peak capacity all night is waste.

Add Auto Scaling groups, one per server type, each across several Availability Zones, with a minimum and maximum size.
Scale on a schedule for predictable load, or on CloudWatch metrics (CPU, latency, network, or a custom metric).
The primer lists the costs: more complexity, and a delay before new capacity arrives.
Automate operations, and keep monitoring at host level, load-balancer level, in logs, and from outside.

Users+++++: keep finding bottlenecks.

Keep only recent data in MySQL and move history to a warehouse such as Redshift.
Scale the memory cache for popular content and spikes. The read replicas may struggle with the cache misses.
400 writes per second on average, with higher peaks, may be too much for one primary. Consider SQL scaling patterns: federation, sharding, denormalization, SQL tuning. Consider moving suitable data to a NoSQL store such as DynamoDB.
Do non-real-time work asynchronously: the application server puts a job on a queue such as SQS, and workers on EC2 or Lambda process it. In the primer's photo example, the worker makes the thumbnail, updates the database and stores the result in S3.

The diagram shows the end of this sequence.

Figure 1: The primer’s design at Users+++++: DNS, CDN, load balancer, stateless web and app servers, memory cache, MySQL primary with read replicas, a queue with workers, and S3. Instance counts are illustrative.

Key takeaways

Every step follows the same loop: benchmark, profile, fix the bottleneck, repeat; the primer marks steps with Users+ symbols because the thresholds depend on the workload.
Separate first (S3 for static content, RDS for MySQL, a VPC), then scale the web tier behind a load balancer across zones with a standby database and a CDN.
For the 100:1 read load, add a cache and read replicas; moving sessions into the cache makes web servers stateless, so Auto Scaling can follow the daily curve.
Then move history to a warehouse, consider SQL scaling patterns for writes, and put non-real-time work on queues.

Sources

System Design Primer: Design a system that scales to millions of users on AWS

Adapted from The System Design Primer by Donne Martin, licensed under CC BY 4.0. Changes were made.