FjellSystem Design Lab
Lesson 16.3• 9 min read

Multiple regions and disaster recovery

Route 53 routing policies and health checks, the four AWS disaster recovery strategies, and how Aurora Global Database replicates and fails over.

What you will learn

  • Decide when several Availability Zones are not enough and a second Region is needed.
  • Choose among Route 53's routing policies and health checks.
  • Compare the four disaster recovery strategies by RPO, RTO and cost, and fail over an Aurora global database.

Why go beyond one Region

The primer's design runs in several Availability Zones of one Region. That protects against losing a data center. AWS's disaster recovery whitepaper says that for this, a well-architected, highly available workload "may only require a backup and restore approach". A second Region is for disasters larger than one data center, or for regulatory requirements. It can also put servers closer to users on other continents.


Route 53 routing policies

A DNS record in Route 53 has a routing policy. AWS's list:

  • Simple: one resource.
  • Failover: active-passive failover.
  • Geolocation: by the location of your users.
  • Geoproximity: by the location of your resources, optionally shifting traffic between them.
  • Latency: to the Region with the best latency for the user.
  • IP-based: by the user's source IP ranges.
  • Multivalue answer: up to eight healthy records chosen at random.
  • Weighted: split traffic in proportions you set.

Latency routing uses latency data that Route 53 has collected between users and AWS Regions over time. The data changes as the internet changes: "A request routed to the Oregon Region this week might go to the Singapore Region next week." The lowest-latency Region is not always the nearest.

Health checks let these policies skip records whose endpoint is down. Health checkers in several locations probe the endpoint every 30 s, or 10 s for an extra charge. An endpoint changes state after a set number of consecutive results, the failure threshold. Even then, DNS resolvers and clients may keep the old answer until its TTL expires. The worst case for a client is roughly interval × threshold + TTL.


Four disaster recovery strategies

The AWS whitepaper orders the strategies from cheapest and simplest to most complex:

Strategy What runs in the recovery Region Before it can take traffic
Backup and restore Backups only Restore data and redeploy infrastructure (use infrastructure as code)
Pilot light Replicated data stores; servers defined but "switched off" Deploy or start servers and scale out
Warm standby A scaled-down but working copy Can take traffic at once, at reduced capacity; scale up
Multi-site active/active Full deployment serving users Nothing: every Region serves users

The recovery point objective (RPO) is how much data you can afford to lose. The recovery time objective (RTO) is how long you can be down. Both shrink as you go down the table, and cost rises. Active/active "can reduce your recovery time to near zero for most disasters". Data corruption is different: it is replicated too, so it still needs point-in-time backups, and its recovery point is never zero.

Two pieces of the whitepaper's advice:

  • Use data plane operations to fail over. Control planes such as Auto Scaling, or changing record weights, have lower availability goals than data planes such as health checks. Use Route 53 health checks, including ones you flip by hand through Amazon Application Recovery Controller.
  • Automatic failover can trigger on a false alarm. It costs some data and availability even when it was not needed, so many teams start failover by hand, with every step automated.

Aurora Global Database

An Aurora global database has one primary Region, which takes writes, and up to ten read-only secondary Regions. Each secondary has its own cluster of up to 16 readers. Replication runs at the storage level and is asynchronous, with lag "typically under a second". Readers in each Region serve local users. Write forwarding lets a secondary pass write statements to the primary.

Changing the primary Region can be done two ways:

  • A switchover is planned: Aurora waits for the secondary to catch up, so no data is lost.
  • A failover is for an outage: a secondary is promoted, and writes not yet replicated can be lost. The whitepaper says a secondary can be promoted "in less than one minute". The data at risk is roughly the write rate × the replication lag.

So an Aurora global database is not active/active for writes: every write goes to one Region. The whitepaper calls this write global. The alternatives:

  • Write local: each Region accepts writes, as DynamoDB global tables do, with "last writer wins" for concurrent updates.
  • Write partitioned: each record has a home Region, for example by user ID, so writes never conflict.

Routing users between regions

10,000 RPS from three user groups. Round-trip times are example values; Route 53 uses its own latency measurements. Records with a failing health check are skipped.

us-east-15,000 RPS
eu-west-13,000 RPS
ap-southeast-12,000 RPS

Average round trip for users: 20 ms.

A client can keep reaching a failed region for up to 150 s: 30 s × 3 failed checks, then up to 60 s of cached DNS answers.

Aurora Global Database replicates to secondary Regions asynchronously. An unplanned failover can lose the writes not yet replicated: about 400 writes at this lag.

Key takeaways

  • Several zones protect against losing a data center; a second Region protects against losing a whole Region, at a real cost.
  • Backup and restore, pilot light, warm standby and multi-site active/active cut the recovery point and time objectives as cost rises.
  • Fail over with data-plane operations such as health checks rather than control-plane changes, and consider starting failover by hand to avoid false alarms.
  • An Aurora global database writes in one Region: a planned switchover loses nothing; an outage failover can lose writes not yet replicated.

Sources

Adapted from The System Design Primer by Donne Martin, licensed under CC BY 4.0. Changes were made.