Replication, failover and client routing

How a client finds the right server

1. Smart client. The client library knows the servers, or Redis Cluster's slot map, and connects straight to the right one. This needs no extra hop, but every client holds connections to every server.
2. Proxy tier. Clients talk to a proxy (Twemproxy, Envoy and others) that hashes the key and forwards the request. Fewer connections reach the cache servers, at the cost of an extra hop and another tier to run.
3. Redis Cluster redirects. A client may send a command to any node. A node that does not own the slot replies -MOVED 3999 127.0.0.1:6381, and the client should update its slot map. During resharding, a node may reply -ASK instead. That means "try the other node for this one request"; the client should not update its map yet.

Replication (Redis)

From Redis's replication docs:

• "Redis uses by default asynchronous replication." The primary streams its write commands to replicas without waiting for them.
• Each dataset history has a replication ID and an offset. A reconnecting replica sends both with PSYNC. If the primary still has the missed part in its backlog, it sends only that part, a partial resync. Otherwise the replica needs a full copy. The default repl-backlog-size is 1 MB in redis.conf; size it for the outages you want to survive.
• WAIT can wait for replicas to acknowledge a write, but "it does not turn a set of Redis instances into a CP system": acknowledged writes can still be lost in a failover.

Failover with Redis Sentinel (non-clustered Redis)

Sentinel monitors primaries and replicas and promotes a replica when the primary fails:

1. A Sentinel that cannot reach the primary for down-after-milliseconds marks it subjectively down (+sdown).
2. When enough Sentinels agree, the quorum, the primary is objectively down (+odown). But "the quorum is only used to detect the failure". To fail over, one Sentinel must be elected leader with "the vote of the majority of the Sentinel processes".
3. The leader picks a replica by replica-priority, then the largest replication offset, then the smaller run ID, and promotes it.
4. Clients learn of the new primary from the +switch-master event.

Writes during a partition. If the old primary is cut off together with some clients, those clients "may continue to write data to the old master". That data is "lost forever" when the old primary rejoins as a replica. min-replicas-to-write makes a primary refuse writes when it cannot reach enough replicas, which bounds the loss.

Failover in Redis Cluster

Redis Cluster does not use Sentinel. Nodes flag an unreachable node PFAIL after NODE_TIMEOUT, and when a majority of primaries agree it becomes FAIL. A replica of the failed primary then starts an election, and the primaries vote; the replica that wins promotes itself and takes over the slots. The spec warns that, with asynchronous replication, "there is always a window of time" in which acknowledged writes can be lost.