Delivery guarantees, ordering and idempotency

The spectrum of distributed delivery semantics

In any distributed network where nodes, switches, and processes can fail, message transport guarantees fall into three categories:

1. At most once: Messages may be lost but are never redelivered.
   • For example, the producer does not wait for the broker to confirm, or the consumer records its position before doing the work. If something fails at the wrong moment, the message is lost.
2. At least once: Messages are never lost but may be redelivered.
   • The broker keeps the message until the consumer confirms it has been processed; if that confirmation is lost or times out, the message is delivered again, so the work can happen twice.
3. Exactly once: Each message is processed once and only once.
   • A sender cannot tell a lost message from a lost acknowledgment, so over an unreliable network it must either risk losing the message (at most once) or risk sending it twice (at least once). "Exactly once" is therefore achieved in processing, not delivery.
   • Inside Kafka this is supported by an idempotent producer and transactions that write to several partitions atomically. When the output goes to an external system, the consumer must store its position in the same place as its output (or make the processing idempotent), so a retry cannot apply the same message twice.

Designing idempotent consumers with idempotency keys

An operation is idempotent if applying it multiple times produces the exact same system state as applying it once:

Implementation: record processed message IDs in the same transaction

The reliable guard is the database itself. Record each message's ID in a table whose primary key is that ID, inside the same transaction as the business update. If the message was already processed, the insert fails and the whole transaction rolls back, so the update is never applied twice:

async function processPaymentEvent(event: PaymentEvent) {
  const { messageId, userId, amount } = event;
  try {
    await db.transaction(async (tx) => {
      // Fails with a unique-key violation if this message was already processed.
      await tx.insert('processed_messages', { message_id: messageId });
      await tx.update('accounts').where({ id: userId }).decrement('balance', amount);
    });
  } catch (err) {
    if (isUniqueViolation(err)) return { status: 'DUPLICATE_SKIPPED' }; // already done: ignore
    throw err; // a real failure: let the broker redeliver
  }
  return { status: 'PROCESSED' };
}

A fast check in Redis (SET key value NX) can sit in front of this, but it must not be the only guard: if the consumer crashes after setting the Redis key and before the database commit, every retry would be skipped and the payment never applied.

Message ordering hazards: Global vs partition ordering

A frequent misconception in system design is requiring global FIFO ordering across all system events. Global FIFO requires a single monolithic queue with concurrency $N=1$, causing an extreme performance bottleneck.

• Solution: Enforce causal ordering by entity key. A customer's banking actions (Deposit $\rightarrow$ Transfer $\rightarrow$ Balance Check) must execute sequentially. But Customer A's actions are completely independent of Customer B's actions.
• By setting PartitionKey = userId, all events for a single user land on the same partition and process in order, while many partitions are processed in parallel across the worker cluster.