Stream processing versus batch recomputation

The primer's answer: an hourly MapReduce

Results only need to be hourly. So the primer's Sales Rank Service runs a two-step MapReduce over the log files and writes the result to sales_rank:

1. Sum: the mapper keeps lines from the past week (within_past_week) and emits (category, product_id), quantity. The reducer sums them.
2. Sort: a second mapper turns each result into (category, total_sold), product_id. MapReduce's shuffle then sorts by that key, giving each category's products in order of sales.

def mapper(self, _, line):
    timestamp, product_id, category_id, quantity, total_price, seller_id, buyer_id = line.split('\t')
    if self.within_past_week(timestamp):
        yield (category_id, product_id), quantity

def reducer(self, key, values):
    yield key, sum(values)

def mapper_sort(self, key, value):
    category_id, product_id = key
    quantity = value
    yield (category_id, quantity), product_id

Recomputing from the raw log has a strong advantage. The log is complete and immutable, so the job counts every transaction in the window, including ones that were delayed in transit, and a buggy run can simply be rerun.

Updating continuously: stream processing

If rankings must change faster than hourly, a stream processor (Flink, Kafka Streams and others) can consume purchase events and keep running totals per window. The window types, as Flink defines them:

• Tumbling windows "have a fixed size and do not overlap".
• Sliding windows "can be overlapping if the slide is smaller than the window size", so one event is counted in several windows.

Late data. Events arrive out of order. Windows are defined by event time, when the purchase happened. A watermark tells the processor how far event time has progressed and triggers the windows that have ended. In Flink, "by default, late elements are dropped when the watermark is past the end of the window". Allowed lateness keeps windows open longer, at the cost of holding more state. The visual shows a late purchase dropped by the stream but counted by the batch recount.

Lambda and Kappa

Running both is the Lambda architecture. Jay Kreps describes it: "an immutable sequence of records is captured and fed into a batch system and a stream processing system in parallel", with results combined at query time. His objection: "maintaining code that needs to produce the same result in two complex distributed systems" is a lasting burden.

His alternative, which he jokingly named the Kappa architecture: keep the log, and to reprocess, "start a second instance of your stream processing job that starts processing from the beginning of the retained data", writing to a new output table.

For this feature, the primer's hourly batch already meets the requirement. A stream helps only if fresher rankings are worth running two systems, or reprocessing from the log.

Weighting recent sales

The primer ranks by total sales over the past week. To favor recent sales smoothly instead, weight each purchase by its age with an exponential decay. With half-life $T_{1/2}$:

With $T_{1/2} = 24$ hours, a sale from 24 hours ago counts half and one from 48 hours ago a quarter. This is one design option. Amazon says only that recent activity is "weighted more heavily".