Start learning free

LESSON 67 OF 107 · DATA ENGINEERING: STREAMING BASICS

The Same Event Twice in Python

Practice

A queue promises that every event is delivered at least once. If a consumer handles an event but crashes before telling the queue "got it", the queue can't know the event was handled, so it sends it again. Nothing is lost, but some events arrive twice. Each event's offset is its place in the queue (0, 1, 2, …): here event e4 (12 for order 5004) comes at offset 3 and again at offset 5, so a plain running total ends at 269 instead of 257.

The fix is an idempotent consumer: handling an event twice gives the same result as handling it once. Keep the event_id of every event you handle in a set, and skip any event whose id is already there. A set holds each value once, s.add(x) puts x in it, and x in s checks fast whether it is there, even among millions of values:

ids = set()
ids.add("a7")
ids.add("a7")
print(len(ids))      # 1
print("a7" in ids)   # True
print("b2" in ids)   # False

To skip an event, continue ends this turn of the loop at once and goes on to the next event. Why a data engineer needs it: a report that counts a repeated payment twice shows money that never came in, and nobody notices until the books don't match. Kafka also delivers at least once by default: after a crash, a consumer reads again events it may already have handled, so real consumers keep the handled ids too, often in a database table with a unique key rather than a set in memory.

Your program must print (the last line comes from the check, which counts the ids in your seen):

handled e1 | revenue: 50
handled e2 | revenue: 70
handled e3 | revenue: 105
handled e4 | revenue: 117
handled e5 | revenue: 177
skip e4 at offset 5 | seen before
handled e6 | revenue: 202
handled e7 | revenue: 242
handled e8 | revenue: 257
revenue: 257
ids in seen: 8

Your task: the events and the loop that adds up the revenue are in the starting code. Make an empty set seen and add each handled event's event_id to it. At the top of the loop, when the id is already in seen, print skip, the id, at offset and the offset, then | seen before, and go on to the next event with continue.


Assignment
Follow the Coding Tutorial to make the consumer skip events it has already handled.


Hint
Check the id before anything else in the loop, and add it to seen only for the events you handle. Printing "skip" doesn't skip anything: continue is what keeps the copy away from the revenue line.


STEP 1 OF 5
    Ctrl + Enter to check

    De Stream Idempotent in Python

    Keep going with Python1,600+ step-by-step lessons, from your first line of code to coding interviews. Free to start.
    Start learning free