Start learning free

LESSON 69 OF 107 · DATA ENGINEERING: STREAMING BASICS

Late Events and Watermarks in Python

Practice

Events don't always arrive in the order they happened. A phone was offline, a network was slow: an order placed at 10:01 arrives after orders from 10:03. Its event time, the moment it happened, still puts it in the 10:01 window. But a stream never ends, so each window must print a final answer at some point. When is it safe to stop waiting?

A watermark answers that. It is the newest event time seen so far minus an allowed lateness, and it means "every event older than this should be here by now". A window closes, printing its answer and forgetting it, once the watermark reaches the window's end. An event whose window has already closed is too late: you keep its id aside instead of counting it, so a closed window never opens again. Without a watermark, a data engineer either never gets a final number for a minute, or sends out a second, wrong number when a straggler reopens it.

A timedelta, from datetime, is a length of time: subtract it from a datetime to go back in time. With 5 minutes of lateness and 14:20:00 as the newest time, the watermark is 14:15:00. The window 14:14 ends at 14:15, so it closes, and an event from 14:13:30 that arrives now is too late.

from datetime import datetime, timedelta

seen = datetime(2026, 5, 1, 14, 20, 0)
print(seen - timedelta(minutes=5))   # 2026-05-01 14:15:00

The starting code holds 10 events, two of them out of order, and close_windows(watermark), which prints and forgets every window the watermark has passed. Its loop already turns each event_time into a datetime with datetime.strptime, finds the window's start (the time with its seconds set to 0) and keeps newest up to date. Your code must print:

watermark 10:01:15 | window 10:00 closed | revenue: 70
watermark 10:02:20 | window 10:01 closed | revenue: 60
watermark 10:03:30 | window 10:02 closed | revenue: 12
watermark 10:04:10 | window 10:03 closed | revenue: 60
too late: ['e9']
still open: 10:04 | revenue: 40
still open: 10:05 | revenue: 15
still open: 10:06 | revenue: 22

Kafka itself only keeps events in the order they arrived; stream tools that read from it, such as Flink or Kafka Streams, track a watermark for you and let you set the allowed lateness on each window (Kafka Streams calls it a grace period).

Your task: set lateness to 2 minutes, compute watermark in the loop and call close_windows with it at the end of each turn. Then, above the revenue line, put an event whose window has already closed in too_late and skip it with continue, which ends this turn of the loop and goes on to the next event.


Assignment
Follow the Coding Tutorial to add a watermark that closes windows and sets too-late events aside.


Hint
A window that starts at start ends at start + minute. It has closed when that end is at or before watermark (<=). The check must come before the event reaches revenue.


STEP 1 OF 6
    Ctrl + Enter to check

    De Stream Late Events in Python

    Keep going with Python1,600+ step-by-step lessons, from your first line of code to coding interviews. Free to start.
    Start learning free