Start learning free

LESSON 66 OF 107 · DATA ENGINEERING: STREAMING BASICS

Read from a Queue in Python

Practice

In a real system, the code that writes events and the code that reads them are separate programs. The producer (here, the shop's app) writes each new order event; the consumer (your program) reads and handles them. Between them sits a queue: a line where events wait until the consumer is ready. They leave in the order they came in: first in, first out.

Python's collections.deque (say "deck") is a list built for this. append(x) adds x at the right end, popleft() removes the item at the left end and returns it, and len() says how many items are waiting. Both are fast at any size, while a list's pop(0) has to move every other item one place:

from collections import deque

tickets = deque()
tickets.append("ann")
tickets.append("bob")
print(tickets.popleft())   # ann
print(len(tickets))        # 1

Why a data engineer needs it: without a queue, the shop would have to wait for a slow consumer, or lose orders while the consumer is down. With one, the producer keeps writing and the consumer catches up later, with nothing lost and nothing out of order. At work, Kafka plays the deque's part: a log kept on disk that many producers write to and many consumers read from, which keeps each event even after it has been read.

In the starting code, produce() writes e1 to e3, consume() reads, then e4 to e8 are written and consume() runs again. Your program must print:

producer wrote e1 | waiting: 1
producer wrote e2 | waiting: 2
producer wrote e3 | waiting: 3
consumer starts | waiting: 3
consumer read e1 | order 5001 | waiting: 2
consumer read e2 | order 5002 | waiting: 1
consumer read e3 | order 5003 | waiting: 0
producer wrote e4 | waiting: 1
producer wrote e5 | waiting: 2
producer wrote e6 | waiting: 3
producer wrote e7 | waiting: 4
producer wrote e8 | waiting: 5
consumer starts | waiting: 5
consumer read e4 | order 5004 | waiting: 4
consumer read e5 | order 5005 | waiting: 3
consumer read e6 | order 5006 | waiting: 2
consumer read e7 | order 5007 | waiting: 1
consumer read e8 | order 5008 | waiting: 0

Your task: the events, the produce() and consume() functions with their first prints, and the calls are in the starting code. Make an empty deque named queue, let produce() append each event to it, and in consume() loop while queue isn't empty: take the oldest event with popleft() and print what the consumer read and how many events are still waiting.


Assignment
Follow the Coding Tutorial to connect a producer and a consumer through a deque.


Hint
append puts an event at the back of the line and popleft() takes the one at the front, so e1 comes out first. The consumer's loop should stop once len(queue) is 0.


STEP 1 OF 5
    Ctrl + Enter to check

    De Stream Queue in Python

    Keep going with Python1,600+ step-by-step lessons, from your first line of code to coding interviews. Free to start.
    Start learning free