A DataFrame That Isn't in Memory in Python

Apache Spark runs data work on many machines at once. It cuts the rows into pieces and hands each piece to a worker: one core of your laptop, or one machine in a cluster of a hundred. You write the same code either way. Data engineers need it because pandas keeps every row in one program's memory, and a big shop's day of orders can be larger than any one computer's memory.

The starting code makes spark, a SparkSession: your program's connection to Spark. spark.read.csv(path, header=True, inferSchema=True) gives a DataFrame. It looks like a pandas DataFrame, but its rows are not in Python's memory: Python holds a small handle that knows the file, the column names and their types, and Spark reads the rows when you ask for a result. header=True takes the column names from the first line; inferSchema=True makes Spark go through the file once to guess each column's type (without it, every column is text).

customers = spark.read.csv("customers.csv", header=True, inferSchema=True)
print(customers.columns)   # the column names: no row is sent to Python

Three ways to look at a DataFrame: printSchema() prints the schema, each column with its type (nullable = true means a value may be missing); show(5) asks Spark for 5 rows and prints them as a table, like head() in pandas; count() makes Spark go through every row and send back one number. orders.csv has one row per order line: an order with 3 products has 3 lines. Your program must print:

root
 |-- order_id: integer (nullable = true)
 |-- customer_id: integer (nullable = true)
 |-- date: date (nullable = true)
 |-- product: string (nullable = true)
 |-- category: string (nullable = true)
 |-- quantity: integer (nullable = true)
 |-- unit_price: integer (nullable = true)
 |-- status: string (nullable = true)

+--------+-----------+----------+--------+--------+--------+----------+------+
|order_id|customer_id|      date| product|category|quantity|unit_price|status|
+--------+-----------+----------+--------+--------+--------+----------+------+
|    1001|          1|2026-10-01|    Lamp|    home|       5|        29|  paid|
|    1001|          1|2026-10-01|     Mug| kitchen|       2|        12|  paid|
|    1001|          1|2026-10-01|     Pen|  office|       3|         2|  paid|
|    1002|          1|2026-11-10|   Plant|    home|       2|        15|  paid|
|    1002|          1|2026-11-10|Notebook|  office|       1|         7|  paid|
+--------+-----------+----------+--------+--------+--------+----------+------+
only showing top 5 rows
1497

A Check takes about 10 seconds here, because Spark starts a Java program before your code runs.

Your task: below the spark line, read orders.csv into a DataFrame named orders, with its first line as the header and the types guessed. Then print its schema, its first 5 rows and its number of rows.

Assignment
Follow the Coding Tutorial to read orders.csv into a Spark DataFrame and look at it with printSchema(), show(5) and count().

Hint
The read needs both header=True and inferSchema=True. printSchema() and show(5) print by themselves; count() returns a number, so wrap it in print().