The Schema Travels with the File in Python

A CSV has no types and no row count: to learn a column's type, pandas reads the values and guesses, and to learn how many rows there are, you count every line. A Parquet file writes both down. At its end sits a small part called the footer, saved with the file: the schema (every column's name and type), the number of rows, and how the rows are split into row groups, blocks of rows stored together (a big file has many).

The schema: with import pyarrow.parquet as pq, pq.read_schema(path) reads only the footer and returns the schema, without reading any row. Each item of it is a field, one column, with field.name and field.type. Arrow, the library behind pq, calls text large_string and a date with a time timestamp[us] (to the microsecond). For a file customers.parquet, these lines print the list of column names, then the type of one column:

s = pq.read_schema("customers.parquet")
print(s.names)
print(s.field("age").type)

The footer's numbers: pq.ParquetFile(path).metadata gives the footer as an object: num_rows is the row count stored when the file was written, num_row_groups the number of row groups, num_columns the number of columns. Nothing is counted when you ask. A data engineer needs this because a pipeline checks every new file before loading it (are the expected columns there, with the expected types, and how many rows came in?): with Parquet that check reads only the footer, even for a file of many gigabytes, where a CSV would need a full read.

The starting code turns date into real dates with pd.to_datetime and writes orders.parquet. Your code must print:

order_id int64
customer_id int64
date timestamp[us]
product large_string
category large_string
quantity int64
unit_price int64
status large_string
rows: 1497
row groups: 1

Your task: read the schema of orders.parquet into schema, print each field's name and type in a loop, then keep the file's metadata in meta and print its row count and its number of row groups.

Assignment
Follow the Coding Tutorial to print the schema, the row count and the row groups of orders.parquet, all read from its footer.

Hint
The print inside the loop is indented under for field in schema:; meta = … starts at the left edge again. Each of the last two prints takes a label in quotes, a comma, then the value.