An Eval Report in Python

Your eval runs twenty cases with three scorers, and all it leaves behind is lines scrolling past on a screen. A report is the same information in a file, made to be read later and shared: the totals first, then one row per case with its id, its result (PASS or FAIL) and a reason.

Markdown: the report is plain text with a few marks, which GitHub, chat tools and most editors show as a formatted page. A line that starts with # is a heading and - starts a list item. A table is one line per row, with | between the cells; its first two lines are the column names and a line of dashes.

# Shopping list

- Bread: 2
- Milk: 1

| item | price |
| --- | --- |
| Bread | 3 |

Writing a file: f.write does not end the line for you the way print does. You end it yourself with \n:

with open("notes.md", "w") as f:
    f.write("# Notes\n\n")
    f.write("- Buy tea\n")

The first \n ends the title line, and the second leaves an empty line under it. A with block closes its file when it ends, so after the block everything you wrote is really in the file.

After the label lines of the last lesson, your program must print the report it wrote to report.md:

# Eval report

- Labels: 8/10 (80%)
- Tool calls: 3/6 (50%)
- Answers: 2/4 (50%)

| id | result | reason |
| --- | --- | --- |
| c1 | PASS |  |
| c2 | PASS |  |
| c3 | FAIL | expected shipping, got refund |
…
| a4 | FAIL | It gives the 3 to 5 days, but it also promises free delivery. |

Your task: the starting code has a list rows and a function record(case_id, result, reason) that adds one row to it; the label cases and the tool cases already call it. Record the judged answers too, with the judge's reason. Then write report.md: the title, the three totals and the table with one line per row. Open the file again and print it.

Assignment
Follow the Coding Tutorial to record every case as a row, write the report to report.md in Markdown and print the file.

Hint
Every f.write(...) needs its own "\n" at the end. The block that prints the file starts at the left edge, below the block that writes it, so the file is closed before you read it. Mind the spaces around every |.