AI coding agents are becoming very good at producing large amounts of code and content.

When an agent changes 20 lines of code, th. You run the application. You spot a mistake. You ask the agent to fix it.

But what happens when an agent modifies 120 files?

Or generates 1,000 coding tutorials?

Or produces 100,000 lines of code and content?

Suddenly, the human becomes the bottleneck.

This is a problem I have been facing while building AlgoCademy, an interactive platform that teaches programming and algorithmic thinking through step-by-step coding tutorials.

A tutorial might teach Two Sum like this:

  1. Create a map.
  2. Write an empty for loop.
  3. Calculate the complement.
  4. Check whether the complement exists in the map.
  5. Store the current number.
  6. Return the matching indexes.

AI agents are surprisingly good at generating this type of content.

The problem is volume.

Imagine an agent generates 500 tutorials with 10 steps each.

That is 5,000 individual tutorial steps.

Even if reviewing one step takes only 20 seconds, manually checking everything would require almost 28 hours of uninterrupted work.

At that point, what exactly have we automated?

This led me to a much more important question:

Can AI agents produce large amounts of code or content without a human checking every single output?

I think the answer is yes.

But not by making the agent more careful.

The real solution is to build a system that is good at detecting bad output.

The Wrong Goal: Make the Agent Stop Making Mistakes

The natural response to unreliable AI output is to improve the prompt.

You start with:

Generate a step-by-step coding tutorial.

Then you find a mistake.

So you add:

Make sure every step is technically correct.

Then another mistake appears.

Do not introduce variables before explaining them.

Then:

Make sure the code from each step is consistent with the previous step.

Then:

VERY IMPORTANT: Never reveal code from a future step.

Six months later, your system prompt looks like the terms and conditions for a mortgage.

And the agent still occasionally makes mistakes.

I am not saying prompts do not matter. They obviously do.

But there is a fundamental difference between:

Reducing the probability of an error.

and:

Building a system that detects errors when they happen.

For serious automation, you need both.

Traditional software engineering already works this way.

We do not assume programmers never create bugs.

We use:

The developer is not the entire reliability system.

The engineering process is the reliability system.

AI agents need the same thing.

Stop Reviewing AI Output. Start Building an AI CI Pipeline

The simplest AI workflow looks like this:

AI generates artifact
        ↓
Human reviews artifact
        ↓
Publish

The problem is that human review scales linearly with AI output.

Generate ten times more content and you create ten times more review work.

A better architecture looks like this:

AI generates artifact
        ↓
Deterministic checks
        ↓
Semantic evaluators
        ↓
Adversarial execution
        ↓
Automatic repair
        ↓
Risk assessment
        ↓
Publish or escalate to human

The human no longer checks every artifact.

The human designs and improves the system that decides which artifacts are safe.

This is very close to how current agent-evaluation guidance is evolving. Anthropic describes agent evaluations as commonly combining code-based, model-based, and human graders, and explicitly recommends deterministic graders where possible, model graders where necessary, and human review as an additional validation layer. escribes a practical agent eval as:

prompt
   ↓
captured run: trace + artifacts
   ↓
checks
   ↓
score

The goal is to turn “this seems better” into concrete, repeatable tests. of this as AI CI.

Traditional CI asks:

Does the code compile?
Do the tests pass?
Did linting succeed?

AI CI might ask:

Is the artifact structurally valid?
Is the code correct?
Did the output violate a rule?
Does the instruction match the generated code?
Does the artifact depend on knowledge it should not require?
Can another agent successfully execute the instructions?
How much disagreement exists between our evaluators?

The important mental shift is this:

Do not try to make the generator trustworthy enough that validation becomes unnecessary.

Build validation assuming the generator will eventually fail.

My Specific Problem: Step-by-Step Coding Tutorials

Let’s use AlgoCademy as a concrete example.

The final Two Sum solution might be:

function twoSum(nums, target) {
    const seen = new Map();

    for (let i = 0; i < nums.length; i++) {
        const complement = target - nums[i];

        if (seen.has(complement)) {
            return [seen.get(complement), i];
        }

        seen.set(nums[i], i);
    }

    return [];
}

An AI can explain this solution in a few seconds.

But AlgoCademy does not just show the final solution.

The student might start with:

function twoSum(nums, target) {

}

Step 1:

Create a Map called seen.

The expected transformation is:

function twoSum(nums, target) {
    const seen = new Map();
}

Step 2:

Create an empty for loop that iterates through the indexes of nums.

The code becomes:

function twoSum(nums, target) {
    const seen = new Map();

    for (let i = 0; i < nums.length; i++) {

    }
}

Step 3:

Inside the loop, calculate the difference between target and the current number. Store it in a variable called complement.

And so on.

An AI agent can create a tutorial that looks perfectly reasonable while still making subtle mistakes.

It might:

A normal unit test cannot detect all of these problems.

But that does not mean a human has to inspect every tutorial.

It means different problems require different validators.

First: Turn Content Into Structured Artifacts

Before validating AI-generated content, make the content easy to validate.

Do not ask an agent to return one giant Markdown document containing instructions, explanations, and code blocks.

Instead, ask it to return structured data.

For example:

{
  "problemId": "two-sum",
  "language": "javascript",
  "concepts": [
    "arrays",
    "hash-maps"
  ],
  "initialCode": "function twoSum(nums, target) {\n\n}",
  "finalCode": "...",
  "steps": [
    {
      "id": "create-map",
      "instruction": "Create a Map called `seen`.",
      "concept": "hash-map",
      "previousCode": "...",
      "expectedCode": "...",
      "introducedSymbols": [
        "seen"
      ]
    }
  ]
}

Now the tutorial is not just text.

It is an artifact with a contract.

You can enforce rules such as:

Every step must have a unique ID.

Every referenced concept must exist in the concept registry.

Every declared prerequisite must already be available to the student.

The final step must reach a valid solution.

A symbol cannot be marked as introduced twice.

None of these checks require an LLM.

Your application already knows whether two IDs are identical.

Do not ask Claude:

Are all step IDs unique?

Write:

const ids = tutorial.steps.map(step => step.id);

if (new Set(ids).size !== ids.length) {
    throw new Error("Duplicate step ID");
}

This is one of the most important rules I have learned while building AI workflows:

Never use an LLM to check something that ordinary code can check reliably.

LLMs should handle ambiguity.

Software should enforce invariants.

Think of Every Tutorial Step as a State Transition

A step-by-step coding tutorial is essentially a sequence of program states:

S0 → S1 → S2 → S3 → S4

Each tutorial instruction describes a transition:

Sn → Sn+1

For example:

Previous state:

function twoSum(nums, target) {
    const seen = new Map();
}

Next state:

function twoSum(nums, target) {
    const seen = new Map();

    for (let i = 0; i < nums.length; i++) {

    }
}

The transformation is approximately:

ADD ForStatement

initializer:
    let i = 0

condition:
    i < nums.length

increment:
    i++

The instruction says:

Create an empty for loop that iterates through the indexes of nums.

The instruction and transformation agree.

Now imagine the instruction says:

Loop through every number in nums.

But the generated code iterates over indexes.

That instruction is ambiguous.

A student might reasonably write:

for (const num of nums) {

}

The final algorithm may still be possible.

But the generated tutorial expects an index-based loop.

This is exactly the kind of problem a compiler will not catch.

So instead of asking:

Is this a good tutorial step?

I can ask a much narrower question:

Does the instruction accurately describe the transformation from the previous program state to the next program state?

That is a much easier evaluation problem.

But Do Not Require Exact AST Equality

This is an important detail.

My first instinct was to compare the student’s Abstract Syntax Tree with the expected AST.

For example:

student AST === expected AST

I now think that is too rigid for many coding tutorials.

Consider:

for (let i = 0; i < nums.length; i++) {
}

and:

for (let index = 0; index < nums.length; index++) {
}

The ASTs are different.

The transformation may be pedagogically equivalent.

Or consider:

const complement = target - nums[i];

versus:

const needed = target - nums[i];

Again, different structure.

Potentially the same valid idea.

Anthropic’s evaluation guidance explicitly warns that deterministic graders can be brittle when valid variations do not match the expected pattern exactly. It also cautions against overly rigid process checks that punish valid approaches an evaluator did not anticipate. etter approach is not:

Does the AST exactly equal our expected AST?

It is:

Does the program satisfy the structural contract for this step?

For example:

A loop was added.

The loop traverses nums.

The loop exposes the current index.

The loop body is empty.

Existing required statements were not removed.

The validator might accept several AST patterns that satisfy those properties.

This is a major difference.

You are validating invariants, not formatting.

For AlgoCademy, this is especially important because a coding-learning platform should avoid teaching students that there is only one textual way to write correct code.

Make Evaluations Atomic

One of the worst evaluators you can create is:

Review this tutorial.

Make sure it is:
- correct
- clear
- educational
- beginner friendly
- progressive
- technically accurate

Score it from 1 to 10.

Suppose the result is:

8/10

What do I do with that?

Why was it an eight?

What failed?

Can an agent automatically repair it?

Did version two improve?

Instead, I want atomic validators.

For example:

V001: Tutorial schema is valid

V002: Step IDs are unique

V003: Final solution passes hidden tests

V004: Required checkpoints parse successfully

V005: Symbols are introduced before they are referenced

V006: Instruction matches the code transformation

V007: No unrelated code is modified

V008: Step introduces at most one primary learning objective

V009: Explanation does not reveal future implementation details

V010: Terminology matches the curriculum

V011: Variable references are consistent

V012: Required concepts were previously introduced

V013: The tutorial reaches a correct solution

V014: Complexity explanation matches the implementation

V015: A student can reasonably perform the requested transformation

The result might be:

{
  "V001": "PASS",
  "V002": "PASS",
  "V003": "PASS",
  "V004": "PASS",
  "V005": "PASS",
  "V006": "FAIL",
  "V007": "PASS",
  "V008": "PASS",
  "V009": "PASS",
  "V010": "PASS",
  "V011": "PASS",
  "V012": "PASS",
  "V013": "PASS",
  "V014": "PASS",
  "V015": {
    "score": 0.81
  }
}

Now we know exactly what happened.

This is the agent equivalent of:

48 tests passed
2 tests failed

Instead of:

Your code feels pretty good.

OpenAI’s current eval guidance similarly emphasizes defining measurable success and using a small set of concrete checks rather than relying on whether a new version merely “feels better.” alidate Code With Code

For coding education, we have an enormous advantage.

Code is much easier to validate than most AI-generated content.

Use that advantage aggressively.

For each final solution, I can run:

Public examples
Hidden tests
Edge cases
Randomly generated tests
Reference implementations

Consider Two Sum.

Generate:

const nums = [-3, 7, 2, 11, 5];
const target = 9;

Run both:

generated solution
reference solution

Then validate the result.

For problems with several valid answers, I do not necessarily need an exact expected output.

I can validate properties.

For example:

function isValidTwoSumAnswer(
    nums: number[],
    target: number,
    result: number[]
): boolean {
    if (result.length !== 2) {
        return false;
    }

    const [i, j] = result;

    if (
        !Number.isInteger(i) ||
        !Number.isInteger(j) ||
        i < 0 ||
        j < 0 ||
        i >= nums.length ||
        j >= nums.length
    ) {
        return false;
    }

    if (i === j) {
        return false;
    }

    return nums[i] + nums[j] === target;
}

The validator does not know the exact answer.

It knows what must be true about a correct answer.

This leads to a much broader principle for AI systems:

Instead of asking whether the AI produced exactly what you expected, identify the properties every acceptable result must satisfy.

The more quality requirements you can convert into invariants, the less human review you need.

Not Every Intermediate Step Has to Compile

There is another subtle issue with step-by-step coding education.

It is tempting to create this rule:

Every intermediate code state must compile.

That may work for a specific tutorial format.

It is not universally correct.

Some educational systems intentionally let a learner stop at a partially complete expression or temporarily incomplete block.

For example, a lesson could ask the student to first write:

if (

before teaching the condition.

A normal JavaScript parser will reject that state.

You have several choices.

You can design AlgoCademy so every completed step is a valid parsing checkpoint.

Or you can use a tolerant parser.

Or you can validate the edited region without requiring the whole program to execute.

The important part is that this should be an explicit product rule.

Do not accidentally turn:

Our current examples happen to parse.

into:

Every educational programming step must always compile.

Your validator should enforce your pedagogy, not silently define it.

Use LLM Judges Only Where Rules Become Semantic

Some questions are difficult to answer with ordinary code.

For example:

Is this instruction ambiguous?

Does this step require a concept the student probably does not understand yet?

Does the explanation accidentally reveal how to solve the next step?

Is this learning step too cognitively large?

These are reasonable jobs for an LLM evaluator.

But there is a trap.

Do not simply ask:

Is this a good instruction?

Instead, give the evaluator a narrow rubric.

For example:

The student knows:
- variables
- arrays
- basic for loops

The student does NOT know:
- Maps
- Sets
- destructuring

Instruction:
[INSTRUCTION]

Does completing this instruction require a programming concept
outside the student's known concepts?

Return JSON:

{
  "pass": boolean,
  "unknownConcepts": [],
  "evidence": []
}

The judge has one job.

I generally prefer binary or narrowly scored questions over broad quality scores.

Compare:

Score the tutorial's consistency from 1 to 10.

with:

Does the instruction reference a symbol that is unavailable
in the previous program state?

The second result is much more useful.

Do Not Blindly Trust AI to Judge AI

LLM judges are useful.

They are not objective truth machines.

Research on LLM-as-a-judge has documented limitations including position, verbosity, and self-enhancement biases. s not mean we should avoid model-based evaluators.

It means we should treat them as imperfect components.

Some things I would do:

Give each evaluator one narrow job

A pedagogy evaluator should evaluate pedagogy.

A technical correctness evaluator should evaluate technical correctness.

Do not create:

SUPER REVIEW AGENT

Check absolutely everything.

That is just recreating the original context problem.

Require structured evidence

Instead of:

{
  "score": 0.72
}

I prefer:

{
  "pass": false,
  "stepId": "calculate-complement",
  "rule": "FUTURE_STEP_LEAKAGE",
  "evidence": [
    "The explanation references seen.has(), which is introduced in the next step."
  ]
}

Measure evaluator disagreement

Suppose three evaluators return:

PASS
PASS
PASS

That is different from:

PASS
PASS
FAIL

Even if your majority-vote result is still PASS.

Disagreement itself is information.

Calibrate judges against humans

Take a manually reviewed dataset.

Run your model judge.

Measure what it misses.

A semantic judge should earn trust based on its performance on your specific tutorial defects.

Do not trust it merely because you used an expensive model.

Build a Simulated Student Agent

This is the validator I find most interesting for AlgoCademy.

Instead of asking:

Is the instruction clear?

Try to execute the instruction.

Give an agent only:

Student knowledge
Current code
Tutorial instruction

Do not show it the expected solution.

Then say:

Modify the code by following the instruction.

Imagine the instruction is:

Create a loop through nums.

The expected tutorial transformation uses:

for (let i = 0; i < nums.length; i++) {

}

But the simulated student writes:

for (const num of nums) {

}

This does not automatically mean the student’s answer is wrong.

In fact, that is the important part.

It may mean the instruction allows more than one reasonable interpretation.

The evaluator can compare the student’s transformation with the step contract:

Required:
- traverse nums
- expose the current index

Student result:
- traverses nums: YES
- exposes current index: NO

Now ask:

Did the student ignore an explicit requirement, or did the instruction fail to communicate the requirement?

If “index” never appears in the instruction, the problem may be the tutorial.

The output could be:

{
  "result": "AMBIGUOUS_INSTRUCTION",
  "requiredProperty": "INDEX_AVAILABLE",
  "studentInterpretation": "VALUE_ITERATION",
  "reason": "The instruction says to loop through nums but does not specify index-based iteration."
}

This is much better than asking another AI:

Do you think this instruction is clear?

We attempted to follow the instruction and observed what happened.

This leads to one of my favorite ideas for evaluating AI-generated workflows:

Do not only review instructions. Execute them.

If an AI creates installation instructions, run them in a clean container.

If an AI writes API documentation, give the documentation to an agent that has never seen the implementation and ask it to integrate the API.

If an AI creates a migration plan, execute the plan in a disposable environment.

If an AI creates a coding tutorial step, simulate a student performing the step.

Execution is an evaluator.

Build an Automatic Repair Loop

Once your validators are atomic, repair becomes much easier.

A bad repair prompt looks like:

There are a few issues with the tutorial.

Please improve it.

The agent may rewrite half the tutorial.

It may fix one problem and introduce three new ones.

Instead, give it exact failures:

{
  "failures": [
    {
      "validator": "V006",
      "stepId": "iterate-array",
      "error": "The instruction does not specify that the loop must expose the current index."
    },
    {
      "validator": "V009",
      "stepId": "calculate-complement",
      "error": "The explanation references seen.has(), which is introduced in a future step."
    }
  ]
}

Then constrain the repair agent:

Fix only the reported failures.

Do not modify passing steps unless a modification is required
to maintain consistency.

Return a patch containing:
- changed step IDs
- old values
- new values
- validator failures addressed

Then run the entire validation suite again.

The loop becomes:

GENERATE
   ↓
VALIDATE
   ↓
PASS? ───────────────→ ACCEPT
   │
   NO
   ↓
REPAIR
   ↓
VALIDATE
   ↓
PASS? ───────────────→ ACCEPT
   │
   NO
   ↓
REPAIR

But set a limit.

For example:

Generation: attempt 0
Repair: attempt 1
Repair: attempt 2
Repair: attempt 3
Human escalation

Do not let an agent repair forever.

An infinite agent loop is not autonomy.

It is an expensive while (true).

OpenAI has published an example of a broader agent-improvement flywheel where traces, human feedback, and model feedback are converted into reusable evals and then into changes to the surrounding harness. In that example, the harness is explicitly defined as the contract around the model: instructions, tools, routing, output requirements, and validation checks. inition is very close to how I now think about these systems.

The Secret to Huge Outputs: Never Review the Huge Output as One Thing

Suppose Claude Code modifies 120 files.

One of the worst review prompts is:

Review all 120 files carefully and tell me whether everything is correct.

You have recreated the original problem.

The review agent now has almost the same context problem as the generation agent.

Instead, break the artifact into the smallest meaningful evaluation units.

For a repository:

Repository
   ↓
Package
   ↓
Module
   ↓
File
   ↓
Function
   ↓
Invariant

For AlgoCademy:

Curriculum
   ↓
Track
   ↓
Tutorial
   ↓
Step
   ↓
State transition

Then validate at several levels.

Step-level

Does the instruction describe the required code transformation?

Tutorial-level

Does the tutorial introduce concepts progressively?

Track-level

Does this tutorial require concepts that appear later in the track?

Curriculum-level

Are prerequisite dependencies acyclic and complete?

Most checks should receive the minimum context required for their decision.

This is one reason subagents can be useful in Claude Code. Anthropic’s current documentation explicitly describes subagents as separate context windows with focused prompts and tool access, and recommends them for isolating high-volume operations so logs, file reads, and test output do not flood the main conversation. er principle is not specific to Claude Code:

Context should be treated as a dependency problem, not an ever-growing chat history.

Artifacts Are Better Than Agent Memory

It is tempting to keep one enormous Claude Code or Codex session running.

The agent has all the context.

At first, this feels powerful.

Eventually, the session contains:

200 file reads
50 tool calls
17 failed approaches
8 temporary scripts
3 architecture changes
several abandoned assumptions

Everything may technically be somewhere in the context.

That does not mean the context is useful.

For high-volume workflows, I prefer explicit artifacts.

For example:

/tutorials/two-sum/spec.json
/tutorials/two-sum/generated.json
/tutorials/two-sum/validation.json
/tutorials/two-sum/repair-1.json
/tutorials/two-sum/final.json

Along with shared definitions:

/evals/tutorial-step-rules.json
/concepts/registry.json
/languages/javascript.json
/reference-solutions/two-sum.js

The repair agent does not need the generator’s entire conversation.

It receives:

Input artifact
Validation failures
Repair contract
Output schema

That’s it.

Instead of:

Remember the issue we discussed 45 minutes ago with step 17?

The workflow says:

Read validation.json.

Repair BLOCKER failures.

Write repair-1.json.

The source of truth lives outside the model.

The model is a worker operating on artifacts.

That is a far more scalable architecture.

Claude Code Should Not Be the Human Workflow Hidden in a Terminal

Claude Code can already support pieces of this architecture.

Its documentation describes non-interactive execution for CI and batch processing. Its hook system supports automatic lifecycle actions, including command-based checks and agent-based verification. Its Agent SDK exposes the same general tools, agent loop, and context-management model programmatically in Python and TypeScript. e is still an important distinction.

Imagine I use Claude Code like this:

Build 20 tutorials.

Now inspect them.

Run the tests.

Check the instructions.

Fix the problems.

Review them again.

Claude Code is doing a lot of work.

But the actual workflow still lives in my head.

I am the harness.

I decide:

when to test
what to inspect
what looks suspicious
when to ask for another review
when to retry
when the result is good enough

That works for interactive development.

It does not scale to thousands of artifacts.

The decisions need to move into software.

Conceptually:

const tutorial = await generateTutorial(spec);

const deterministicResults =
    await runDeterministicValidators(tutorial);

if (deterministicResults.hasBlocker) {
    return runRepairLoop(tutorial, deterministicResults);
}

const semanticResults =
    await runSemanticEvaluators(tutorial);

const simulationResults =
    await simulateStudent(tutorial);

const risk = calculateRisk({
    deterministicResults,
    semanticResults,
    simulationResults
});

if (risk >= HUMAN_REVIEW_THRESHOLD) {
    return sendToHumanReview(tutorial);
}

return publishTutorial(tutorial);

Claude Code, Codex, or another agent may do some of the work.

But the workflow exists outside the conversation.

That is the difference between:

I have an AI coding assistant.

and:

I have an AI production system.

Do Not Ask “Did It Pass?” Ask “How Likely Are We to Have Missed Something?”

Eventually, every artifact may have dozens of validation signals.

Some are hard requirements.

For example:

Schema valid                         REQUIRED
Final code passes tests              REQUIRED
No broken dependency references      REQUIRED
Required tutorial states exist        REQUIRED

Others are probabilistic:

Instruction clarity                  0.94
Pedagogical progression              0.91
Beginner solvability                 0.87
Future-step leakage                  0.03

I would be careful about making an LLM judge a hard blocker before calibrating it.

For example:

LLM says instruction is unclear
        ↓
ALWAYS REJECT

may produce too many false positives.

Instead, the release decision can combine signals:

Deterministic failures
Semantic scores
Evaluator disagreement
Number of repair attempts
Novel concepts
Tutorial difficulty
Historical defect rate for this template

Imagine:

Risk < 0.10
    Auto approve

Risk 0.10–0.30
    Run extra evaluation

Risk > 0.30
    Human review

Those numbers are only examples.

The thresholds need to be calibrated using real AlgoCademy tutorials and real defects.

The architecture is the important part.

You Still Need Humans, Just in a Different Place

I do not think the goal should be zero human involvement.

The goal should be to move human effort from:

Review every tutorial.

to:

Review the evaluation system.

Humans should inspect:

Imagine the system publishes 1,000 tutorials.

You manually audit 50.

You discover two bad tutorials.

The least valuable response is:

Fix those two tutorials.

Of course you should fix them.

But the more important question is:

Why did the harness approve them?

Suppose the first problem is:

The instruction says “store the previous value,” but there are two possible values it could mean.

Great.

Create:

V016: Ambiguous referent detection

The second problem is:

The tutorial uses modulo before modulo has been introduced.

Create:

V017: Concept prerequisite violation

Add both failures to your evaluation dataset.

Run the validators against every future tutorial.

Anthropic describes evals as a way to avoid purely reactive loops where problems are discovered only in production and individual fixes can create new regressions elsewhere. there is an even simpler way to remember this:

Every escaped AI mistake is a missing regression test.

Do not waste human-discovered failures.

Turn them into evals.

Build a Gold Dataset Before You Trust the Harness

Before automatically publishing thousands of tutorials, I would create a smaller collection of extremely well-reviewed tutorials.

For example:

50 beginner tutorials
30 intermediate tutorials
20 advanced tutorials

The exact numbers do not matter.

The quality and variety do.

Then intentionally create broken versions.

Examples:

Ambiguous instruction
Future-step leakage
Wrong variable name
Skipped learning step
Oversized step
Incorrect loop bound
Instruction/code mismatch
Broken state chain
Unknown concept dependency
Incorrect complexity explanation
Unnecessary code modification

Now you have:

Known good examples
Known bad examples
Known defect categories

Run your validators.

Measure:

True positives
False positives
False negatives

The most dangerous number is usually:

False negatives

These are known bad tutorials your harness approves.

Suppose you have 500 seeded defects.

Your system detects 490.

Ten escape.

Now you have ten concrete failures to study.

That is much more useful than saying:

Claude seems pretty good at reviewing tutorials.

Every time you change something, rerun the dataset.

Changed the generator prompt?

Run evals.

Changed the model?

Run evals.

Added a repair agent?

Run evals.

Changed the curriculum schema?

Run evals.

Changed the semantic judge?

Run evals.

Your AI workflow now has regression tests.

What I Would Actually Build for AlgoCademy

If I were designing the tutorial-generation workflow from scratch today, I would build this.

1. Tutorial Specification

Before generation:

{
  "problem": "two-sum",
  "language": "javascript",
  "studentLevel": "beginner",
  "knownConcepts": [
    "variables",
    "arrays",
    "for-loops"
  ],
  "conceptsToTeach": [
    "hash-maps"
  ],
  "targetSteps": {
    "min": 7,
    "max": 12
  },
  "referenceSolution": "..."
}

The generator receives a contract.

It does not invent the entire curriculum.

2. Structured Generation

The agent returns strict structured data.

No parsing arbitrary Markdown.

No guessing where an explanation ends and code begins.

3. Deterministic Validation

Run:

Schema validation
Code parsing where required
Code execution
Hidden tests
Dependency checks
State-chain validation
AST or syntax-tree analysis
Symbol validation
Concept-registry validation

4. Transformation Contracts

For every tutorial step, define the required change.

For example:

{
  "stepId": "create-index-loop",
  "requiredProperties": [
    "LOOP_ADDED",
    "TRAVERSES_NUMS",
    "CURRENT_INDEX_AVAILABLE"
  ],
  "forbiddenProperties": [
    "MAP_LOOKUP_ADDED",
    "RETURN_ADDED"
  ]
}

Validate properties instead of exact code equality.

5. Semantic Evaluators

Use narrow evaluators for:

Instruction ambiguity
Instruction/transformation consistency
Pedagogical step size
Concept difficulty
Future-step leakage
Explanation clarity

Require structured evidence.

6. Simulated Student

Give another agent:

Student knowledge
Current code
Instruction

Ask it to perform the step.

Check its result against the transformation contract.

Investigate reasonable alternative interpretations.

7. Repair Loop

Give the repair agent exact validator failures.

Require a patch.

Re-run the complete validator suite.

Limit the number of repair attempts.

8. Risk Routing

Auto-approve low-risk tutorials.

Run additional checks on uncertain tutorials.

Send high-risk tutorials to humans.

Randomly audit auto-approved content.

9. Evaluation Flywheel

Every student-reported or human-discovered defect becomes:

A regression example

and possibly:

A deterministic validator
A semantic rubric
A simulated-student scenario
A new adversarial test

Over time, the target is:

Human review rate ↓

Validator coverage ↑

Escaped defect rate ↓

That is what scalable AI generation should look like.

The Metrics I Would Track

I would not use:

Number of tutorials generated

as my primary success metric.

AI can generate enormous amounts of content.

Volume is no longer the hard problem.

I would track:

Auto-Approval Rate

automatically approved tutorials
/
generated tutorials

Human Escalation Rate

tutorials requiring human review
/
generated tutorials

Escaped Defect Rate

bad published tutorials
/
published tutorials

This may be the most important quality metric.

Repair Success Rate

automatically repaired tutorials
/
tutorials that initially failed

Average Repair Iterations

How many repair loops are needed before acceptance?

Evaluator Disagreement Rate

How frequently do semantic evaluators disagree?

False-Negative Rate

How many known defects in the gold dataset does the harness miss?

Student-Reported Defect Rate

How frequently do real students find problems the harness failed to detect?

Defect-to-Eval Conversion Rate

This is a metric I would seriously consider.

human-discovered defect categories converted into reusable evals
/
unique defect categories discovered

If the same class of mistake reaches a human five times, the evaluation system is not learning.

So, Is Fully Autonomous AI Content Generation Possible?

It depends on what we mean by autonomous.

If we mean:

Give Claude Code one prompt and let it generate 10,000 tutorials without external validation.

I would not trust that system.

If we mean:

Agents generate structured artifacts inside a constrained workflow with deterministic validation, semantic evals, execution-based tests, automatic repair, risk-based escalation, and statistical human auditing.

Yes.

I think that can become highly autonomous.

And there is an interesting paradox here.

The way to create more autonomous AI systems is to trust the AI less.

Assume the generator will eventually make a mistake.

Assume the AI reviewer will occasionally miss it.

Assume your prompt has blind spots.

Assume a valid solution exists that your original evaluator did not anticipate.

Then build independent mechanisms for finding those failures.

This is how reliable engineering has always worked.

A reliable distributed system is not reliable because no server ever crashes.

It is reliable because the architecture expects servers to crash.

AI systems will probably work the same way.

Do not try to build an agent that never makes mistakes.

Build a harness that expects mistakes, detects them, repairs what it can, and knows when to ask a human.

That, to me, is the difference between using an AI coding assistant and building an AI-native production system.