Skip to content

Evaluator-Optimizer

Level: Guided. See the patterns and their levels.

Also called: review loop, generator–critic, maker-checker, iterative refinement.

The evaluator-optimizer pattern uses iterative refinement through two components: a generator that produces output and an evaluator that assesses it against predefined criteria and gives specific feedback. The workflow cycles between generation and evaluation until the output meets the quality bar or a maximum number of rounds is reached.

Think of it like a writer and editor working together. The writer produces a draft, the editor marks it up with specific feedback (“this paragraph is unclear,” “this claim needs a citation”), and the writer revises. They go back and forth until the piece is ready.

The evaluator’s grade decides what happens next (accept, or send back), and it grades by your criteria. That makes this pattern Guided.

  • High-quality outputs — The evaluator catches and refines suboptimal results before delivery
  • Measurable improvement — Each round is scored, so you can see whether a revision helped
  • Adaptable to complex tasks — Ideal for nuanced or multi-faceted outputs that benefit from incremental improvement
  • Checks the generator can’t make — The evaluator can run tests, validate against a schema, or compare against reference examples

A separate evaluator has an advantage over self-critique. In Reflection, one agent reviews its own work. A separate evaluator, with its own instructions and tools, judges work it didn’t write, so it’s less likely to share the generator’s blind spots.

┌───────────┐ output ┌───────────┐
│ Generator │─────────────────▸│ Evaluator │
│ │◂─────────────────│ │
└───────────┘ feedback + └───────────┘
score │
▲ │
│ score ≥ threshold? │
│ ├── YES → final output │
│ └── NO → loop back ────┘
└────────────────────────────────┘
  1. Generate — The generator produces an initial output from the task input.
  2. Evaluate — The evaluator scores the output against defined criteria and gives specific, actionable feedback.
  3. Check — If the score meets the threshold, or the round cap is reached, output the final version. Otherwise, continue.
  4. Revise — The generator receives the feedback and produces an improved version.
  5. Repeat — Steps 2–4 loop until the quality bar is met or the cap stops the loop.

Feedback must be specific. Not “this needs improvement” but “paragraph 3 contradicts the data in paragraph 1; the conclusion doesn’t follow from the evidence presented.” The more specific the feedback, the better the revision.

Every loop needs a cap. Set a maximum number of rounds, and decide what happens when the cap is reached before the output passes (deliver the best version with a note, or hand it to a person).

ComponentPurposeExample
GeneratorProduces an initial solution or responseDrafts a marketing tagline for a product
EvaluatorReviews output against evaluation criteria and provides actionable feedbackAssesses whether the tagline aligns with brand voice and audience preferences
Feedback LoopIteratively refines output until it meets criteriaIf the tagline misses brand guidelines, the evaluator provides specific improvement guidance
Accepted OutputDelivers the finalized result once criteria are satisfiedA polished tagline ready for the campaign
  • Clear evaluation criteria — When you can define measurable standards for what “good” looks like (test pass rate, rubric score, schema compliance)
  • High-stakes outputs — When accuracy and quality are critical (legal documents, strategic reports)
  • Worth the extra rounds — When the cost of iteration is lower than the cost of a wrong first answer
  • A natural stopping point — When you can define a termination condition (score threshold or round cap)
  • Quality criteria are subjective or undefined — the evaluator can’t provide useful feedback without clear criteria
  • The task is simple enough that single-pass generation is sufficient
  • Iteration costs (tokens, time, API calls) aren’t justified by the quality improvement
  • The generator can’t meaningfully improve from the feedback (if the feedback is “this is wrong” without specifics)
  • No natural termination condition — the loop may never converge

A company needs compelling marketing content (taglines, ad copy, social posts) that must reflect brand tone, resonate with the audience, and meet platform guidelines:

  1. Generator — Produces initial drafts of taglines, ad copy, and social media posts
  2. Evaluator — Reviews drafts against brand tone, audience fit, and platform requirements; provides actionable feedback (“Make the tagline more emotionally engaging for the target audience”)
  3. Feedback Loop — Refines outputs until all criteria are met, or three rounds have run
  4. Accepted Output — Polished, ready-to-use marketing content aligned with campaign goals

A documentation system for API references:

  • Generator — Writes endpoint documentation from the source code and inline comments.
  • Evaluator — Checks against a rubric: (1) all parameters documented, (2) request/response examples included, (3) error codes listed, (4) description matches actual behavior. Scores 0–4.
  • Loop — The generator revises until the evaluator scores 4/4, with a cap on rounds.

A coding assistant producing a function to specification:

  • Generator — Writes the function based on the requirements and type signatures.
  • Evaluator — Runs the test suite, checks type safety, measures code complexity. Returns: tests passed (8/10), type errors (1), complexity score (acceptable).
  • Loop — The generator fixes the 2 failing tests and 1 type error. Next round: 10/10 tests pass, 0 type errors. Done.

The evaluator has a concrete advantage here — it can actually run the code, something the generator can’t do while writing it.

  1. Define evaluation criteria — Establish clear, measurable standards (quality, alignment, tone)
  2. Set up the feedback loop — Pass the evaluator’s specific feedback back to the generator for each revision
  3. Set iteration limits — Prevent infinite loops by capping the number of refinement cycles, and decide what happens at the cap
  4. Test the workflow — Validate that evaluator feedback effectively guides the generator toward desired outcomes
  • Anthropic — Building effective agents (December 19, 2024) — “Workflow: Evaluator-optimizer”, which works best “when we have clear evaluation criteria”
  • Microsoft — AI Agent Orchestration Patterns, Azure Architecture Center (accessed 2026-10-24) — “Maker-checker loops”, also known as evaluator-optimizer, which repeat “until the checker approves the result or the orchestration reaches a maximum iteration limit”
  • Andrew Ng — Agentic Design Patterns Part 2, Reflection (The Batch, March 27, 2024) — one agent to generate and a second to critique