Canonical flow
Support response agent
Evaluator maturity pipeline for support replies.
Abstract pattern
Evaluator maturity pipeline
The evaluator-maturity example showing how the gate itself can evolve from an expensive judge into a distilled classifier while the worker stays the same.
- Assisted Human-led with automation support.
- HITL Human approves each action.
- HOTL Human samples or monitors.
- Autonomous Automation acts within guardrails.
Task contract
Support response agent
The evaluator-maturity example showing how the gate itself can evolve from an expensive judge into a distilled classifier while the worker stays the same.
- Boundary
- Input is the ticket, customer context, and knowledge base. Output is a response draft.
- Evidence log
- Ticket, draft, judge verdict, human edit or approval, and customer reaction such as resolved, reopened, or escalated.
- Evaluator
-
- Target
- Output
- Technique
- LLM judge first, then a classifier distilled from the judge's labels once enough examples accumulate.
- Oracle
- Human-gold support-agent edits and approvals validate the judge before the judge is used to label data for distillation.
- Position policy
-
- hitl
- blocking — every draft is held for human approval until judge agreement and reopen rate are proven stable
- hotl
- sampling — sampled monitoring replaces blocking review once the judge meets the bar; novel and sensitive tickets remain gated
- Promotion rule
- Judge agreement with human approvals stays above threshold and reopen rate stays low, which allows sampled monitoring instead of blocking every send.
- Demotion
-
- Rule
- Reopen or escalation rate rises, or the distilled classifier drifts from fresh human gold, which demotes both the task and the evaluator.
- Fallback
- Human approval stays in place for anything outside the routine ticket classes.
Teaching point
What this flow proves
Eval-the-eval is the point here, because the evaluator is validated, promoted, demoted, and eventually made cheap enough to own the gate.
Teaching context
Why this flow is governed this way
- Worker context
- The LLM that drafts the response.
- Without PAA
- Responses go out without a systematic quality gate; the model's quality is invisible until a customer complains; there is no audit trail distinguishing what the model produced from what a human approved.
- Never full-auto
- Anything outside the routine ticket classes — novel, sensitive, or escalating tickets stay human-gated as a permanent design constraint, not a classifier gap.
This page is linked from the canonical card set on the flows index.