Narrative flow profile
Support response agent
Evaluator maturity pipeline for support replies.
Abstract pattern
Evaluator maturity pipeline
Shows how an expensive evaluator can be replaced by a validated classifier while the system producing replies stays the same.
- Manual Instrumented boundary; human performs the governed effect.
- HITL Automation proposes or prepares the effect and a human authorizes it before execution.
- HOTL The effect executes with asynchronous human oversight.
- Autonomous The effect executes under declared evaluator and demotion policy.
Narrative task profile
Support response agent
Shows how an expensive evaluator can be replaced by a validated classifier while the system producing replies stays the same.
- Boundary
- Input is the ticket, customer context, and knowledge base. Output is a response draft.
- Evidence log
- Ticket, draft, judge verdict, human edit or approval, and customer reaction such as resolved, reopened, or escalated.
- Evaluator
-
- Property
- response_quality
- Target
- Output
- Technique
- LLM judge first, then a classifier distilled from the judge's labels once enough examples accumulate.
- Evaluation basis
- human_gold: support_agent_edit_and_approval_human_gold
- Epistemic status
- ground_truth
- Version
- 1
- Authority
- blocking
- Placement notes
-
- hitl
- blocking, every draft is held for human approval until judge agreement and reopen rate are proven stable
- hotl
- sampling, sampled monitoring replaces blocking review once the judge meets the bar; novel and sensitive tickets remain gated
- Promotion rule
- Judge agreement with human approvals stays above threshold and reopen rate stays low, which allows sampled monitoring instead of blocking every send.
- Demotion
-
- Rule
- Reopen or escalation rate rises, or the distilled classifier drifts from fresh human gold, which demotes both the task and the evaluator.
- Fallback
- Human approval stays in place for anything outside the routine ticket classes.
Teaching point
What this flow proves
Eval-the-eval is the point here, because the evaluator is validated, promoted, demoted, and eventually made cheap enough to own the gate.
Teaching context
Why this flow is governed this way
- Worker context
- The LLM that drafts the response.
- Without PAA
- Responses go out without a systematic quality gate; the model's quality is invisible until a customer complains; there is no audit trail distinguishing what the model produced from what a human approved.
- Never full-auto
- Anything outside the routine ticket classes, novel, sensitive, or escalating tickets stay human-gated as a permanent design constraint, not a classifier gap.
This page is linked from the canonical card set on the flows index.