Property
The named property this evaluator produces a verdict about.
Evaluator primitive
Measure performance without confusing judgment with authority
An evaluator's ability to judge something and its authority within governance are different things. A classifier may be technically capable of identifying unsafe responses. That does not automatically mean its result may justify a promotion or block an action.
PAA therefore records both what an evaluator measures and the role its judgments are allowed to play. Multiple evaluators may judge the same property without having the same technique or authority.
This page explains how PAA identifies evaluators, when they run, what authority they have, and how replacements affect evidence. For the validated declaration contract, see Schema. For reusable patterns, see Flows. For where governance begins, see the Recorded decision: spectrum entry.
An unvalidated evaluator is not a gate. It is a guess with a UI.
An evaluator may produce a useful judgment before governance permits that judgment to block work or support promotion.
Evaluator fields
A score alone cannot tell you what was judged, how the judgment was produced, or what
decisions may rely on it. PAA records seven separate answers. Placement is not an
evaluator field; the task's position_policy determines when evaluation runs.
Target
The layer inspected: input, process, output, or outcome.
Technique
What produces the verdict: deterministic, classifier, llm_judge, or human.
Evaluation basis
The criteria, reference, or procedure that supports the verdict: invariant, reference_label, rubric, human_gold, or downstream_result.
Epistemic status
Whether the verdict is proxy or ground_truth.
Version
The identity of this evaluator instance. Comparable evaluators may assert the same property without sharing technique, evaluation basis, epistemic status, version, or authority.
Authority
Whether this evaluator's verdict can halt the governed effect: advisory or blocking. Placement, when it runs, comes from the task's position_policy, not from the evaluator.
Target
Evaluate at the cheapest layer that actually catches the failure.
| Layer | Question | Cost | Note |
|---|---|---|---|
| Input | Was the right context present and well-formed before acting? | Cheapest | Catches structural failures before work starts. |
| Process | Were the present inputs used correctly during generation? | Hard / often sealed | Usually inferred from output and outcome. |
| Output | Is the produced artifact good against some standard? | Needs an evaluation basis | The default eval layer. |
| Outcome | Did it work downstream? | Truest, most lagged | Noisy and delayed, but the real numerator. |
Technique
Four closed values. Ordered by cost, cheapest first.
| Technique | Examples | Cost / latency | Best use |
|---|---|---|---|
deterministic | Schemas, rules, bounds, assertions, allow/deny lists, reference validators | Zero | Structural correctness and hard invariants |
classifier | Learned models on embeddings, fine-tuned classifiers, distilled decision models | Cheap inference | The mature high-volume evaluator after sufficient label accumulation |
llm_judge | A model scores output against a rubric, with or without a reference | Expensive / latent | Bootstrap labels before distilling to cheaper evaluators |
human | Expert review, authorization decisions, quality labels | Highest | Ground truth for ambiguous or high-stakes cases; authorization gates |
Evaluation basis
The evaluation basis is the criteria, reference, or procedure that supports the
evaluator's verdict. It is a nested field, { kind, ref }, not a
single closed word. Five closed kind values:
| Kind | What it means |
|---|---|
invariant | A structural rule or assertion that must hold. |
reference_label | A comparison against a known reference label. |
rubric | Written criteria a technique applies to produce a verdict. A rubric-driven LLM judge is commonly proxy, rubric is the basis, proxy is the separately-declared epistemic status. |
human_gold | Human expert judgment used as the reference. Human gold ordinarily carries ground_truth status, but the two fields vary independently. |
downstream_result | A measured downstream outcome. |
Epistemic status
Epistemic status is a separate field from evaluation basis: it declares whether the verdict is an approximation of ground truth or ground truth itself. Two closed values:
proxy: The verdict is an approximation of ground truth.ground_truth: The verdict is treated as ground truth itself.
Evaluation basis and epistemic status vary independently. A rubric-driven LLM judge is
commonly proxy. Rubric is the basis; proxy is the separately declared
status. A human_gold basis ordinarily carries ground_truth
status, but the schema does not couple them: do not map proxy to
rubric as if one implies the other.
Authority
Authority is the gate effect of an evaluator: whether its verdict can halt the governed effect. Two closed values:
advisory: The verdict is recorded and surfaced but cannot halt the governed effect.blocking: When the task placement is blocking, the verdict can halt the current governed effect and all blocking evaluators must pass. With async or offline placement, the verdict can affect a later cycle or demotion instead.
Authority is not the same as technique. A human technique evaluator may be
advisory (post-execution quality review). A deterministic
evaluator may be blocking (structural invariant gate). Authority is also not
deployment: shadow is a task-level deployment mode, never an evaluator
authority value.
Multiple evaluators, one property
A property is not limited to one evaluator. response_quality commonly
appears twice on the same declaration: one version is an llm_judge against a
rubric, epistemic status proxy, authority advisory; a second
version is human against human_gold, epistemic status
ground_truth, a governance designation rather than a claim of infallibility;
authority is also advisory while the judge is being
validated. Both evaluators run and both are recorded. Comparable evaluators may assert
the same property without sharing technique, evaluation basis, epistemic status, version,
or authority.
Evaluator succession changes which version holds blocking authority, and it does so only after measured agreement between the candidate and the evaluator it would replace; it does not mutate historical evidence. Every evidence record retains the complete identity of the evaluator that produced it, so a later authority handoff never rewrites what an earlier version actually verified.
Placement (task level)
position_policy is a top-level task field, not an evaluator field. It maps
the task's current autonomy position to a placement value: blocking, async, offline.
The policy is fixed in the declaration; the task's current position changes at runtime.
Placement describes where an evaluator participates in task execution. The task declares it through position_policy.
Authority describes whether an evaluator's verdict can halt the governed effect.
'Blocking' appears in both vocabularies, but names different dimensions. Blocking placement means the workflow waits for evaluation before the effect. Blocking authority means that evaluator may prevent the effect when its verdict fails.
Placement does not imply authority, and authority does not imply placement.
A position may set one placement for every evaluator or a default with per-evaluator overrides. Those overrides remain part of the task declaration; evaluators do not declare them.
A position may declare a single placement for every evaluator, or a default
placement with explicit per-evaluator overrides. Overrides stay task-owned:
they are declared in position_policy, never on the evaluator, and exist for
the case where one position needs two placements at once. At HOTL the human evaluator
leaves the synchronous path while machine evaluators keep blocking; that is the definition
of the position, and a single value cannot express it. See
Schema: Position policy for the
selector form and its validation rules.
| Autonomy position | Typical placement value | Timing | Effect |
|---|---|---|---|
manual | offline | Batch / aggregate | Periodic evaluation; no per-execution gate. The human still performs the governed effect. |
hitl | blocking | Before execution | Halts the write until all blocking evaluators clear. |
hotl | async | After execution | Watches outputs and triggers demotion or alert. |
autonomous | offline | Batch / aggregate | Measures whether the task has earned more autonomy. |
Evaluator economics
The evaluator must cost less than the action it guards.
| Principle | Rule | Implication |
|---|---|---|
| Evaluator economics | The evaluator must cost less than the action it guards. | If the evaluator is more expensive than the task, the economics fail. |
| Cost curve | Expensive at cold start, cheap at volume. | A reviewer-heavy phase can fund itself because the volume is still low. |
| Distillation | Move from llm_judge to classifier once labels exist. | The accumulated review stream pays for the cheaper evaluator you will need later. |
| Boundary discipline | Instrument from the start. | The data must already be there when the evaluator needs to get cheaper. Instrumentation is the entry criterion for manual, the first on-spectrum position, not something added later. |
Maturity curve
Evaluator maturity describes how much responsibility the evaluation mechanism has earned. Task autonomy describes how much authority the worker has earned for a governed task. These dimensions can influence each other operationally, but they do not advance automatically together. Evaluator maturity does not automatically change a task's autonomy position.
| Stage | Evaluator setup | Evaluation role | Meaning |
|---|---|---|---|
| Cold start | human plus llm_judge; epistemic status is ground_truth or proxy pending validation | Advisory and heavily checked against human judgment | Not sufficient alone for authority-changing decisions. |
| Growing evidence | Distilled classifier begins to replace the judge; evaluation basis shifts toward reference_label or rubric signals | May support defined governance decisions after demonstrating agreement and stability | Review labels become validation and training data while authority remains explicit. |
| Mature volume | Cheap classifier on the hot path; proxy epistemic status validated against downstream outcomes | Trusted for a broader evaluation role within declared authority bounds | May reduce evaluation cost and human-review burden without granting worker authority. |
| Regression | Fallback to llm_judge or human after agreement or calibration drifts past the bar | Narrowed or suspended pending renewed validation | Evaluator authority and review burden change without automatically changing task autonomy. |
Metric choice
Optimize according to failure cost, not preference.
| Failure profile | Metric to favor | Why |
|---|---|---|
| Rare, high-cost failures | Recall | Misses are catastrophic, so escalate liberally. |
| Common, low-cost failures | Precision and throughput | False escalations become the expensive error. |
| Bootstrap labels | Human agreement | Use the judge to create the training set, not to pretend the evaluator is already proven. |
How an evaluator gets promoted and demoted
The evaluator itself follows a governed maturity loop. Its maturity can begin to be
measured once it is instrumented and its judgments can be compared with a defined
reference. A human_gold basis with ground_truth status is the usual
reference. The schema also permits human_gold with proxy status, but
that combination still represents an approximation and does not by itself provide a
ground-truth comparison. As agreement and calibration evidence accumulates, governance may
allow a cheaper technique to take on a defined evaluation role. If those measurements later
drift past the threshold, governance can narrow that role or return to human review. This
changes evaluator responsibility, not the task's autonomy position.
The same governance pattern applies recursively. Task autonomy, evaluator maturity, and worker capability move independently, each with its own evidence and transition rules.
This page describes evaluator maturity: review labels accumulate, a cheaper technique earns authority, and cost per verdict falls. Worker capability consumes the same labels to a different end: a better implementation behind an unchanged boundary, re-entering as a governed swap. Both draw on one stream, which is precisely why they must not advance in the same attribution window. See Grading for the stream, and Improvement for the worker-capability dimension.