Build PAA / Evidence & Labels
Review decisions become reusable labels
Every recorded review is a potential label. It can support promotion, train a cheaper evaluator, improve the worker, or expose a task boundary that is too broad, provided the judgment remains tied to the exact work it judged.
A task produces useful labels only when its review decisions are recorded.
Primer
Behavioral evidence records judgments about quality, reliability, safety, and failures. Economic evidence accompanies those judgments: worker and evaluator costs, human review, retries, escalation, and other relevant operating measurements. Together they help assess a defined configuration at its current or proposed oversight.
In the current contract, a consumer can carry cost and worker configuration identity in
the task-specific payload, typed by its payload_schema.
These are consumer-declared measurements, not top-level PAA fields. The evidence record's
producer identifies the verdict producer, not the worker. Conformance does not
require economic data, and runtime eligibility logic does not consume it.
Retain the underlying usage and pricing identities so task or pipeline totals can be attributed and repriced. Count each charge once even when several evaluators judge an output or a pipeline summary includes individual task costs. Keep failed and retried attempt costs, including attempts with no output. Missing cost is not zero.
Economic evidence may justify a configuration change at unchanged authority. Compare effective cost only among behaviorally qualified candidates, using matching populations and observation windows. See fitness terminology.
| Term | In plain terms | What it means here |
|---|---|---|
| A label | A recorded judgment bound to its context | Given this input, this output was acceptable, unacceptable, or worse than another output. In a gated task these judgments are produced as a byproduct of governance. A human approving, rejecting, or editing a gated output is labeling it, whether or not anyone uses the word. |
| A preference pair | Two outputs for one input, ranked | A better output and a worse one for the same context. An approve/reject history over a single boundary is already this shape, which is why a gated log fits improvement methods that were never designed with governance in mind. |
| Distillation | Replacing an expensive judge with a cheap model trained on its verdicts | The term the maturity curve already uses. An expensive evaluator produces verdicts; those verdicts train a cheaper evaluator; the cheaper one takes authority once measured agreement clears the bar. The same move, applied to the worker instead of the evaluator, is the second curve. |
| A held-out split | Evidence deliberately never trained on | A subset of the log set aside so that a candidate can be measured on judgments it has never seen. Measure a candidate on what it trained on and the measurement reports memory rather than ability. |
Two rules for using review data
Almost everything about training is outside this framework's scope. Two facts are not, because the commitments below exist to satisfy them.
Training amplifies whatever the labels actually encode. If the verdicts secretly track topic, source, or which reviewer was on shift rather than output quality, that is what gets learned. This is why labels are profiled before first use.
A candidate must be measured on judgments it never trained on. Otherwise the measurement reports memory rather than ability. This is why a held-out split is frozen, and why it is frozen before any training begins rather than after.
One set of judgments, four uses
The evidence log is not an audit trail with a side benefit. It is a single label stream with several consumers that progress along different dimensions.
The label stream
Every gated decision that produces a verdict, whether from a human, judge, or measured downstream result, recorded against the input it judged.
Cheaper gating
A less expensive evaluator
Moves: The economics
Better work
An improved implementation behind the boundary
Moves: The metric
Sharper scope
A better-cut boundary
Moves: Part of the task
Trustworthy measurement
The right to believe the other three
Moves: Nothing, by itself
Labeling throughput sets the promotion rate
A task produces labels at the rate its work is reviewed and recorded. A promotion window of two hundred cases fills only when two hundred admissible cases have been captured. Unrecorded work cannot count toward that window, so instrumentation is part of becoming eligible for a change in oversight.
This is the strongest form of the instrument-from-the-start criterion, which is the entry condition for manual, the first on-spectrum position, rather than something added once a task looks promising.
Reservation order
Four uses draw from one finite set of judgments. A case frozen into the held-out split cannot train. A case spent on calibration overlap is not in the promotion window. The sizes belong to the task owner; the order does not vary.
| Order | Use | Rule | Why it ranks here |
|---|---|---|---|
| 1 | The frozen held-out split | Reserved first, before any other consumer draws. | It is the measuring instrument for every future candidate comparison. Freezing it after candidates have already been shaped by its contents invalidates it, and nothing later can repair that. |
| 2 | Calibration overlap | Reserved next, sized to the agreement bar the task declares. | A proxy evaluator whose agreement is unmeasured is not a validated gate. Calibration is what lets a cheap evaluator hold authority at all, so it outranks the improvement it would otherwise fund. |
| 3 | The promotion window | Counted from admissible evidence under the declared window type. | Windows are declared before accumulation starts. They draw on the same stream and must not be quietly widened by counting cases already committed elsewhere. |
| 4 | Training export | Takes the residual, deduplicated at export. | Improvement is the consumer that can most easily wait. A worker candidate produced from a stream that starved measurement is a candidate nobody can evaluate. |
The training-ready evidence log
These commitments are additive to the evidence-log commitments a conformant task already makes. A task can be fully PAA-conformant without a training-ready log: this is what makes the improvement branch available, not what makes the task governable.
- Verdicts resolve to the exact bytes that were judged
- Each logged decision binds its verdict to the input and output the evaluator actually observed. The evidence record carries references rather than inlined context, so training-readiness is a durability property of those references: they must resolve, for the full retention period, to the exact bytes, content-addressed the way a decision artifact already binds evidence by digest.
- Why. A verdict that cannot be replayed against its exact input is still promotion evidence. It is not training signal, because the pair it would form is unreconstructable.
- Administrative fields stay separate from quality fields
- Verdicts encoding a policy or routing decision live in their own fields and never merge into the quality signal. Suppression flags, do-not-act designations, and queue assignments are recorded independently, while quality grades on the same item remain valid quality signal.
- Why. A log that conflates "we chose not to act on this" with "this output was bad" will train the conflation, and the resulting worker will decline good work for administrative reasons nobody can trace.
- A held-out split is frozen before first training use
- Before the log funds improvement the first time, a subset is frozen and excluded from all training use permanently. The freeze is a one-time recorded event, and the exclusion has a natural home: the decision artifact already records excluded evidence by reference, digest, and reason.
- Why. The split is the bar’s measuring instrument for every future candidate. Freezing it late, after candidates have been shaped by its contents, means every subsequent comparison measures memory rather than ability.
- Labels are profiled before first use
- A bounded manual inspection, tens of examples, recorded, confirms the verdict tracks output quality rather than a confound: topic mix, source type, reviewer identity, or time of day.
- Why. Training amplifies whatever the labels actually encode. If they secretly track source rather than quality, the worker learns source. This is eval-the-eval applied to the label stream, and an unprofiled label set is a confident dataset nobody has checked.
- Provenance is preserved, not flattened
- Every evidence record already carries the complete evaluator identity, technique, evaluation basis, epistemic status, and version. Training consumers filter and weight on that identity rather than collapsing it into an undifferentiated pile of verdicts. A promotion decision for a trained candidate must be able to state what proportion of its training signal was human-verified.
- Why. Human gold, a validated judge, and a measured downstream result are not interchangeable evidence. The released identity already distinguishes them; the commitment is to keep using it after export, when the temptation to flatten is strongest.
- Deduplication happens at the training boundary
- Near-duplicate contexts and outputs are collapsed at export so repeated routine items do not dominate the signal. The log itself stays complete, deduplication is a property of the export, never of the record.
- Why. The log is the audit trail and must remain the full record. But an export in which one high-frequency template accounts for most pairs trains that template and little else.
Read more
- Improvement: the capability dimension this stream funds, the ladder ordered by revocability, and the rules a candidate re-enters under
- Evaluators: the other branch, evaluator identity, the technique and evaluation-basis vocabularies, and the maturity curve
- Lifecycle: Establish evidence
- Schema: the evidence record that carries every verdict, field by field