Label stream
Review decisions become reusable labels
Every recorded review decision can help in more than one way. It can support a future change in oversight, train a less expensive evaluator, improve the system doing the work, or reveal that part of the task needs a different boundary. All four uses depend on the same requirement: each judgment must remain tied to the work it evaluated.
Read this page before Improvement. The evaluator branch it also feeds is at Evaluators.
A task produces useful labels only when its review decisions are recorded.
Primer
This section exists so the rest of the page reads without a machine learning background. If the vocabulary is already familiar, skip to the four consumers.
| Term | In plain terms | What it means here |
|---|---|---|
| A label | A recorded judgment bound to its context | Given this input, this output was acceptable, unacceptable, or worse than another output. In a gated task these judgments are produced as a byproduct of governance. A human approving, rejecting, or editing a gated output is labeling it, whether or not anyone uses the word. |
| A preference pair | Two outputs for one input, ranked | A better output and a worse one for the same context. An approve/reject history over a single boundary is already this shape, which is why a gated log fits improvement methods that were never designed with governance in mind. |
| Distillation | Replacing an expensive judge with a cheap model trained on its verdicts | The term the maturity curve already uses. An expensive evaluator produces verdicts; those verdicts train a cheaper evaluator; the cheaper one takes authority once measured agreement clears the bar. The same move, applied to the worker instead of the evaluator, is the second curve. |
| A held-out split | Evidence deliberately never trained on | A subset of the log set aside so that a candidate can be measured on judgments it has never seen. Measure a candidate on what it trained on and the measurement reports memory rather than ability. |
Two rules for using review data
Almost everything about training is outside this framework's scope. Two facts are not, because the commitments below exist to satisfy them.
Training amplifies whatever the labels actually encode. If the verdicts secretly track topic, source, or which reviewer was on shift rather than output quality, that is what gets learned. This is why labels are profiled before first use.
A candidate must be measured on judgments it never trained on. Otherwise the measurement reports memory rather than ability. This is why a held-out split is frozen, and why it is frozen before any training begins rather than after.
One set of judgments, four uses
The evidence log is not an audit trail with a side benefit. It is a single label stream with several consumers that progress along different dimensions.
The label stream
Every gated decision that produces a verdict, whether from a human, judge, or measured downstream result, recorded against the input it judged.
Cheaper gating
A less expensive evaluator
Moves: The economics
Better work
An improved implementation behind the boundary
Moves: The metric
Sharper scope
A better-cut boundary
Moves: Part of the task
Trustworthy measurement
The right to believe the other three
Moves: Nothing, by itself
| Consumer | What it moves | Detail |
|---|---|---|
| Cheaper gating | The economics | Accumulated verdicts can train a classifier to replace an expensive judge or routine human review. The new evaluator takes authority only after measured agreement clears the declared bar. Lower evaluation cost makes higher-volume operation affordable. |
| Better work | The metric | The same verdicts can train or guide whatever produces the output. A candidate is compared with the current implementation using the same boundary and evaluation bar. If it performs better, it can replace the current implementation without changing the task’s oversight. |
| Sharper scope | Part of the task | Patterns in the verdicts can show that one part of a task is ready for less oversight while another part is not. Splitting those cases into explicit scopes lets each part advance on its own evidence. |
| Trustworthy measurement | Nothing, by itself | Some judgments must remain reserved for calibration, held-out comparison, and monitoring after promotion. They do not change oversight by themselves. They make future comparisons credible and provide the signal for demotion when performance degrades. |
Labeling throughput sets the promotion rate
A task produces labels at the rate its work is reviewed and recorded. A promotion window of two hundred cases fills only when two hundred admissible cases have been captured. Unrecorded work cannot count toward that window, so instrumentation is part of becoming eligible for a change in oversight.
This is the strongest form of the instrument-from-the-start criterion, which is the entry condition for manual, the first on-spectrum position, rather than something added once a task looks promising.
Allocation: who draws first
Four uses draw from one finite set of judgments. A case frozen into the held-out split cannot train. A case spent on calibration overlap is not in the promotion window. The sizes belong to the task owner; the order does not vary.
| Order | Claimant | Rule | Why it ranks here |
|---|---|---|---|
| 1 | The frozen held-out split | Reserved first, before any other consumer draws. | It is the measuring instrument for every future candidate comparison. Freezing it after candidates have already been shaped by its contents invalidates it, and nothing later can repair that. |
| 2 | Calibration overlap | Reserved next, sized to the agreement bar the task declares. | A proxy evaluator whose agreement is unmeasured is not a validated gate. Calibration is what lets a cheap evaluator hold authority at all, so it outranks the improvement it would otherwise fund. |
| 3 | The promotion window | Counted from admissible evidence under the declared window type. | Windows are declared before accumulation starts. They draw on the same stream and must not be quietly widened by counting cases already committed elsewhere. |
| 4 | Training export | Takes the residual, deduplicated at export. | Improvement is the consumer that can most easily wait. A worker candidate produced from a stream that starved measurement is a candidate nobody can evaluate. |
The training-ready evidence log
These commitments are additive to the evidence-log commitments a conformant task already makes. A task can be fully PAA-conformant without a training-ready log: this is what makes the improvement branch available, not what makes the task governable.
- Verdicts resolve to the exact bytes that were judged
- Each logged decision binds its verdict to the input and output the evaluator actually observed. The evidence record carries references rather than inlined context, so training-readiness is a durability property of those references: they must resolve, for the full retention period, to the exact bytes, content-addressed the way a decision artifact already binds evidence by digest.
- Why. A verdict that cannot be replayed against its exact input is still promotion evidence. It is not training signal, because the pair it would form is unreconstructable.
- Administrative fields stay separate from quality fields
- Verdicts encoding a policy or routing decision live in their own fields and never merge into the quality signal. Suppression flags, do-not-act designations, and queue assignments are recorded independently, while quality grades on the same item remain valid quality signal.
- Why. A log that conflates "we chose not to act on this" with "this output was bad" will train the conflation, and the resulting worker will decline good work for administrative reasons nobody can trace.
- A held-out split is frozen before first training use
- Before the log funds improvement the first time, a subset is frozen and excluded from all training use permanently. The freeze is a one-time recorded event, and the exclusion has a natural home: the decision artifact already records excluded evidence by reference, digest, and reason.
- Why. The split is the bar’s measuring instrument for every future candidate. Freezing it late, after candidates have been shaped by its contents, means every subsequent comparison measures memory rather than ability.
- Labels are profiled before first use
- A bounded manual inspection, tens of examples, recorded, confirms the verdict tracks output quality rather than a confound: topic mix, source type, reviewer identity, or time of day.
- Why. Training amplifies whatever the labels actually encode. If they secretly track source rather than quality, the worker learns source. This is eval-the-eval applied to the label stream, and an unprofiled label set is a confident dataset nobody has checked.
- Provenance is preserved, not flattened
- Every evidence record already carries the complete evaluator identity, technique, evaluation basis, epistemic status, and version. Training consumers filter and weight on that identity rather than collapsing it into an undifferentiated pile of verdicts. A promotion decision for a trained candidate must be able to state what proportion of its training signal was human-verified.
- Why. Human gold, a validated judge, and a measured downstream result are not interchangeable evidence. The released identity already distinguishes them; the commitment is to keep using it after export, when the temptation to flatten is strongest.
- Deduplication happens at the training boundary
- Near-duplicate contexts and outputs are collapsed at export so repeated routine items do not dominate the signal. The log itself stays complete, deduplication is a property of the export, never of the record.
- Why. The log is the audit trail and must remain the full record. But an export in which one high-frequency template accounts for most pairs trains that template and little else.
Read more
- Improvement: the capability dimension this stream funds, the ladder ordered by revocability, and the rules a candidate re-enters under
- Evaluators: the other branch, evaluator identity, the technique and evaluation-basis vocabularies, and the maturity curve
- Lifecycle: Accumulate, where the window and its admissibility rules are declared, and stage 9 for the swap this stream enables
- Schema: the evidence record that carries every verdict, field by field