Task lifecycle
Govern a task over time
First define the task. Then observe the work, evaluate it, and accumulate evidence about performance. Governed decisions can reduce or increase oversight. Evaluation continues as workers and evaluators change.
The rest of this page defines the mechanics behind those transitions. For one task walked through them in order, start with the Journey.
The lifecycle in five steps
- Bound. Define the task.
- Observe. Run and evaluate it.
- Establish evidence. Accumulate information about performance.
- Change oversight. Use governed decisions to move between positions.
- Continue governing. Keep evaluating, demote when necessary, and account for changes to workers or evaluators.
Why one state value is not enough
“How autonomous is this task?” sounds like one question, but PAA separates three things that can change independently.
- How much human oversight does the task require?
- The autonomy position changes through promotion or demotion. It does not say how capable the worker is.
- How reliable and economical is the evaluation?
- Evaluator maturity changes through validation and replacement. It does not grant the task a new position.
- How capable is the system doing the work?
- Worker capability changes when the worker improves or is replaced. A new worker does not inherit earned autonomy.
The same evidence should not both justify changing a worker or evaluator and prove that the changed system deserves more autonomy. Otherwise, you cannot tell which version produced the result.
PAA separates those evidence periods into attribution windows: change the worker or evaluator, collect evidence attributable to the changed system, and only then use that evidence for the next governance claim. The dimensions may use related evidence, but they must not advance from the same attribution window. Read the rationale.
Stage 1
Bound
Establish the typed boundary: typed input, typed output, explicit success shape, explicit termination. An unbounded task cannot be evaluated, so it cannot be gated, so it cannot progress.
| Technique | When |
|---|---|
| Typed result envelope (success / failure / partial as first-class states) | Default for any agent task |
| Bounded loops (max iterations, max cost, max wall time) | Any task with retry or multi-step reasoning |
| Task decomposition (tree of bounded subtasks) | Work too large for one boundary |
| Explicit non-goals in the task definition | Tasks prone to scope creep |
- Evidence produced
- The task definition itself. The boundary is the first auditable artifact, and it is what every later evidence record’s boundary reference points back to.
- Common failure
- Boundary defined by prompt language only (“do X, don’t do Y”) with no typed contract. A model can violate either prose or a declared type; a typed boundary makes the violation mechanically detectable and rejectable at runtime.
Stage 2
Instrument
Instrumentation makes executions observable so later judgments can be connected to the work that produced them.
Record the observations and judgments in a form that can later support governance. Without observable work, PAA cannot measure performance or govern changes in oversight.
When an evaluator produces evidence, the record identifies that evaluator along with the subject, task boundary, and time. The Evaluators page defines the complete identity model.
Stage 2 attaches a minimal structural evaluator, so it does not produce evaluator-free evidence. Stage 3 adds the evaluators that judge task quality.
See Evaluators for the complete identity model.
| Technique | When |
|---|---|
| Structured trace logging (per-step, machine-readable) | Default |
| State-as-spine (single canonical state object, transitions logged atomically) | Pipelines where partial state corruption is possible |
| Cost and latency capture per step | Anything with a budget or SLA |
| Replay capability (trace sufficient to reconstruct the run) | High-stakes domains, debugging evaluator disagreements |
- Evidence produced
- The raw material behind every evidence record: subject reference, boundary references, and timestamps. Traces are what promotion and demotion decisions ultimately cite.
- Common failure
- Logging designed for human debugging (print statements, prose) rather than for evaluation (structured, queryable). Retrofitting structure onto prose logs costs more than instrumenting correctly at the start.
Stage 3
Evaluate
How do we determine whether this execution was good enough? Evaluators judge defined properties of the bounded, instrumented task.
Each judgment identifies the evaluator that produced it. PAA records those attributable judgments as evidence that later governance can reason about.
A property may have more than one evaluator. See the Evaluators page for the complete identity model; placement is a separate task-level concern introduced in Stage 4.
See Evaluators for the complete identity model.
| Technique | Verdict basis | Cost | When |
|---|---|---|---|
| Exact match / assertion | Deterministic comparison | Near zero | Output has a single correct form |
| Property checks | Invariants hold (schema valid, sums balance, no PII present) | Near zero | Correctness is partially checkable even when the full answer is not |
| Reference-based scoring | Distance from known-good reference labels | Low | Golden datasets exist |
| Rubric LLM judge | Model applies written criteria; typically a proxy epistemic status | High per verdict | Fuzzy quality, cold start, no labels yet |
| Trained classifier | Learned from accumulated reference labels | Low per verdict, high setup | Enough labels accumulated (see stage 8) |
| Human review | Person renders verdict; human-gold basis, ordinarily ground truth | Highest | Cold start, calibration, stakes exceed all automated confidence |
- Evidence produced
- Verdicts, attached to traces as evidence records carrying the complete producing evaluator identity.
- Common failure
- One evaluator asked to judge everything about the output. Verdicts become uninterpretable. Decompose: one evaluator per property, composed, and multiple evaluators may legitimately share a property.
Stage 4
Gate
Placement determines where an evaluator participates in the workflow. The task declares blocking, async, or offline placement for its current autonomy position.
Authority determines whether an evaluator's verdict may halt the governed effect. Each evaluator declares advisory or blocking authority.
Placement and authority are independent. An evaluator may run before an effect without being allowed to block it, or may contribute governance evidence without sitting directly in the execution path.
| Position | Behavior | When |
|---|---|---|
| blocking (pre-execution) | Gates at this position run before the effect; a failing blocking evaluator prevents it | Irreversible or costly actions |
| async (post-execution) | Gates at this position run after execution, before the next cycle; verdicts are recorded and may trigger review or demotion | Reversible actions, high volume |
| offline (batch / aggregate) | Evaluation happens in batch, not per-execution; feeds promotion and demotion evidence rather than gating any single run | Manual position monitoring; matured tasks; cost control |
| HITL surface | Human is the blocking gate; the system produces a review artifact, a plan, a brief, a proposed action with evidence attached | Cold start, high stakes, calibration |
- Evidence produced
- Gate decisions (passed, blocked, escalated) recorded as evidence, which are themselves input to stage 5.
- Common failure
- Placement chosen once and never revisited, or evaluator authority confused with task placement. Placement should move as evidence accumulates through promotion and demotion; authority is a separate per-evaluator decision that does not automatically follow position.
Stage 5
Accumulate
Collect evidence records over a declared, typed window (cases or duration) and apply admissibility: which records are included, which are excluded, and why. A decision artifact snapshots that window immutably, exact included and excluded evidence references and reasons, content-hashed. A single passing verdict is an anecdote; an admissible windowed snapshot is what a motion can bind to.
| Technique | When |
|---|---|
| Rolling window over N cases (kind: cases) | Default; volume-based tasks |
| Rolling window over a time period (kind: duration) | Low-volume tasks where N takes too long |
| Stratified windows (per input category) | Task performance varies by input type; aggregate hides weak strata |
| Disagreement tracking (evaluator vs human on overlapping cases) | Any HITL stage; feeds stage 8 |
- Evidence produced
- Windowed metrics and the decision-artifact snapshot they support: pass rate, escalation precision/recall, cost per verdict, human-override rate, plus the included/excluded evidence list and exclusion reasons.
- Common failure
- Window defined after the fact to make the numbers pass, or a snapshot generated without recording why excluded evidence was excluded. The window and threshold must be declared before accumulation starts, and every exclusion must carry a reason, or the promotion is a guess with paperwork.
Stage 6
Promote
Evidence can make a task eligible for a different position. Eligibility means the evidence supports considering the change; it does not make the change happen. A motion proposes the change, governance approves or rejects it, and an approved change is recorded before the new position takes effect.
| Technique | When |
|---|---|
| Threshold rule (metric ≥ bar over window, eligible for one level) | Default |
| Staged promotion (one position at a time: manual → hitl → hotl → autonomous) | Default path; skip positions only with explicit justification |
| Promotion with probation (elevated sampling for M cases after the motion executes) | First promotion of any task; cheap insurance |
| operator_approval execution (a person ratifies the motion) | Regulated domains; the rule proposes, a person ratifies |
- Evidence produced
- The autonomy-event motion sequence: motion_proposed, then motion_approved, then position_changed, each bound to the decision artifact that justified it by evidence_ref and evidence_sha256. This sequence is what makes autonomy auditable.
- Common failure
- Promotion happens informally, with no decision artifact and no motion record. The system's actual autonomy level and its declared level diverge, and no artifact records when or why.
Stage 7
Demote / Fall Back
Promotion does not end evaluation. If later evidence no longer supports the current position, PAA can increase oversight again. A declared policy can trigger that response, and an operator can act immediately on one confirmed high-stakes failure without waiting for a promotion-sized window.
| Technique | When |
|---|---|
| Metric-triggered demotion (policy-fired motion proposal) | Default; arm it at promotion time |
| Immediate operator demotion (kind: cases, size: 1 window) | Policy failure or a single high-stakes confirmed failure; does not wait for a larger window |
| Retry with modification | Transient failures |
| Fallback route (cheaper model, simpler method, cached answer) | Availability matters more than peak quality |
| Human escalation | Verdict fails and stakes are high |
| Rollback (undo the action) | Only where actions are actually reversible; verify, do not assume |
| Circuit breaker (task suspended entirely after K failures in window) | Failures are correlated, not independent |
- Evidence produced
- Demotion records and fallback firings, the same motion/event audit trail as promotion, whichever actor (policy trigger or operator) proposed the motion.
- Common failure
- Non-atomic state transitions. A failure mid-transition leaves the system in a state no rule anticipated. Transitions between autonomy positions must be atomic because the same seam failures that affect pipelines can also affect changes in authority.
Stage 8
Distill / Replace
Changing how performance is measured can change what the accumulated evidence means. PAA therefore identifies and versions each evaluator. A replacement earns authority through measured agreement, while historical evidence keeps the identity of the evaluator that produced it.
| Technique | When |
|---|---|
| Judge-as-labeler (rubric judge verdicts become training data) | From day one; labels are a byproduct, capture them |
| Human-review-as-labeler | Every HITL decision is a free label; capture or lose it |
| Distillation ladder (judge → embedding classifier → fine-tuned small model) | Label count crosses viability thresholds per rung |
| Eval-the-eval (calibration sample: classifier vs judge vs human on overlap) | Always, once a cheap evaluator is in production |
| Evaluator versioning (verdicts tagged with evaluator version) | Always; otherwise one evidence window may mix verdicts from different evaluator versions |
| Authority transfer (new version takes blocking authority; prior version may drop to advisory or retire) | Once eval-the-eval clears the agreement bar; the outgoing version’s records are left untouched |
- Evidence produced
- Evaluator performance records: agreement rates, cost per verdict over time, label counts, and, at succession, which version now holds authority.
- Common failure
- Labels thrown away. Months of human review decisions with no capture path, then a classifier project starts from zero. The single cheapest mistake to avoid in the whole lifecycle.
Stage 9
Improve / Swap
Existing evidence describes the worker that produced it. Replacing that worker raises a provenance question: how much of the old evidence still tells us about the new system? PAA keeps the boundary and evaluation bar fixed during comparison. A replacement must establish its own evidence and does not inherit the prior worker's earned autonomy.
| Technique | When |
|---|---|
| Context or instruction revision (change what the worker sees, weights untouched) | First rung; fully revocable and spends no label budget |
| Relevance-selected precedent (retrieve the few most relevant past judgments per item) | Accumulated examples are overflowing the input; selection stays flat where accumulation grows without bound |
| Imitation training on approved outputs | The worker cannot reliably produce the required form at all; consumes approvals only |
| Preference training on approve/reject pairs | A gated task’s verdict stream is already preference-shaped; the rung that uses rejections rather than discarding them |
| Trial-and-verifier training against a programmatic scorer | Long-horizon multi-step work with a trustworthy verifier; heaviest rung and least revocable |
| Baseline-then-candidate on the frozen held-out split | Always, before any candidate enters; a candidate is compared against a measured baseline, never an impression |
| Parallel-arm comparison under shadow deployment | A candidate should earn its evidence without displacing a production incumbent |
| Versioned incumbent retained as the demotion target | Any weight-changing rung; demotion on this axis needs somewhere concrete to go |
- Evidence produced
- Baseline and candidate metrics on the frozen held-out split, the deterministic-layer check that no cheaper evaluator regressed, and the recorded proportion of training signal that was human-verified, all of which the released contracts already carry. Which worker version holds the boundary, and which retained version is the fallback, is not among them: the contract models no worker identity, so that binding lives in an external operational record and is not contract-verifiable until the gap is closed.
- Common failure
- Evaluator maturity and worker capability changed in the same window, making the result impossible to attribute. Another common failure is leaving the worker unversioned, so evidence cannot be tied to the implementation that produced it.