Build PAA / Worked Example
One task, walked end to end
Start with a task whose boundary is explicit and whose work remains under human control. Observe the existing work, run a candidate beside it, and collect behavioral and economic evidence. First choose a configuration at unchanged authority. Later, decide whether continued evidence justifies changing oversight.
All numbers and operating decisions on this page are illustrative. This is a teaching example, not a report of production results or a real autonomy transition.
The executable quickstart uses a refund task, but follows the same governance sequence shown here. The refund approval quickstart runs propose, approve, and emergency demotion against a disposable store.
-
Stage 1
Declare the task
State: none yet; the boundary is not yet declared.
Before measuring performance, define the input, output, real-world action, and work that remains outside the boundary.
Our running example is bounded_publication. It accepts a draft_post and may produce a published_post for either a community or professional channel. Publishing anything outside those channels is excluded.
Artifact:
declaration-
Task
bounded_publication, declaration version1 -
Boundary: input
draft_post, outputpublished_post -
initial_position:manual·deployment:disabled -
scopes:publish:community,publish:professional
deployment is disabled, required on every declaration, with no default, and set to disabled until Stage 2 instruments the existing manual effect.
Rule: An unbounded workflow cannot enter the spectrum: no typed boundary, no evidence, no gate.
-
Task
-
Stage 2
Instrument the existing work
State: position
manual, deploymentactivefor scopepublish:communityRecord the human-run workflow before adding authority to automation. Capture each draft, published result, and human decision so later comparisons use observed work rather than memory.
A person still writes and publishes every post. The task is active at the manual position.
Artifact:
evidence_records-
Scope
publish:community, subject kindcase -
output_recorded: techniquedeterministic, evaluation basis kindinvariant, epistemic statusground_truth, authorityblocking, versionv1
A minimal structural evaluator on the human-produced output, enough to satisfy the evidence-record contract while the real quality evaluators are still absent.
-
Scope
-
Stage 3
Run the worker in shadow
State: position
manual, deploymentshadowfor scopepublish:communityThe candidate runs on real drafts, but its output never reaches a reader. The human continues to publish the real result.
Record the candidate beside the human-controlled outcome so they can be compared. PAA calls this shadow operation.
Compare worker configurations against the same behavioral bar and the operating cost per accepted outcome. The illustrative comparison below selects a worker while authority remains manual and deployment remains shadow.
Illustrative configuration comparison
Choose a worker; keep the same authority
Both configurations run the same 200 frozen cases for
bounded_publication, scopepublish:community. Only the worker model changes. The prompt, evaluator set, rubric, human-review procedure, and fallback stay fixed. Position remainsmanualand deployment remainsshadow.Before the comparison, the operator requires an acceptance rate of at least 95% and at most 0 hard safety failures. An accepted outcome meets the publication-quality rubric and all safety checks. Every candidate is reviewed; acceptance here does not authorize publication. The economic policy prefers lower operating cost per accepted outcome among qualified configurations.
Illustrative results: same 200 cases per configuration; costs in USD Configuration Behavioral pass rate Hard failures Human-review rate Worker cost Evaluation cost Accepted outcomes Total operating cost Cost per accepted outcome Configuration A 96% · qualified 0 100% $12.00 $4.00 192 $60.00 $0.313 Configuration B (selected) 98% · qualified 0 100% $6.00 $4.00 196 $52.00 $0.265 Configuration A: $60.00 includes $12.00 worker inference, $4.00 evaluation, $40.00 human review, and $4.00 additional retry costs. Each charge is counted once, including attempts that did not produce an accepted outcome.
Configuration B: $52.00 includes $6.00 worker inference, $4.00 evaluation, $40.00 human review, and $2.00 additional retry costs. Each charge is counted once, including attempts that did not produce an accepted outcome.
Both clear the behavioral bar. The operator selects Configuration B for its lower effective cost. Humans continue to control publication; no autonomy event is produced by this choice. The implementation records the selected configuration and retains the incumbent as a fallback.
These fabricated costs cover all components declared for this comparison. With real data, disclose missing costs and distinguish estimates from actual spend. Zero accepted outcomes makes the ratio undefined. Aggregate task or pipeline costs only when the underlying usage and pricing remain attributable.
The later promotion window below is separate evidence for the selected configuration. A real experiment could instead justify retaining the incumbent, rejecting every candidate, or collecting more evidence.
Artifact:
evidence_records-
Scope
publish:community, subject kindcase -
publication_quality: techniquellm_judge, evaluation basis kindrubric, epistemic statusproxy, authorityadvisory, versionv1 -
publication_quality: techniquehuman, evaluation basis kindhuman_gold, epistemic statusground_truth, authorityadvisory, versionv1
Both evaluators share property publication_quality without sharing technique, evaluation basis, epistemic status, or authority. This is the multiple-evaluators-per-property case.
-
Scope
-
Stage 4
Evaluate and establish evidence
State: position
manual, deploymentshadowfor scopepublish:communityA rubric-driven model and a human reviewer judge publication_quality for the same candidates. Each judgment records what was measured and which evaluator produced it.
Freeze the exact cases used for the decision, including why any case was excluded. This decision artifact records the 200-case window and whether it cleared the promotion rule.
This is a later window collected for the selected configuration, separate from the comparison population. The consumer retains configuration and cost attribution; the current runtime does not enforce worker-specific windows or read cost for eligibility.
Artifact:
decision_artifact-
Scope
publish:community, decision kindpromotion, outcomeeligible -
Window:
cases:200 -
Included
194, excluded6
Every excluded reference carries a reason. An admissibility snapshot without a reason for each exclusion is not a valid decision artifact.
Rule: Recorded evidence is not automatically admissible evidence; admissibility is a declared, snapshotted decision.
-
Scope
-
Stage 5
Promotion eligibility
State: position
manual, deploymentshadowfor scopepublish:communityThe evidence now clears the promotion rule. Nothing changes yet; the task is only eligible for a motion.
Artifact:
position_projection-
deployment:shadow -
Scope
publish:communityresolves tomanual -
Scope
publish:professionalresolves tomanual
Both declared scopes still resolve to manual.
-
-
Stage 6
Decide whether to change oversight
State: position
manual, deploymentshadowfor scopepublish:communityA motion proposes moving publish:community from manual to human-in-the-loop review and binds the proposal to its evidence.
operator:jordan_lee reviews the evidence and approves or rejects the motion.
Artifact:
motion_events-
Scope
publish:community:manualtohitl -
motion_proposed: actorsystem:promotion_rule: promotion window (200 cases) admissible and eligible per decision artifact -
motion_approved: actoroperator:jordan_lee: evidence reviewed; publication_quality agreement across both evaluators clears the promotion bar
Both rows share one motion_id and bind to the same evidence_ref and evidence_sha256 as the Stage 4 decision artifact.
Rule: A motion is proposed and resolved by a named actor or a system actor. It does not fire itself.
-
Scope
-
Stage 7
Apply the scoped change
State: position
hitl, deploymentshadowfor scopepublish:communityThe approved motion moves publish:community from manual to human-in-the-loop review. The professional channel remains manual.
The event stream records the scoped change and is used to reconstruct the current position.
Artifact:
motion_events-
Scope
publish:community:manualtohitl -
position_changed: actoroperator:jordan_lee: motion approved; committing publish:community from manual to hitl
This row shares its motion_id with the motion_proposed and motion_approved rows from Stage 6: three rows, one motion, one scope.
-
Scope
-
Stage 8
Operate and keep evaluating
State: position
hitl, deploymentactivefor scopepublish:communityThe community worker now prepares each publication. Its blocking checks must pass, and a human still authorizes the final action. Low-confidence cases can route directly to the human without changing the position of the whole scope.
Evaluation continues during active operation. Replacing an evaluator creates a new declaration version and evidence partition; replacing a worker also requires new evidence, although the current contract cannot enforce worker-version attribution.
Artifact:
position_projection-
deployment:active -
Scope
publish:communityresolves tohitl -
Scope
publish:professionalresolves tomanual
The active declaration is a new version. Its two scopes resolve independently: publish:community reaches hitl through a new exact-scope motion, while publish:professional remains manual.
Rule: Deployment and evaluator succession require new declaration versions; neither silently mutates a position or an evidence record already on file.
-
-
Stage 9
Demotion
State: position
manual, deploymentactivefor scopepublish:communityOne confirmed publishing failure is enough under this policy. The operator moves publish:community back to manual immediately; the professional channel is unaffected.
Artifact:
motion_events-
Scope
publish:community:hitltomanual -
motion_proposed: actoroperator:jordan_lee: confirmed publish_authorization failure on one case; demotion.window (1 case) satisfied -
motion_approved: actoroperator:jordan_lee: operator-initiated demotion, committed immediately -
position_changed: actoroperator:jordan_lee: committing publish:community from hitl back to manual
All three rows share one motion_id and are committed in a single immediate operator action, for publish:community alone.
Rule: A policy failure or an operator can demote immediately, on a one-case window, for the affected scope only.
-
Scope