agent-orchestrator control mechanisms
00 · What is Shipyard
Shipyard is a self-hosted platform where AI owns the software development lifecycle end to end. You design a system interactively with an agent and give one approval, at plan level. The platform then works the plan off as a project cascade: wave by wave, each issue is built by a worker, reviewed by an independent worker, merged through a gate you armed, health-verified, and traced end to end. You watch a live tree in Mission Control, you can pause everything with one label, and the platform escalates to you instead of guessing.
The interactive bridge to GitHub is part of the system: Shipyard operates its own GitHub App. Inbound, every webhook (issues, comments, PRs, reviews, labels) flows through it into the event router. Outbound, the platform speaks as its bot identity: it posts PRDs and review findings, sets the labels that are the system state, opens and merges PRs, and links every action to its trace. Install the App on a repo and that repo joins the platform. And because the App is a first-class RBAC role, it can never approve its own gates: loop protection at the permission level. Everything below explains the control mechanisms that make this trustworthy.
01 · Overview
Fast loops run inside slow ones. The eval loop fires on every single AI output; the build loop wraps it per issue; the cascade beats every ten minutes; the engine climbs every six hours. Self-healing watches all of them continuously, and the author is the outermost ring, not a bystander. Hover a ring to read it, click to jump to its chapter.
02 · Interlock
Nesting is only half the story. The loops are wired together by three kinds of signal: work flowing forward, escalation converging on the human, and feedback grounding the numbers. Filter by type to see each layer alone.
03 · The inner loop
The heart of the system. It doesn't count iterations; it measures error, verifies every claim adversarially, and regulates on the trajectory. Confidence is earned from evidence, never self-reported. Play a story below the schematic to walk one signal through the circuit.
Every finding is attacked before it counts: structural claims get grepped, world-knowledge claims get checked against the deployed repo, behavioral claims get tests executed. Disproved phantoms cost nothing; unproven claims count half. The fixer can dispute a finding, but only with an executable proof.
Missing evidence counts zero, not neutral: a round without CI can never score like a round with a green pipeline. Hard disproof dominates everything else. Tier high ≥ 0.75 is the only tier that may auto-merge.
| Round | Error | Confidence | decide() | ||
|---|---|---|---|---|---|
| 0 | 6.0 | 0.24 | continue · first round | ||
| 1 | 3.0 | 0.51 | continue · progress | ||
| 2 | 1.0 | 0.68 | continue · progress | ||
| 3 | 0 | 0.82 | converged → auto-merge |
Budget for this PR: ceil(6/2) = 3 fix rounds, earned by the size of the initial problem, not a constant. A one-line fix gets the floor (3); a heavy review gets up to 8. Both bounds are live-tunable Airflow Variables.
The whole state (error per round, finding fingerprints, evidence) is persisted as a hidden comment on the PR itself. The loop survives restarts, and a human can read every decision.
04 · Loop gallery
Every family follows the same grammar as the inner loop: measure → decide → act → back, with a cap on the ring and a red exit to the human. Once you can read one circuit, you can read them all.
0.25.eval_regression_gate replays pinned cases so rubric changes can't silently degrade the gate.acceptance-blocked, terminal until a human unblocks.05 · In production
Every review↔fix loop renders as a live card in Mission Control: the verdict with its judge score, the round against its budget, the error and judge trajectories as sparklines, agent count, and the token cost of the whole loop. These are unedited captures from the running system.
06 · Stop conditions
"Loop four times, then give up" is not control engineering. The controller distinguishes why it stops, and every escalation names the specific finding that is stuck.
The last fix measurably broke the repo's own reference check. Overrides everything: you don't iterate on top of a regression. The human's job is to revert, not to judge prose.
A finding disappears and comes back: ping-pong between reviewer and fixer. That's a structural disagreement no further round will resolve.
The parametric lid. Escalates with the full error trajectory so the human sees whether it was converging slowly or going nowhere.
The error didn't fall and the identical finding is still open; the fix channel is producing motion, not progress.
The error may still inch down while the wider evidence (judge, CI, tests) has stopped improving. More rounds now add cost, not confidence. Can also file targeted branch-scoped tickets instead of dead-ending.
Progress is measured, not assumed. And convergence (error → 0 → APPROVE) exits the loop entirely, through the four auto-merge locks.
07 · The operational inventory
The full fleet of closed loops in production. Each row is a running DAG; each has a scenario test that replays its decision logic; several replay real incidents.
| Loop | Cadence | Signal | Actuator | Cap / escalation |
|---|---|---|---|---|
| The inner circuit | ||||
| Review↔fix controller | per push | verified weighted error + confidence trajectory | fix worker re-dispatch | budget 3–8 → named-finding escalation |
| Review-loop watchdog | 10 min | push/review timestamps in fix cycles | nudge dead channels, re-trigger lost reviews (SHA-dedup) | 2 nudges → needs-human |
| Eval gate | per output | judge rubric score | regenerate with feedback | 2 retries → escalate with transcript |
| Workers & dispatch | ||||
| Worker-timeout healer | 10 min | timeouts via Loki, 9 root-cause classes | WAL checkpoint · cancel · CPU throttle | acts ≥ 2 timeouts, per-cause cooldown |
| Progress liveness | per sweep | rising tool calls, not wall clock | alive / stalled (5 min) / dead (30 min) verdicts | feeds healer & preemption |
| Dispatch reconciliation | 5 min | ack without PR/branch | reconcile or re-dispatch | unreconcilable → escalate |
| Implementation resume | 15 min | 'implementing' with no PR, no worker | re-dispatch until a PR exists | then human |
| Stuck-issue observer | 15 min | dispatched ticket silent ≥ 90 min | exactly one re-dispatch | 1 re-dispatch → needs-human + page |
| Merge & cascade | ||||
| Project cascade | 10 min | wave completion + system health | unblock next wave | health red → block + page |
| Acceptance gate | on cascade | exit code of the epic's acceptance command | label + gate epic closure | infra failure → blocked, terminal until human |
| Closure gate | on close | acceptance criteria of bot-closed issues | bounded reopen | 1 reopen → needs-human |
| Merge-readiness shepherd | 6 h | approved-but-blocked PRs | branch update, re-trigger CI | true conflict → named files + needs-human |
| Transport & deploy | ||||
| Webhook reconciler | 10 min | GitHub delivery status | redelivery | 4 attempts → single page |
| Infrastructure watchdog | 5 min | HTTP/TCP health, OAuth expiry, inodes | alert → triage → self-heal | unresolved → page |
| Deploy drift monitor | hourly | live DAGs vs deploy manifest | drift alert | 0 out-of-band edits tolerated |
| Post-deploy canary | after deploy | live canary checks | FAIL verdict | page; a deploy proves itself in prod |
| Merge-reached-production | hourly | main HEAD vs deployed commit + smoke checks | reconciliation alert | silent deploy block → loud failure |
| Meta & grounding | ||||
| DAG-failure observer | 30 min | FAILED runs + diagnosis classifier | retry infra flakes · ticket code errors · page auth | 3 tickets, 5 retries per sweep; never itself |
| Coherence auditor | 4 h | contradictions across the dashboard's own payloads | propose only | human or meta-observer disposes |
| Meta-observer | event + 4 h | every needs-human escalation | resolve resolvable classes, brief the rest | 1 re-dispatch → human; modes off/dry_run/active |
| Post-merge outcomes | daily | reverts, CI failures, hotfixes after merge | outcome log + benchmark proposal | revert → page |
| Confidence calibration | weekly | Brier score, reliability per decile | empirical recalibration mapping | overconfidence gap > 0.25 → page |
08 · The observer hierarchy
Three tiers of observation, with the human on top. The design rule at every tier: pure, unit-tested decision cores; reversible actuators; a hard cap; and a kill switch per observer.
Watch workers, issues, and events directly: the timeout healer (9 root causes), progress liveness, dispatch reconciliation, the stuck-issue observer, the webhook reconciler. They fix things (restart, re-dispatch, redeliver) under strict caps.
Watch the control loops from outside: the review-loop watchdog (the inner loop's blind spots), the merge-readiness shepherd (approved-but-blocked), the DAG-failure observer (the whole fleet, including other observers, never itself), the coherence auditor (the reporting layer's own contradictions).
Behind every cap: re-dispatch 1 · nudges 2 · tickets 3 · retries 5 · redeliveries 4. No inner loop can run unbounded without the author hearing about it, and autonomy is granted per issue, never globally.
09 · Grounding
A confidence score that is never checked against reality is just a number. Shipyard closes the outermost circle: real merge outcomes feed back into the scoring.
Every merged PR is re-checked: reverted? CI failed on the merge commit? hotfixed? The verdict is logged next to the loop's final confidence. Reverts page the author and auto-generate a benchmark proposal.
Claimed confidence vs. empirical accuracy, in ten deciles, plus a Brier score. An overconfidence gap > 0.25 pages. Corrections apply only once a decile has ≥ 10 resolved cases; until then, the mapping honestly stays identity.
Zero-touch completion rate, phantom-finding rate, fix-rounds-to-convergence, plus scenario tests that replay whole loop trajectories (convergence, oscillation, plateau, crash-resume) as invariants. Several replay real incidents.
A control loop without proof is a guess with a cron schedule. the design rule every loop in this system is held to