feat(factory): enforce verification contract with Formula checks, validator hardening, and idle watchdog #757

Open

dries wants to merge 1 commit from feat/factory-verification-hardening into main

dries commented

2026-09-29 09:26:33 +02:00

Owner

Summary

Formula checks: verification steps can declare config.commands. The validator agent reviews first. On complete_attempt, ocman answers checks_running, runs the commands in the worktree, and sends the results back to the validator. Completion is accepted only if every command passed at that exact HEAD. Runs are cancelled with their attempt or on shutdown. A failed run reruns on the next completion. Validators started before a restart are told to ask again.
Validator hardening:
- edits denied, and completion rejected if the shared branch moved
- no longer inherits the implementation model; prefers the planning model instead
- the prompt demands a PASS/FAIL line per acceptance criterion and a scan for work that games the checks
- results include a diff scan for deleted tests, edited test/lint config, and new skip or suppression markers
- forge tokens and OCMAN_*/OTEL_* are removed from the check commands' environment
Idle watchdog: a live attempt with no session or subagent activity for 30 minutes opens a recovery gate. The clock starts at the attempt start or the last human resume. Waiting on a permission or question prompt, or on Formula checks, doesn't count as idle.
Acceptance criteria: every non-reference implementation node now requires acceptanceCriteria (1–20). They are added to the Issue description as a checklist and passed to the validator.

Breaking for agents: proposals without acceptanceCriteria are rejected. The planner prompt and MCP examples are updated.

Known limits (documented): remote hosts can't run Formula checks. The diff scan is a hint and doesn't block completion. Watchdog gates are audited as recovery.requested.

Testing

New unit tests for the check runner, diff scan, gate, lifecycle (prune, forget, close), rerun, supersede, restart nudge, watchdog, and the state resume lookup. Plus one end-to-end state DB flow: implement → validator pending → fail → rerun → pass → idempotent retry.
Red/green: the validator-model test fails without the factory_claim.go change; the environment test fails without the filter.
go test ./..., -race on factory/local/server, golangci-lint, and the guard scripts all pass. Coverage went up in every package touched.
tsc -b fails on origin/main in SessionTerminalDock.test.tsx (act missing). The error is on main already; this PR doesn't touch that file.

## Summary - **Formula checks:** verification steps can declare `config.commands`. The validator agent reviews first. On `complete_attempt`, ocman answers `checks_running`, runs the commands in the worktree, and sends the results back to the validator. Completion is accepted only if every command passed at that exact HEAD. Runs are cancelled with their attempt or on shutdown. A failed run reruns on the next completion. Validators started before a restart are told to ask again. - **Validator hardening:** - edits denied, and completion rejected if the shared branch moved - no longer inherits the implementation model; prefers the planning model instead - the prompt demands a PASS/FAIL line per acceptance criterion and a scan for work that games the checks - results include a diff scan for deleted tests, edited test/lint config, and new skip or suppression markers - forge tokens and `OCMAN_*`/`OTEL_*` are removed from the check commands' environment - **Idle watchdog:** a live attempt with no session or subagent activity for 30 minutes opens a recovery gate. The clock starts at the attempt start or the last human resume. Waiting on a permission or question prompt, or on Formula checks, doesn't count as idle. - **Acceptance criteria:** every non-reference implementation node now requires `acceptanceCriteria` (1–20). They are added to the Issue description as a checklist and passed to the validator. Breaking for agents: proposals without `acceptanceCriteria` are rejected. The planner prompt and MCP examples are updated. Known limits (documented): remote hosts can't run Formula checks. The diff scan is a hint and doesn't block completion. Watchdog gates are audited as `recovery.requested`. ## Testing - New unit tests for the check runner, diff scan, gate, lifecycle (prune, forget, close), rerun, supersede, restart nudge, watchdog, and the state resume lookup. Plus one end-to-end state DB flow: implement → validator pending → fail → rerun → pass → idempotent retry. - Red/green: the validator-model test fails without the `factory_claim.go` change; the environment test fails without the filter. - `go test ./...`, `-race` on factory/local/server, `golangci-lint`, and the guard scripts all pass. Coverage went up in every package touched. - `tsc -b` fails on `origin/main` in `SessionTerminalDock.test.tsx` (`act` missing). The error is on main already; this PR doesn't touch that file.

dries added 1 commit

2026-09-29 09:26:33 +02:00