feat(factory): enforce verification contract with Formula checks, validator hardening, and idle watchdog #757

Merged

dries merged 1 commit from feat/factory-verification-hardening into main 2026-09-29 13:04:47 +02:00

Conversation 1 Commits 1 Files changed 33 +1830 -81

dries commented 2026-09-29 09:26:33 +02:00

Owner

Copy link

Summary

• Formula checks: verification steps can declare config.commands. The validator agent reviews first. On complete_attempt, ocman answers checks_running, runs the commands in the worktree, and sends the results back to the validator. Completion is accepted only if every command passed at that exact HEAD. Runs are cancelled with their attempt or on shutdown. A failed run reruns on the next completion. Validators started before a restart are told to ask again.
• Validator hardening:
  • edits denied, and completion rejected if the shared branch moved
  • no longer inherits the implementation model; prefers the planning model instead
  • the prompt demands a PASS/FAIL line per acceptance criterion and a scan for work that games the checks
  • results include a diff scan for deleted tests, edited test/lint config, and new skip or suppression markers
  • forge tokens and OCMAN_*/OTEL_* are removed from the check commands' environment
• Idle watchdog: a live attempt with no session or subagent activity for 30 minutes opens a recovery gate. The clock starts at the attempt start or the last human resume. Waiting on a permission or question prompt, or on Formula checks, doesn't count as idle.
• Acceptance criteria: every non-reference implementation node now requires acceptanceCriteria (1–20). They are added to the Issue description as a checklist and passed to the validator.

Breaking for agents: proposals without acceptanceCriteria are rejected. The planner prompt and MCP examples are updated.

Known limits (documented): remote hosts can't run Formula checks. The diff scan is a hint and doesn't block completion. Watchdog gates are audited as recovery.requested.

Testing

• New unit tests for the check runner, diff scan, gate, lifecycle (prune, forget, close), rerun, supersede, restart nudge, watchdog, and the state resume lookup. Plus one end-to-end state DB flow: implement → validator pending → fail → rerun → pass → idempotent retry.
• Red/green: the validator-model test fails without the factory_claim.go change; the environment test fails without the filter.
• go test ./..., -race on factory/local/server, golangci-lint, and the guard scripts all pass. Coverage went up in every package touched.
• tsc -b fails on origin/main in SessionTerminalDock.test.tsx (act missing). The error is on main already; this PR doesn't touch that file.

## Summary - **Formula checks:** verification steps can declare `config.commands`. The validator agent reviews first. On `complete_attempt`, ocman answers `checks_running`, runs the commands in the worktree, and sends the results back to the validator. Completion is accepted only if every command passed at that exact HEAD. Runs are cancelled with their attempt or on shutdown. A failed run reruns on the next completion. Validators started before a restart are told to ask again. - **Validator hardening:** - edits denied, and completion rejected if the shared branch moved - no longer inherits the implementation model; prefers the planning model instead - the prompt demands a PASS/FAIL line per acceptance criterion and a scan for work that games the checks - results include a diff scan for deleted tests, edited test/lint config, and new skip or suppression markers - forge tokens and `OCMAN_*`/`OTEL_*` are removed from the check commands' environment - **Idle watchdog:** a live attempt with no session or subagent activity for 30 minutes opens a recovery gate. The clock starts at the attempt start or the last human resume. Waiting on a permission or question prompt, or on Formula checks, doesn't count as idle. - **Acceptance criteria:** every non-reference implementation node now requires `acceptanceCriteria` (1–20). They are added to the Issue description as a checklist and passed to the validator. Breaking for agents: proposals without `acceptanceCriteria` are rejected. The planner prompt and MCP examples are updated. Known limits (documented): remote hosts can't run Formula checks. The diff scan is a hint and doesn't block completion. Watchdog gates are audited as `recovery.requested`. ## Testing - New unit tests for the check runner, diff scan, gate, lifecycle (prune, forget, close), rerun, supersede, restart nudge, watchdog, and the state resume lookup. Plus one end-to-end state DB flow: implement → validator pending → fail → rerun → pass → idempotent retry. - Red/green: the validator-model test fails without the `factory_claim.go` change; the environment test fails without the filter. - `go test ./...`, `-race` on factory/local/server, `golangci-lint`, and the guard scripts all pass. Coverage went up in every package touched. - `tsc -b` fails on `origin/main` in `SessionTerminalDock.test.tsx` (`act` missing). The error is on main already; this PR doesn't touch that file.

dries added 1 commit 2026-09-29 09:26:33 +02:00

feat(factory): enforce verification contract with Formula checks, validator hardening, and idle watchdog a7e0452fb9

dries referenced this pull request 2026-09-29 09:27:02 +02:00

feat(factory): route failed verification back to implementation automatically #758

dries referenced this pull request 2026-09-29 09:27:02 +02:00

feat(factory): turn review comments on the delivery PR into follow-up work #759

dries referenced this pull request 2026-09-29 09:27:03 +02:00

feat(factory): cost and token usage per Epic and Attempt #760

dries referenced this pull request 2026-09-29 09:27:03 +02:00

feat(factory): escalate to a stronger model on retry #761

dries referenced this pull request 2026-09-29 09:27:04 +02:00

feat(factory): learning step that proposes AGENTS.md and skill updates after an Epic #762

dries referenced this pull request 2026-09-29 09:27:04 +02:00

feat(factory): browser validator that clicks through UI changes #763

dries referenced this pull request 2026-09-29 09:27:05 +02:00

feat(factory): repository readiness check before pouring an Epic #764

dries referenced this pull request 2026-09-29 09:27:05 +02:00

chore(factory): hardening gaps left by #757 #765

dries force-pushed feat/factory-verification-hardening from a7e0452fb9 to c9267e1877 2026-09-29 10:53:51 +02:00 Compare

dries referenced this pull request 2026-09-29 11:34:04 +02:00

feat(factory): new tests must fail on the Epic's base commit #773

dries referenced this pull request 2026-09-29 11:34:04 +02:00

feat(factory): flag code-smell additions in the validator diff scan #774

forgejo-actions commented 2026-09-29 12:59:14 +02:00

Copy link

Coverage ratchet: ✅ pass

Suite	Baseline	This PR	Δ
frontend	82.90%	83.06%	+0.16	✅
go	84.80%	84.90%	+0.10	✅

Tolerance: -0.1%. Baseline stored on gh-pages.

<!-- coverage-ratchet --> ### Coverage ratchet: ✅ pass | Suite | Baseline | This PR | Δ | | |---|---|---|---|---| | frontend | 82.90% | 83.06% | +0.16 | ✅ | | go | 84.80% | 84.90% | +0.10 | ✅ | _Tolerance: -0.1%. Baseline stored on `gh-pages`._

dries merged commit 160c2bc2fb into main 2026-09-29 13:04:47 +02:00

dries deleted branch feat/factory-verification-hardening 2026-09-29 13:04:47 +02:00

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

backend

Go backend code

  

bug

Something is broken or incorrect

  

chore

Maintenance, tooling, or cleanup

  

duplication

Code duplication to be addressed

  

effort:complex

Significant effort or risk

  

effort:medium

Moderate effort required

  

effort:trivial

Quick fix, low complexity

  

enhancement

New feature or capability

  

follow-up

Follow-up from a PR review or shipped change

  

frontend

TypeScript / React frontend code

  

fullstack

Touches both backend and frontend

  

priority:high

Should be addressed soon

  

ready-for-agent

Agent-grabbable ticket, fully specced

  

refactor

Code restructuring without behaviour change

  

security

Security related

No labels

backend

bug

chore

duplication

effort:complex

effort:medium

effort:trivial

enhancement

follow-up

frontend

fullstack

priority:high

ready-for-agent

refactor

security

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

dries/ocman!757

No description provided.