feat(factory): new tests must fail on the Epic's base commit #773

New issue

Open

opened 2026-09-29 11:34:04 +02:00 by dries · 0 comments

dries commented

2026-09-29 11:34:04 +02:00

Owner

A test that already passes before the change proves nothing about it (Cognition's Frontier Code benchmark penalises this, per the HumanLayer "Why Software Factories Fail" talk).

After Formula checks pass, run only the tests added in the Epic's diff against the base commit (a throwaway worktree). Report any that pass there as weak tests in the validator results. Warn, don't block, at first.

Builds on the #757 check runner and diff scan. Effort M.

A test that already passes before the change proves nothing about it (Cognition's Frontier Code benchmark penalises this, per the HumanLayer "Why Software Factories Fail" talk). After Formula checks pass, run only the tests added in the Epic's diff against the base commit (a throwaway worktree). Report any that pass there as weak tests in the validator results. Warn, don't block, at first. Builds on the #757 check runner and diff scan. Effort M.