feat: add agent workflow engine #338

Merged

dries merged 52 commits from feat/agent-workflows into main 2026-07-16 00:48:05 +02:00

Conversation 1 Commits 52 Files changed 182 +18071 -10441

dries commented 2026-07-15 15:54:47 +02:00

Owner

Copy link

Summary

• add immutable YAML/JSON workflow DAGs with approvals, commands, agents, subworkflows, maps, joins, triggers, retries, and recovery
• add resource pools, workspace leases, artifacts, migration from agent loops, and workflow MCP tools
• add workflow authoring, run inspection, examples, operational controls, and documentation

Testing

• make lint
• make test (Go suites and 1500 frontend tests)

## Summary - add immutable YAML/JSON workflow DAGs with approvals, commands, agents, subworkflows, maps, joins, triggers, retries, and recovery - add resource pools, workspace leases, artifacts, migration from agent loops, and workflow MCP tools - add workflow authoring, run inspection, examples, operational controls, and documentation ## Testing - make lint - make test (Go suites and 1500 frontend tests)

dries added 35 commits 2026-07-15 15:54:47 +02:00

fix: address workflow prototype review feedback (#312) ... 6bba43ff3f

- independent dirty state per YAML/JSON buffer with equivalent fixtures
- realistic state/resource-derived node details plus stable key and pinned subworkflow in inspector
- semantic tablist controls for view and source format
- lazy-load the disposable prototype route via Suspense
- readable font sizes; keep state symbols visible (never color-only)
- branch coverage for every pipeline state derivation, route/nav integration, and narrow-layout stacking order; scoped scrollIntoView mock

feat: run sequential approval workflows (#313) e19339b0cc

fix: preserve workflow prototype navigation d31f21a2ec

feat: execute permission-scoped command nodes (#314) 9ec9ceabc8

feat: execute workflow agent nodes (#315) 01e7ebf741

fix: recover interrupted workflow executors ee59bf3e8e

feat: expose workflow control through MCP (#323) 24f95ae728

feat: trigger immutable workflow runs with overlap policies (#322) ... ce87e82d88

Attach versioned manual/interval/cron/PR/child- and turn-completion
triggers to active workflow versions. Each firing snapshots the trigger
into an immutable run pinned to its version. Overlap supports
skip/queue/parallel (skip default); queued firings and detection state
are durable and survive restart, draining in deterministic order.

Reuses the existing loop cron/schedule timing, forge poller, and
session-status seams; interval/cron avoid backfiring historical slots,
PR reuses forge detection state, and completion triggers fire once per
idle edge. A 5s server tick evaluates triggers, isolating per-trigger
errors. Firing, run creation, and trigger-state updates commit in one
transaction. The Workflows UI shows trigger state/next/last/overlap/
result and refreshes on a workflow.trigger.updated SSE event. Migration
v23 backfills a manual trigger and run snapshot for pre-existing
workflows. Loops are intentionally not migrated (#325).

feat: store, redact, and expire workflow artifacts (#316) ... 25384b3c04

Make node outputs durable, immutable typed artifacts while keeping large
payloads out of SQLite.

- Content-addressed BlobStore (internal/workflows/blobstore.go) dedups
  identical payloads on disk under <ocman data dir>/workflow-artifacts;
  atomic writes, idempotent removal, ErrPayloadMissing after cleanup.
- state.db migration v26: workflow_artifact metadata table (typed kind,
  content hash, size, retention expiry, payload_deleted audit flag) plus
  workflow_version.retention_days. content_hash is non-unique so shared
  payloads keep independent metadata/retention rows.
- Artifact kinds JSON/text/file/diff/diagnostics mapped from command and
  agent collectors; producers publish once at successful completion
  (immutable), consumers read only declared-dependency inputs via
  ConsumableArtifacts.
- Definitions carry secret references (name+env) only; values resolve
  from host env at execution time, are injected into command env, and
  known resolved values are redacted from logs and artifact payloads.
- Default 30-day payload retention with per-workflow override; an hourly
  cleanup sweep drops expired payloads while preserving all metadata.
- REST: GET run artifacts list + artifact payload download (410 Gone
  when cleaned); frontend RunView inspects metadata and links retained
  downloads.

Tests cover dedup, content addressing, immutability, command+agent
redaction, retention cleanup preserving metadata, missing payloads,
restart durability, and the REST endpoints.

feat: schedule through named resource pools (#319) ... 6a9de7eac8

Add a required run-concurrency cap and optional named resource pools to the
workflow engine. Definitions may declare pools with positive capacities and
nodes may request units from multiple pools. The scheduler acquires every
requested pool plus one implicit run-concurrency slot atomically (all-or-
nothing) before an attempt goes active, so competing ready nodes cannot
oversubscribe the run cap or any pool. Held capacity is recorded in a durable
workflow_resource_lease table (migration v26) and released only when an
attempt settles (success, failure, cancel, budget stop, or restart recovery),
which also survives restart reconciliation and is surfaced per-pool (held +
waiting nodes) in the run detail and UI. Optional cost/token/duration limits
aggregate descendant agent work; omission means unlimited.

feat: migrate agent loops into one-node workflows (#325) ... dbce0bb23a

One-time migration (v26) copies every persisted loop into a one-node
workflow definition/version with the matching trigger and preserved loop
policies, and maps each loop iteration to a historical workflow run +
node attempt. loop_workflow_map keeps loop identifiers resolvable. The
copy is idempotent (skips already-mapped loops) and interrupted-safe
(runs inside the migration transaction). Old and fresh databases both
migrate cleanly.

The original loops tables and the loop REST/MCP/UI surfaces are left
intact as compatibility wrappers for one release; architecture docs note
the planned removal timing.

feat: coordinate worktree shards and workspace leases (#320) ... 78c0eebc01

Add a run-owned bounded pool of worktree shards with durable
workspace leases layered on the #319 resource-pool acquisition path.

- Exclusive leases (default) keep a shard single-writer; path leases
  normalize and compare declared scopes so disjoint scopes share a
  shard while ancestor/exact overlaps serialize.
- Path-leased command nodes are denied repository-wide git mutation
  (stash/reset/checkout/commit/push and configurable extras); commit
  coordinators take exclusive or serialized per-shard capacity.
- Leases are acquired in the same transaction as pool capacity,
  linked to attempts, carry an optional owning host, released only
  after the attempt settles, and are surfaced in the run detail + UI.
- Shards are created host-locally through the existing git worktree
  service; migration v29 adds workflow_workspace_lease.

Tests cover path normalization, ancestor/exact/disjoint overlap, git
restrictions, cleanup on completion/failure/cancel, shard exhaustion,
serialized commit capacity, and durability.

fix: recover loop workflow migration fbe488a9c5

feat: compose pinned subworkflows with map and join (#317) e54566dbf9

feat: branch and repeat workflow policies 337476b3d6

feat: author YAML workflow versions (#324) 085199d310

feat: recover interrupted workflow runs safely (#321) 2161005187

fix: stabilize flaky workflow concurrency test (#332) ... 56d12e01a3

Replace wall-clock timing with observable state so
TestCommandConcurrencyCapAndFailureCancelSiblings is deterministic:

- cap: assert the run-concurrency pool holds 1 unit with "two"
  waiting, then prove serialized dispatch (two starts only after one
  releases) instead of guessing with time.After(100ms).
- failure cancels sibling: "fail" waits for the sibling's pid file
  before failing (no fixed sleep .2), and the grandchild reap is
  polled (process gone) instead of a one-shot Kill check that raced
  with async reaping by init.

No production change; the sibling process-group SIGKILL is correct,
only the test assertion assumed instant reap.

fix: finalize workflow integration 47e05f057c

feat: ship adversarial migration workflow 439e535f73

feat: retire loop execution engine 3085b27c54

feat: refine workflow builder controls 8645759be8

chore: remove workflow UI prototype 08707f3fcf

feat: return child results to parents f0bda58dfb

chore: remove legacy loops a870585adb

feat: refine workflow run navigation e5324cbf78

fix: remove optimistic thread messages bd6174f2ff

fix: keep session switches at conversation tail 06af553035

feat: refine workflow controls beaaaf78e4

feat: install embedded OpenCode skills eac9f2d1ff

refactor: extract OpenCode skill installer 03fa6d2cd1

feat: expose workflow schema to agents 0088ab02b7

fix: clarify workflow validation errors 4fdd3ddbcd

fix: align pinned session rows e1e1410121

fix: unify queued shell commands 9a104e6c50

dries added 1 commit 2026-07-15 19:25:21 +02:00

fix: stabilize workflow CI checks e17646eac8

dries added 1 commit 2026-07-15 20:36:35 +02:00

fix: stabilize frontend workflow checks 8520373f98

forgejo-actions commented 2026-07-15 20:43:38 +02:00

Copy link

Coverage ratchet: ✅ pass

Suite	Baseline	This PR	Δ
go	72.80%	74.40%	+1.60	✅
frontend	62.94%	64.28%	+1.34	✅

Tolerance: -0.1%. Baseline stored on gh-pages.

<!-- coverage-ratchet --> ### Coverage ratchet: ✅ pass | Suite | Baseline | This PR | Δ | | |---|---|---|---|---| | go | 72.80% | 74.40% | +1.60 | ✅ | | frontend | 62.94% | 64.28% | +1.34 | ✅ | _Tolerance: -0.1%. Baseline stored on `gh-pages`._

dries added 9 commits 2026-07-15 22:02:50 +02:00

feat: publish command node results (#339) 6e2aaa0551

fix: identify MCP caller session d99ec1272a

feat: publish agent node results (#340) 464cb78a2c

fix: recover queues after missed idle events 6013b84be0

feat: interpolate workflow node results (#341) c0a493a55e

feat: publish aggregate node results (#342) ae04038c08

feat: show session answer duration 547b720692

feat: compact reasoning display 4bdc15eecc

feat: retire workflow output collectors (#343) 8c44a26411

dries added 1 commit 2026-07-15 23:14:54 +02:00

test: cover workflow persistence lifecycle (#343) 47fcc461ec

dries added 1 commit 2026-07-15 23:19:26 +02:00

ci: run tmux tests in coverage job 804e4281c0

dries added 1 commit 2026-07-15 23:24:25 +02:00

fix(ci): install tmux without sudo f79441e14b

dries added 1 commit 2026-07-15 23:59:33 +02:00

fix(ci): set terminal type for tmux tests 41122149af

dries added 1 commit 2026-07-16 00:24:08 +02:00

test: cover workflow core paths (#343) 173fd30b41

dries added 1 commit 2026-07-16 00:34:06 +02:00

feat: add conversation metadata controls and timing ca553f69fa

dries merged commit ca553f69fa into main 2026-07-16 00:48:05 +02:00

dries deleted branch feat/agent-workflows 2026-07-16 00:48:05 +02:00

dries referenced this pull request 2026-07-16 00:55:59 +02:00

feat: first-class DAG workflow engine #311

dries referenced this pull request 2026-07-16 00:55:59 +02:00

feat: publish structured workflow node results #337

dries referenced this pull request 2026-07-16 00:55:59 +02:00

fix: send_message_to_parent fails because child cannot learn its own child_session_id #333

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

backend

Go backend code

  

bug

Something is broken or incorrect

  

chore

Maintenance, tooling, or cleanup

  

duplication

Code duplication to be addressed

  

effort:complex

Significant effort or risk

  

effort:medium

Moderate effort required

  

effort:trivial

Quick fix, low complexity

  

enhancement

New feature or capability

  

frontend

TypeScript / React frontend code

  

fullstack

Touches both backend and frontend

  

priority:high

Should be addressed soon

  

ready-for-agent

Agent-grabbable ticket, fully specced

  

refactor

Code restructuring without behaviour change

  

security

Security related

No labels

backend

bug

chore

duplication

effort:complex

effort:medium

effort:trivial

enhancement

frontend

fullstack

priority:high

ready-for-agent

refactor

security

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

dries/ocman!338

No description provided.