feat(devops): add auto-evolution loop (PR review + BMAD pipeline) (#94)

* feat(devops): add auto-evolution loop (PR review + BMAD bug/feature pipeline)

Extends the DevOps Hand to periodically scan configured GitHub repos and:
- review open PRs via the existing code-reviewer sub-agent, posting a
  single COMMENT review back to GitHub (never auto-APPROVE)
- triage open issues via labels first, single-prompt LLM fallback
- dispatch actionable issues (bug-fix / feature) to a new implementer
  sub-agent which runs the BMAD pipeline (Brainstorm -> Architect ->
  PRD -> Implement) scaled by bmad_strictness and produces a DRAFT PR

Safety floor (always on):
- draft PRs only, never auto-ready, never merge
- never push to main/master/protected branches
- escalates to devops_queue.json when touching workspace Cargo.toml,
  migrations, secrets, or >30 changed files
- 70% per-turn token budget cap so subsequent ticks have headroom

New settings: auto_evolve, evolution_repos, evolution_check_interval,
bmad_strictness. New sub-agent: agents.implementer. New SKILL.md
sections: Issue Triage Playbook, PR Review Automation, Bug Fix
Playbook, BMAD Feature Pipeline, Draft PR Creation. Three new
dashboard metrics: prs_reviewed, issues_processed, draft_prs_opened.

* fix(devops): address PR review — close blocking + medium + style issues

Blocking (5):
- add max_changed_files setting (was referenced in implementer prompt
  but never defined)
- drop metering_query reference (tool isn't in tools = [...] list);
  agent self-paces against budget instead
- fix \n\n literal in jq --arg for issue cross-link comment; compose
  body in shell with printf so newlines survive
- resolve BASE_BRANCH via /repos/owner/repo .default_branch instead
  of relying on an undefined variable
- complete reviewer-verdict → GitHub review-event mapping (4 cases,
  not just request_changes); block routes through REQUEST_CHANGES
  with a blocking-prefix in the body, approve downgrades to COMMENT

Medium (5):
- correct Phase 6 → Phase 7 in the auto-evolution settings comment
- remove schedule_create busy-loop confusion; Phase 7 fires per-turn
  while the Hand is already frequency = "continuous", with cadence
  enforced via devops_evolution_cursor memory key
- generalize the forbid-main-worktree wording — discover and honor
  whatever pre-commit / pre-push / commit-msg hooks the upstream
  repo configures (was librefang-specific)
- clarify the AI-attribution rule: ban LLM-vendor attribution
  (Claude, GPT, 🤖, etc.) but allow process attribution
  (DevOps Hand → implementer) for traceability
- add USER_TYPE = "Bot" short-circuit that was extracted but never
  applied (bots get a token-cheap skip, not a deep review)

Style (2):
- document the four event_publish event names (devops_evolution_*)
  in a new SKILL.md table alongside the memory-keys table
- justify implementer's max_history_messages = 100 with a comment
  (BMAD 4 phases × cargo build/test chains needs headroom)

* docs(devops): tighten evolution snippets (D1-D4 second-review nits)

D1 -- show SUMMARY_BODY (and VERDICT) assignment in PR review snippet:
add explicit jq -r .summary / .verdict extraction from reviewer_output.json
so the agent reading SKILL.md doesn't have to infer where these come from.

D2 -- reword strict-mode wait semantics in both HAND.toml and SKILL.md:
'Stop. Wait...' was misleading because the agent loop has no in-turn
pause primitive. Now spells out: end the current turn after queueing,
let the continuous tick re-read the queue, resume on approved / skip
on pending / abandon on rejected. Explicitly forbids busy-wait and
sleep loops.

D3 -- restructure bot / huge-diff short-circuit so agent-tool calls are
expressed as numbered agent steps, not as '# memory_store ...' comments
inside a bash block. The bash block now only extracts cheap signals;
the decision and the tool calls are clearly agent-level.

D4 -- remove the misleading 'exit 0' from the short-circuit bash and
add a one-liner noting that exit 0 inside shell_exec only ends one
shell session, not the Phase 7 pass; the agent must choose to move on.
This commit is contained in:
Evan authored and GitHub committed 2026-05-14 15:54:42 +09:00
1 parent 990d22009c
commit d215388039
3 files changed
+740 -1

No files matched your search

+272
View File
@@ -123,6 +123,11 @@ aliases = [
"infrastructure monitoring",
"deployment automation",
"incident response",
"auto evolve",
"review github prs",
"triage issues",
"implement issue",
"fix bug from issue",
]
weak_aliases = [
"deploy",
@@ -131,6 +136,10 @@ weak_aliases = [
"container",
"terraform",
"helm",
"bug fix",
"feature implementation",
"bmad",
"draft pr",
]
# ─── Configurable settings ───────────────────────────────────────────────────
@@ -286,6 +295,91 @@ description = "Queue deployment and infrastructure actions for your review inste
setting_type = "toggle"
default = "true"
# ─── Auto-Evolution settings ─────────────────────────────────────────────────
# These drive the Phase 7 evolution loop: periodic scan of configured
# GitHub repos, automated PR review via the reviewer sub-agent, and
# BMAD-style bug fix / feature implementation via the implementer
# sub-agent. All produce draft PRs and respect approval_mode.
[[settings]]
key = "auto_evolve"
label = "Auto Evolution"
description = "Periodically scan configured GitHub repos and run PR review / issue triage / BMAD implementation"
setting_type = "toggle"
default = "false"
[[settings]]
key = "evolution_repos"
label = "Evolution Target Repos"
description = "Comma-separated owner/repo pairs to watch (e.g. librefang/librefang,librefang/librefang-registry)"
setting_type = "text"
default = ""
[[settings]]
key = "evolution_check_interval"
label = "Evolution Check Interval"
description = "How often to scan target repos for new PRs and issues"
setting_type = "select"
default = "15min"
[[settings.options]]
value = "5min"
label = "Every 5 minutes"
[[settings.options]]
value = "15min"
label = "Every 15 minutes"
[[settings.options]]
value = "1hour"
label = "Every hour"
[[settings.options]]
value = "6hour"
label = "Every 6 hours"
[[settings.options]]
value = "1day"
label = "Daily"
[[settings]]
key = "bmad_strictness"
label = "BMAD Strictness"
description = "How thoroughly to run the Brainstorm-Architect-PRD-Implement pipeline before producing a draft PR"
setting_type = "select"
default = "standard"
[[settings.options]]
value = "light"
label = "Light (skip brainstorm, go straight to architect → implement)"
[[settings.options]]
value = "standard"
label = "Standard (full 4-phase pipeline, draft PR at end)"
[[settings.options]]
value = "strict"
label = "Strict (full pipeline + require human approval at each phase via queue)"
[[settings]]
key = "max_changed_files"
label = "Max Changed Files Per Draft PR"
description = "Implementer stops and queues for human triage if a single draft PR would touch more than this many files. Decompose larger work into multiple PRs."
setting_type = "select"
default = "30"
[[settings.options]]
value = "10"
label = "10 files (very conservative)"
[[settings.options]]
value = "30"
label = "30 files (default)"
[[settings.options]]
value = "100"
label = "100 files (large refactors)"
# ─── Agent configuration ─────────────────────────────────────────────────────
[agents.main]
@@ -543,6 +637,80 @@ Stop the current monitoring/incident session when ANY of these conditions is met
---
## Phase 7 — Evolution Loop (auto_evolve)
Gate: skip entirely unless `auto_evolve` is ENABLED **and** `evolution_repos` is non-empty.
The Hand is already `frequency = "continuous"`, so this Phase fires once per turn while gates pass. On entry, read `memory_recall devops_evolution_cursor_<owner>_<repo>`. If less than `evolution_check_interval` has elapsed since the last tick for THAT repo, skip the repo for this turn — the next turn will check again. Never busy-loop or self-schedule inside a turn.
For every repo in `evolution_repos` (comma-separated `owner/repo` pairs) that passes the cadence gate, interleave PR review and issue triage.
### 7.1 PR Review Pass
1. List open PRs (filter out drafts unless explicitly enabled):
```
curl -s -H "Authorization: Bearer $GITHUB_TOKEN" \\
"https://api.github.com/repos/OWNER/REPO/pulls?state=open&per_page=50" \\
-o open_prs.json
```
2. For each PR, look up `devops_pr_review_<owner>_<repo>_<number>` in memory. Skip if `head_sha` matches the last reviewed sha — already reviewed at this revision.
3. Fetch the diff + file list:
```
curl -s -H "Authorization: Bearer $GITHUB_TOKEN" \\
-H "Accept: application/vnd.github.v3.diff" \\
"https://api.github.com/repos/OWNER/REPO/pulls/NUM" -o pr.diff
curl -s -H "Authorization: Bearer $GITHUB_TOKEN" \\
"https://api.github.com/repos/OWNER/REPO/pulls/NUM/files" -o pr_files.json
```
4. Delegate to the `code-reviewer` sub-agent with: PR title, body, diff, file list, target branch's `AGENTS.md`/`CLAUDE.md` if present. Capture the reviewer's structured output (approve / request changes / block + issues + positives).
5. Post the review back to GitHub:
```
curl -s -X POST -H "Authorization: Bearer $GITHUB_TOKEN" \\
-H "Content-Type: application/json" \\
-d "$REVIEW_BODY_JSON" \\
"https://api.github.com/repos/OWNER/REPO/pulls/NUM/reviews"
```
Event `"COMMENT"` for advisory passes. Reserve `"REQUEST_CHANGES"` for blocking findings flagged by the reviewer; never auto-`"APPROVE"`.
6. Record the result in memory: `memory_store devops_pr_review_<owner>_<repo>_<number>` with `{ head_sha, verdict, timestamp }`. Bump dashboard counter `devops_hand_prs_reviewed`.
### 7.2 Issue Triage + Implementation Pass
1. List open issues that match the configured triage filter (default: issues with no `wontfix` / `duplicate` / `invalid` labels and no existing linked PR):
```
curl -s -H "Authorization: Bearer $GITHUB_TOKEN" \\
"https://api.github.com/repos/OWNER/REPO/issues?state=open&per_page=50" \\
-o open_issues.json
```
2. For each issue, classify via the **Issue Triage Playbook** (see `SKILL.md`):
- Labels first (`bug` / `enhancement` / `feature` / `question`) — cheap, deterministic
- LLM fallback only if labels are absent — single short prompt, never multi-turn
- Result is one of: `bug-fix`, `feature`, `needs-info`, `skip`
3. For `bug-fix` and `feature`, dispatch to the `implementer` sub-agent with the BMAD pipeline whose depth is set by `bmad_strictness`.
4. The implementer produces a **draft PR**. Always draft, never ready-for-review, regardless of `approval_mode` — this is the safety floor on auto-generated code. The user (or another reviewer) marks it ready.
5. Comment on the originating issue with a link to the draft PR and a one-line summary.
6. Record `memory_store devops_issue_state_<owner>_<repo>_<number>` with `{ classification, pr_url, timestamp }`. Bump dashboard counter `devops_hand_issues_processed`.
### 7.3 Safety Floor (NEVER bypass)
- Always create a fresh git worktree per implementation task — never write to the user's working tree.
- Never commit to `main` / `master` / `trunk` directly.
- Never use `--no-verify`, `--force`, or `git push -f` against any remote branch.
- Honor whatever pre-commit / pre-push / commit-msg hooks the upstream repo configures (run via `git config core.hooksPath` discovery + executing each non-skipped hook). Abort the task on hook failure rather than retrying.
- Stop and queue (`devops_queue.json`) if the implementer wants to touch:
- `Cargo.toml` workspace members (any `members = [...]` change)
- migration files (paths under `*/migrations/*`, `*/migrate/*`, or matching `*.sql`)
- any path containing `secrets`, `.env`, `credentials`, `*.pem`, `*.key`, `id_rsa`, `id_ed25519`
- more than the configured `max_changed_files` setting (default 30) files in one PR
- Token budget: each evolution tick must stop on its own when the agent senses it is nearing the per-turn budget (target ~70% so the next tick has headroom). Estimate by tracking cumulative output tokens since turn start; the kernel-enforced hard cap is the upstream guard rail, not the primary control.
### 7.4 Failure Handling
- Network / API errors → exponential backoff, max 3 retries, then surface a `devops_evolution_blocked` event and skip this PR/issue for the current tick.
- Reviewer or implementer sub-agent times out → record a `timed_out` verdict in memory so we don't retry on the same head_sha next tick.
- `git push` rejected (protected branch, stale, etc.) → open the PR target as `wontfix` for this tick, surface to the user via event.
---
## Guidelines
- NEVER execute destructive commands without explicit user confirmation
@@ -647,6 +815,95 @@ OUTPUT FORMAT:
Be thorough but constructive. Focus on bugs and risks, not style preferences."""
[agents.implementer]
invoke_hint = "BMAD pipeline executor — turns an issue into a draft PR via brainstorm, architect, PRD, implement phases"
name = "implementer"
description = "BMAD implementer. Takes a triaged issue (bug or feature) and produces a draft PR following the Brainstorm → Architect → PRD → Implement methodology, scaled by `bmad_strictness`."
module = "builtin:chat"
provider = "default"
model = "default"
max_tokens = 16384
temperature = 0.2
max_iterations = 80
# Raise the history cap above the kernel default. BMAD work fans out
# across 4 phases (Brainstorm / Architect / PRD / Implement), each of
# which spawns shell_exec chains (cargo build/test cycles, git ops,
# file edits). 100 buys enough headroom that a single PR doesn't get
# truncated mid-implementation while still bounding worst-case cost.
max_history_messages = 100
system_prompt = """You are Implementer, the BMAD execution sub-agent inside the DevOps Hand. You convert a single triaged GitHub issue into a draft pull request. You DO NOT push to protected branches, you DO NOT mark PRs ready-for-review, and you DO NOT skip phases unless `bmad_strictness = "light"`.
## Inputs
You will receive:
- `issue`: full GitHub issue payload (title, body, labels, comments)
- `classification`: `"bug-fix"` or `"feature"`
- `repo`: `owner/name`
- `bmad_strictness`: `"light"` | `"standard"` | `"strict"`
- `repo_context`: workspace root path of a freshly-created git worktree off `origin/<default-branch>` — your sandbox
## Pipeline (skip phases per strictness)
### Phase B — Brainstorm (skipped when strictness = light)
- Re-read the issue. What is the actual user-visible problem or capability being asked for? Restate in your own words.
- Generate 2–3 distinct approaches. For each: rough sketch, files touched, risk level, estimated diff size.
- Pick ONE. Record the trade-off justification in a `BMAD.md` you'll commit alongside the change. Length: ≤ 200 words.
### Phase A — Architect
- For the chosen approach, identify exact crates / modules / files to change.
- Decide types, function signatures, and module boundaries before writing code.
- Call out any interface changes that ripple to other crates and confirm the ripple is bounded (or escalate via queue if it isn't).
- Append to `BMAD.md` under `## Architecture`.
### Phase P — PRD (skipped when strictness = light)
- Acceptance criteria as a bulleted checklist (what must pass for the PR to be ready).
- Test plan: enumerate the unit / integration tests you will add or update.
- Rollback plan: how a reviewer can revert if this lands and breaks something.
- Append to `BMAD.md` under `## PRD`.
### Phase I — Implement
- For bug fixes: write a failing test first (TDD), make it pass, then refactor. The failing test must commit before the fix, in the same PR.
- For features: write tests alongside code; do not commit untested branches.
- Use only `shell_exec` + `file_*` tools to edit. Never edit outside `repo_context`.
- Run the project's own lint/test gate (e.g., `cargo clippy --workspace --all-targets -- -D warnings`, `cargo test -p <crate>`). If the project has a `justfile` or `xtask`, prefer those. Fail-fast: if the gate doesn't pass, fix; if it can't be made to pass within `max_iterations`, stop and surface.
## Output
A **draft PR** (`draft: true`) on `repo` whose body contains:
```
## Summary
<one paragraph — what this PR does and why>
## BMAD Pipeline Output
<inline copy of BMAD.md sections, or link to the committed BMAD.md if too large>
## Acceptance Checklist
- [ ] <copied from PRD>
## Risk
<one paragraph>
## Generated By
DevOps Hand → implementer sub-agent (issue: #<num>, strictness: <level>)
```
## Hard Rules (NEVER violate, regardless of strictness)
- ALWAYS work in a fresh git worktree provided as `repo_context`. Never `cd` out of it.
- ALWAYS create a feature branch named `auto/<classification>-<issue-number>-<slug>`.
- NEVER push to `main`, `master`, `trunk`, or any branch protected by ruleset.
- NEVER use `git push --force`, `--no-verify`, `--no-gpg-sign`, or `--amend` against a remote branch.
- NEVER commit anything matching: `.env*`, `*.pem`, `*.p12`, `id_rsa`, `id_ed25519`, `credentials*`, `secrets*`, `vault_*.key`.
- NEVER include LLM-vendor attribution in commit messages or PR bodies — no `Co-Authored-By: Claude`, no `Generated with Claude / GPT / Anthropic / OpenAI`, no `🤖` emoji crediting an AI vendor. "Generated by DevOps Hand → implementer" (process attribution) is fine and encouraged for traceability; vendor attribution is not. Many upstream repos enforce this via commit-msg hook; we apply the rule regardless of upstream enforcement.
- Touch limit: stop at `max_changed_files` (default 30 files). If the implementation legitimately needs more, decompose into multiple draft PRs and stop after the first.
- If any acceptance-test command in PRD fails after your last fix attempt, DO NOT push. Write the partial state + failure details to `devops_queue.json` and surface for human triage.
- Token budget: stop on your own at 70% of the per-turn budget so the next tick has headroom.
## On `bmad_strictness = "strict"`
Between every phase, write the produced artifact to `devops_queue.json` with `phase: "bmad-<letter>-pending"` and `status: "pending"`, then **end the current turn**. The Hand is continuous, so the next tick re-reads the queue: if the user (out-of-band) flipped `status` to `approved`, resume from the next phase; if still `pending`, skip this issue for this tick and re-check on the next one. Never poll or `sleep` for approval within a single turn — the agent loop has no in-turn pause primitive, and busy-waiting would block other Hand work and burn tokens. This is how a human keeps a leash on autonomous code changes without forcing the daemon to stall."""
[dashboard]
[[dashboard.metrics]]
label = "Health Checks Run"
@@ -668,6 +925,21 @@ label = "Deployments Managed"
memory_key = "devops_hand_deployments_managed"
format = "number"
[[dashboard.metrics]]
label = "PRs Reviewed"
memory_key = "devops_hand_prs_reviewed"
format = "number"
[[dashboard.metrics]]
label = "Issues Processed"
memory_key = "devops_hand_issues_processed"
format = "number"
[[dashboard.metrics]]
label = "Draft PRs Opened"
memory_key = "devops_hand_draft_prs_opened"
format = "number"
# ─── Token & Performance Metadata ─────────────────────────────────────────────
[metadata]