8000 fix: master CI red + live e2e run 8 (second intent) gate false positives by adeelahmad · Pull Request #12 · adeelahmad/aloop · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,8 @@ tag in lockstep. Tag releases as `vMAJOR.MINOR.PATCH`.
## [Unreleased]

### Added
- **Second live e2e intent**: `make e2e-live YES=1 INTENT=wordfreq` runs two dependent stories
(word counting, then top-N ranking).
- **`gate-final` checks every planned test is on main**: each `path::fn` bullet in a story's
`plan.md` must exist at its path. `plan-ready.md` lists only what RED wrote, so a test a
re-plan dropped used to vanish unnoticed (e2e run 6).
Expand Down Expand Up @@ -89,6 +91,17 @@ tag in lockstep. Tag releases as `vMAJOR.MINOR.PATCH`.
reading it stays allowed. 2 checks failed before.

### Fixed
- `ensure-tools` rebuilds the bundled md-db / ctx-symbols when their sources change, not
only on a version bump; e2e run 9 ran a stale ctx-symbols that still had the duplicate bug.
- `gate-stage2-complete` names the expected `sN-NN-<slug>/` story dir (and a misnamed one it
found); the planner prompt states the naming. e2e run 9: 56 planner retries on `S1-01/`.
- Scope globs are no longer expanded against the worktree: `tests/**` became the files that
happened to exist, so GREEN was blocked for an in-scope `tests/common/mod.rs` (e2e run 8: 29 blocks).
- `ctx-symbols conflicts` no longer reports the same name in two integration-test files
(`mod common;`, a local helper) as a HIGH duplicate; each `tests/*.rs` is its own crate.
e2e run 8's structural review halted 57 times on it. The grep fallback skips `tests/` too.
- `gate-scaffold-verify`'s grep fallback (no ctx-symbols) located a test fn instead of the stub,
depending on directory order; it now uses the same name-anchored match as `symbol_count`.
- `gate-red-verify` no longer blocks a RED report that lists a test already on the base as
PASS (an earlier task's test, for context); only a NEW test may not pass. e2e run 7: 22 blocks.
- `make shellcheck` no longer reports a lint error as "shellcheck not installed" and passes.
Expand Down
3 changes: 2 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,7 @@ test-hosts: ## Budget/worktree regression + Codex hooks emulation + OpenCode plu
python3 scripts/test/supervisor-scope.test.py
python3 scripts/test/hook-matchers.test.py
python3 scripts/test/dispatch.test.py
python3 scripts/test/ensure-tools.test.py
python3 scripts/test/fail-closed.test.py
@command -v node >/dev/null && node scripts/test/opencode-plugin.test.mjs || { echo "node not found — skipped OpenCode test (CI runs it)"; }

Expand All @@ -95,7 +96,7 @@ conformance: ## ROADMAP §6 conformance fixtures on today's gates (needs cargo;
.PHONY: e2e-live
e2e-live: ## Live end-to-end run: one-line intent -> planned, tested, merged code (YES=1; spends dollars)
@test -n "$(YES)" || { echo "spends tokens: make e2e-live YES=1"; exit 2; }
scripts/smoke/e2e-live.sh --yes
scripts/smoke/e2e-live.sh --yes $(if $(INTENT),--intent $(INTENT))

.PHONY: smoke-live
smoke-live: ## Live Claude Code session: hooks fire, supervisor write blocked, transcripts + ledger (YES=1; spends ~$0.005)
Expand Down
2 changes: 1 addition & 1 deletion docs/ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ spec; this table is the plan. Versions are targets, not promises.
| Milestone | Target | Contents | Status |
|---|---|---|---|
| **M0 — Hygiene** | v0.10.x | Mark `DESIGN` / `ARCHITECTURE` / `KICKSTART` historical; fix stale counts (8 → 9 roles) and the README CI section; add the `CONTRIBUTING.md` the README badge links to; one live-session smoke test of the hook wiring (open since `DEVLOG` v0.1) | done: live smoke test `make smoke-live YES=1` (6 checks, ~$0.004 on haiku); its first run found the Linux fail-open in `gate-supervisor-scope` (fixed, regression test added) |
| **M0.8 — North-star run** | v0.11.x | `make e2e-live YES=1`: a one-line intent (semver parser) goes through retrospective, intake, standards, planner, RED/SCAFFOLD/GREEN per task, structural review and the final gate to merged, tested code in a real Claude Code session, with the approval given up front. Reason: nothing had ever run the whole loop live; the first run found that no role gate fired on plugin installs | run 1: done end to end, 19 min, $1.15 (sonnet supervisor, haiku workers), 22 tests on main, 9/11 checks; found and fixed the `aloop:<role>` matcher bug, a selfcheck crash and copy-merging. Run 2: **11/11**, 12m46s, $0.92, host gates on every role, worktree branches merged with `--no-ff`; found gates looping on the supervisor's contract slips (fixed). Run 3: 11/11, 12m00s, **$0.79**; found GREEN looping on RED's unformatted tests (fixed: RED/SCAFFOLD run the format check) and selfcheck missing the contract in Claude Code worktrees (fixed: the contract is copied in). Run 4 (all fixes): 11/11, 17m56s, $1.10, 12 gate blocks and no loops (28 → 17 → 12). Cost varies $0.79–1.15 per run; next: cut it (scaffold symbol parsing slips such as `(enum,`, supervisor turns), then a second intent beyond the semver fixture. Run 5: 11/11 but 29 min, $1.83, 44 blocks (the planner made 5 tasks; 27 false isolation blocks after workers `cd`'d to the main tree, fixed; subagents also read the plugin's own source, which costs tokens). Run 6: 11/11, **11m10s, $0.65**, plugin-source reads 30 → 2 after the agent prompts said not to; found a dispatch-before-contract loop (25 blocks, fixed by `gate-dispatch`) and annotated scaffold-symbol lines (fixed). Run 7 (after #9): 11/11, 13m46s, $0.98, 20 blocks, no dispatch loop; found RED blocked for listing an earlier task's passing tests, scaffold-symbols tables with a header row and trait-impl lines, and a guessed memory.md frontmatter (all fixed; `aloop init` now seeds memory.md). The re-plan coverage loss is now caught by `gate-final` (every planned test must be on main). Next: supervisor-written GREEN instructions overreaching the next task, then a second intent |
| **M0.8 — North-star run** | v0.11.x | `make e2e-live YES=1`: a one-line intent (semver parser) goes through retrospective, intake, standards, planner, RED/SCAFFOLD/GREEN per task, structural review and the final gate to merged, tested code in a real Claude Code session, with the approval given up front. Reason: nothing had ever run the whole loop live; the first run found that no role gate fired on plugin installs | run 1: done end to end, 19 min, $1.15 (sonnet supervisor, haiku workers), 22 tests on main, 9/11 checks; found and fixed the `aloop:<role>` matcher bug, a selfcheck crash and copy-merging. Run 2: **11/11**, 12m46s, $0.92, host gates on every role, worktree branches merged with `--no-ff`; found gates looping on the supervisor's contract slips (fixed). Run 3: 11/11, 12m00s, **$0.79**; found GREEN looping on RED's unformatted tests (fixed: RED/SCAFFOLD run the format check) and selfcheck missing the contract in Claude Code worktrees (fixed: the contract is copied in). Run 4 (all fixes): 11/11, 17m56s, $1.10, 12 gate blocks and no loops (28 → 17 → 12). Cost varies $0.79–1.15 per run; next: cut it (scaffold symbol parsing slips such as `(enum,`, supervisor turns), then a second intent beyond the semver fixture. Run 5: 11/11 but 29 min, $1.83, 44 blocks (the planner made 5 tasks; 27 false isolation blocks after workers `cd`'d to the main tree, fixed; subagents also read the plugin's own source, which costs tokens). Run 6: 11/11, **11m10s, $0.65**, plugin-source reads 30 → 2 after the agent prompts said not to; found a dispatch-before-contract loop (25 blocks, fixed by `gate-dispatch`) and annotated scaffold-symbol lines (fixed). Run 7 (after #9): 11/11, 13m46s, $0.98, 20 blocks, no dispatch loop; found RED blocked for listing an earlier task's passing tests, scaffold-symbols tables with a header row and trait-impl lines, and a guessed memory.md frontmatter (all fixed; `aloop init` now seeds memory.md). The re-plan coverage loss is now caught by `gate-final` (every planned test must be on main). Run 8, **second intent** (`INTENT=wordfreq`, two dependent stories): 12/12, 18m05s, $1.53, 54 blocks; found scope globs expanded against the worktree (29 GREEN blocks) and `mod common;` across test crates reported as a HIGH duplicate (57 review halts), both fixed. Run 9 (wordfreq again): 12/12, 19m42s, $1.56, 29 blocks; the ctx-symbols fix had not reached the run (`ensure-tools` rebuilt only on a version bump, fixed) and the planner named story dirs `S1-01/` (block message now names `s1-01-<slug>/`). Next: rerun wordfreq with the rebuilt tools, then supervisor-written GREEN instructions overreaching the next task |
| **M0.6 — Gates fail closed** | v0.11.x | A gate that crashes (exit 1, e.g. an unbound variable) is a non-blocking hook error on Claude Code, so the action goes through. Make every gate trap unexpected errors into exit 2 with the reason, and add a conformance case per gate. Reason: the smoke run showed one real instance | done: `_failclosed.sh` on all 10 sub-agent gates + supervisor-scope inside a sprint; `scripts/test/fail-closed.test.py` (5 checks). Left fail-open on purpose: `gate-commit-author` and `gate-validate-artifact` outside a sprint, which see every session on the machine |
| **M0.7 — Supervisor-owned task contract** | v0.11.x | The contract lives in the main tree, `.agentic/state/tasks/<TASK_ID>.env`, and the dispatch prompt's first line names the task; gates and the time box find it from the host's transcript, a worktree `task.env` that disagrees blocks, and sub-agents may not Write/Edit `.agentic/state/` or `task.env`. Settled by a live probe: Claude Code's `isolation: "worktree"` makes `.claude/worktrees/agent-<id>` at dispatch, so a pre-written in-worktree contract was impossible there. Bash writes are guarded too (a command naming the contract with a write verb is blocked; a heuristic, so an obfuscated write still gets through). On hosts with no sub-agent transcript the worktree's `TASK_ID` (or its `.agentic/worktrees/<id>` path) picks the contract | done: `make conformance` (30) + `scripts/test/{budget-worktree,supervisor-scope}.test.py` |
| **M0.5 — Rename to aloop** | v0.11.0 | Plugin, marketplace, skill, CLI (`aloop`, `aloop init`), OpenCode plugin file and docs renamed; `agentic`, `agentic init`, `agentic-init` and `agentic bind` kept as aliase F438 s; OpenCode/Codex installs clean up pre-rename files. Data layout and settings keys unchanged (moved to M1). GitHub repo rename is the owner's call | done: `scripts/test/rename-compat.test.py` |
Expand Down
3 changes: 2 additions & 1 deletion plugin/agents/planner.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,8 @@ until this persona's output is complete and Stage-2-valid.
## Mandate
- Step 1 stories.md -> Step 2 tasks.md -> Step 3 validate.md -> Step 4 plan.md
(tests-only) -> Step 5 sprintN/plan.md (waves, deps, parallelism).
- Keep artifacts consistent: every story has a dir; no "TBW" at Stage 2.
- Keep artifacts consistent: every story has a dir `sprintN/sN-NN-<slug>/` (lowercase,
e.g. `s1-01-count-words/` for S1-01) holding its tasks.md, validate.md and plan.md; no "TBW" at Stage 2.

## Sprint containment (every sprint is a clean-context unit)
The supervisor CLEARS ITS CONTEXT between sprints. Nothing survives except what is on
Expand Down
9 changes: 7 additions & 2 deletions plugin/bin/_gatelib.sh
Original file line number Diff line number Diff line change
Expand Up @@ -284,8 +284,13 @@ changed_paths() {

# True if $1 matches any space-separated glob in $2.
path_matches_any() {
local path="$1" globs="$2" g
for g in $globs; do
# Split on whitespace WITHOUT pathname expansion: `tests/**` must stay a pattern, not
# become the files that happen to exist in the cwd (live e2e run 8).
local path="$1" g
local -a globs
read -r -a globs <<<"$2"
for g in "${globs[@]}"; do
# shellcheck disable=SC2254 # $g is a glob pattern on purpose
case "$path" in $g) return 0;; esac
done
return 1
Expand Down
11 changes: 8 additions & 3 deletions plugin/bin/ensure-tools
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,11 @@ LOG="$DATA/install.log"
LOCK="$DATA/install.lock"
STAMP="$DATA/bin/.version"
VERSION="$(sed -nE 's/.*"version"[[:space:]]*:[[:space:]]*"([^"]+)".*/\1/p' "$ROOT/.claude-plugin/plugin.json" 2>/dev/null | head -1)"
# The build stamp is the version PLUS a fingerprint of the tools' sources: a fix to a bundled
# tool reaches every session without a version bump (live e2e run 9 ran a stale ctx-symbols).
SRC_SUM="$(cd "$ROOT/tools" 2>/dev/null && find ctx-symbols md-db -name target -prune -o -type f \
\( -name '*.rs' -o -name Cargo.toml -o -name Cargo.lock \) -print 2>/dev/null | LC_ALL=C sort | xargs cat 2>/dev/null | cksum | cut -d' ' -f1)"
STAMP_VALUE="$VERSION+${SRC_SUM:-0}"

mkdir -p "$TOOLS_BIN" 2>/dev/null || true

Expand All @@ -50,11 +55,11 @@ find_tool() {

have_all() { find_tool md-db >/dev/null && find_tool ctx-symbols >/dev/null; }

# Our own build is stale when the plugin version moved (a user-installed copy elsewhere
# Our own build is stale when the plugin version or the tools' sources moved (a user-installed copy elsewhere
# on PATH is theirs to manage — we never rebuild over it).
own_stale() {
[ -x "$TOOLS_BIN/md-db" ] || [ -x "$TOOLS_BIN/ctx-symbols" ] || return 1
[ "$(cat "$STAMP" 2>/dev/null)" != "$VERSION" ]
[ "$(cat "$STAMP" 2>/dev/null)" != "$STAMP_VALUE" ]
}

build_running() {
Expand All @@ -75,7 +80,7 @@ do_build() {
&& cp "$tgt/ctx-symbols/release/ctx-symbols" "$TOOLS_BIN/ctx-symbols" || rc=1
( cd "$ROOT/tools/md-db" && CARGO_TARGET_DIR="$tgt/md-db" cargo build --release --locked ) \
&& cp "$tgt/md-db/release/md-db" "$TOOLS_BIN/md-db" || rc=1
[ "$rc" = 0 ] && printf '%s\n' "$VERSION" > "$STAMP"
[ "$rc" = 0 ] && printf '%s\n' "$STAMP_VALUE" > "$STAMP"
echo "==> done rc=$rc"
return "$rc"
}
Expand Down
4 changes: 3 additions & 1 deletion plugin/bin/gate-scaffold-verify
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,9 @@ for sym in $symbols; do
if have_ctx_symbols; then
file="$(ctx-symbols locate "$sym" --tree "$root" | head -1 | cut -d: -f1)"
else
file="$(grep -rlE "\\b(fn|struct|enum|impl)\\b.*$sym" --include='*.rs' "$root" 2>/dev/null | grep -v '/target/' | head -1)"
# the same name-anchored definition match as symbol_count (`.*sym` also hit `fn sub_works`
# in tests/, so the answer depended on directory order)
file="$(grep -rlE "\\b(fn|struct|enum|trait|type|const|static|mod)\\s+$sym\\b" --include='*.rs' "$root" 2>/dev/null | grep -v '/target/' | head -1)"
fi
[ -z "$file" ] && fail "scaffolded symbol \`$sym\` not found on disk"
grep -q 'SUB-AGENT-TODO' "$file" \
Expand Down
7 changes: 6 additions & 1 deletion plugin/bin/gate-stage2-complete
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,12 @@ if [ -f "$stories" ]; then
# match dir like s1-01-<slug> for S1-01
lc="$(echo "$id" | tr 'A-Z' 'a-z')" # s1-01
dir="$(find "$sprint_dir" -maxdepth 1 -type d -name "${lc}-*" | head -1)"
[ -n "$dir" ] || { echo "BLOCK[$GATE_NAME]: no story dir for $id" >&2; exit 2; }
if [ -z "$dir" ]; then
echo "BLOCK[$GATE_NAME]: no story dir for $id: expected $sprint_dir/${lc}-<slug>/ (lowercase, e.g. ${lc}-count-words/) holding tasks.md, validate.md and plan.md" >&2
other="$(find "$sprint_dir" -maxdepth 1 -type d -iname "${lc}*" | head -1)"
[ -n "$other" ] && echo " found $other: rename it, e.g. git mv '$other' '$sprint_dir/${lc}-<slug>'" >&2
exit 2
fi
for f in tasks.md validate.md plan.md; do
[ -f "$dir/$f" ] || { echo "BLOCK[$GATE_NAME]: $id missing $f" >&2; exit 2; }
done
Expand Down
5 changes: 3 additions & 2 deletions plugin/bin/gate-structural-integrity
Original file line number Diff line number Diff line change
Expand Up @@ -26,9 +26,10 @@ if have_ctx_symbols; then
note "structural review clean (no HIGH findings)"
else
warn "ctx-symbols absent; grep fallback cannot see semantic duplication (weaker check)"
# Fragile fallback: flag any top-level fn name defined in 2+ files.
# Fragile fallback: flag any top-level fn name defined in 2+ files. Not under tests/: each
# integration-test file is its own crate, so helpers there cannot collide.
dups="$(grep -rhoE '^[[:space:]]*(pub[[:space:]]+)?fn[[:space:]]+[A-Za-z_][A-Za-z0-9_]*' \
--include='*.rs' "$root" 2>/dev/null | grep -v '/target/' \
--include='*.rs' --exclude-dir=target --exclude-dir=tests "$root" 2>/dev/null \
| sed -E 's/.*fn[[:space:]]+//' | sort | uniq -d)"
if [ -n "$dups" ]; then
echo "HIGH (grep): duplicate fn name(s): $(echo "$dups" | tr '\n' ' ')" >&2
Expand Down
3 changes: 2 additions & 1 deletion plugin/pipeline/planning/02-planner/persona.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,8 @@ until this persona's output is complete and Stage-2-valid.
## Mandate
- Step 1 stories.md → Step 2 tasks.md → Step 3 validate.md → Step 4 plan.md
(tests-only) → Step 5 sprintN/plan.md (waves, deps, parallelism).
- Keep artifacts consistent: every story has a dir; no "TBW" at Stage 2.
- Keep artifacts consistent: every story has a dir `sprintN/sN-NN-<slug>/` (lowercase,
e.g. `s1-01-count-words/` for S1-01) holding its tasks.md, validate.md and plan.md; no "TBW" at Stage 2.

## Sprint containment (every sprint is a clean-context unit)
The supervisor CLEARS ITS CONTEXT between sprints. Nothing survives except what is on
Expand Down
52 changes: 50 additions & 2 deletions plugin/tools/ctx-symbols/src/conflicts.rs
Original file line number Diff line number Diff line change
Expand Up @@ -115,14 +115,25 @@ pub fn count_symbol(root: &Path, symbol: &str) -> Result<Vec<(PathBuf, usize)>>
pub fn detect_all(root: &Path) -> Result<Vec<Finding>> {
let index = index_tree(root)?;
let mut out = Vec::new();
out.extend(detect_duplicate_definitions(&index));
out.extend(detect_duplicate_definitions(root, &index));
out.extend(detect_duplicate_bodies(root, &index)?);
out.extend(detect_orphans(root, &index)?);
out.sort_by_key(|f| if f.severity() == Severity::High { 0 } else { 1 });
Ok(out)
}

fn detect_duplicate_definitions(index: &[(PathBuf, String, SymbolDef)]) -> Vec<Finding> {
/// A file under a `tests/` directory: in Rust each such file is its own integration-test crate.
fn in_test_crate(root: &Path, file: &Path) -> bool {
file.strip_prefix(root)
.unwrap_or(file)
.components()
.any(|c| c.as_os_str() == "tests")
}

fn detect_duplicate_definitions(
root: &Path,
index: &[(PathBuf, String, SymbolDef)],
) -> Vec<Finding> {
let mut by_name: BTreeMap<String, Vec<(PathBuf, usize)>> = BTreeMap::new();
for (file, _src, sym) in index {
by_name
Expand All @@ -133,6 +144,13 @@ fn detect_duplicate_definitions(index: &[(PathBuf, String, SymbolDef)]) -> Vec<F
by_name
.into_iter()
.filter(|(_, locs)| locs.len() >= 2)
// the same name once per integration-test file is one per crate: no collision
.filter(|(_, locs)| {
let mut files: Vec<&PathBuf> = locs.iter().map(|(f, _)| f).collect();
files.sort();
files.dedup();
!(files.len() == locs.len() && files.iter().all(|f| in_test_crate(root, f)))
})
.map(|(name, mut locations)| {
locations.sort();
Finding::DuplicateDefinition { name, locations }
Expand Down Expand Up @@ -273,6 +291,36 @@ mod tests {
&& f.severity() == Severity::High));
}

#[test]
fn same_name_in_separate_integration_test_crates_is_not_a_duplicate() {
// Each tests/*.rs is its own crate: `mod common;` and a local helper in two of them
// cannot collide (live e2e run 8 halted the chain 57 times on exactly this).
let dir = tempfile::tempdir().unwrap();
let t = "mod common;\nfn map() -> u32 { 1 }\n#[test]\nfn t() { assert_eq!(map(), 1); }\n";
write(dir.path(), "tests/count.rs", t);
write(dir.path(), "tests/top.rs", t);
write(dir.path(), "tests/common/mod.rs", "pub fn shared() {}\n");
let findings = detect_all(dir.path()).unwrap();
assert!(
!findings
.iter()
.any(|f| matches!(f, Finding::DuplicateDefinition { .. })),
"{findings:?}"
);
}

#[test]
fn a_test_crate_redefining_a_source_symbol_is_still_high() {
let dir = tempfile::tempdir().unwrap();
write(dir.path(), "src/lib.rs", "pub fn top_n() {}\n");
write(dir.path(), "tests/top.rs", "fn top_n() {}\n");
let findings = detect_all(dir.path()).unwrap();
assert!(findings
.iter()
.any(|f| matches!(f, Finding::DuplicateDefinition { .. })
&& f.severity() == Severity::High));
}

#[test]
fn unique_symbol_has_no_duplicate_finding() {
let dir = tempfile::tempdir().unwrap();
Expand Down
Loading
Loading
0