Repository navigation
build(spec-harness): runnable executable-spec test harness - #67
Merged
Merged
Conversation
Python harness that runs docs/**/*.spec.md per the grammar in docs/executable_spec.md: execution contexts with accumulated ancestor preambles, isolated per-leaf temp dirs, the 📄/✏️/⌨️/```bash/🖥️/🚨 markers, output normalisation, exit-code defaults, and 🟡 skip reporting. Includes unit tests and a discovery runner. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012h4Cz9NyGC91eQuxU2XSzb
…abit-hooks The spec harness is dev/test tooling, not the main project. Collapse the specharness package into a single tests/harness.py module (parser, executor, and CLI runner) and point the root pyproject.toml at habit-hooks itself (package = false; pytest configured with tests/ on the path). Unit tests run via `uv run pytest`; specs via `uv run python tests/harness.py`. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012h4Cz9NyGC91eQuxU2XSzb
Dogfooded habit-hooks (npm v0.2.0, python preset) against the harness; it flagged two over-complex functions and an unused import. Address the coaching rather than the metric: - Replace the hand-rolled Markdown tokenizer with markdown-it-py (a real CommonMark parser) — no more low-level line scanning. - Turn the two if/elif dispatch chains (_build_steps, execute) into polymorphic step objects (step.apply(ctx)) plus a small _build_marker builder, so each marker/step owns its own behaviour. - Drop the unused test import. Behaviour unchanged: 23 unit tests pass, adapter.spec.md green, mapper/snoozer 🟡 cases skipped. habit-hooks now reports clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012h4Cz9NyGC91eQuxU2XSzb
Reviewer flagged that the suite proved equivalence but didn't pin the cases most at risk from swapping in markdown-it: multiple markers grouped into one paragraph, the explicit-dest copy form 📄dst @src, and a required payload block missing on ⌨️. Add tests for each and a comment explaining why marker extraction reads from paragraph inline tokens. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012h4Cz9NyGC91eQuxU2XSzb
…tokens _elements scanned the flat token stream with manual index arithmetic (tokens[i+1], tokens[i-1]) — under ruff's complexity threshold, so habit-hooks stayed silent, but procedural and hard to read. Use markdown-it-py's SyntaxTreeNode and dispatch over semantic node types (heading / fence / paragraph) with two small pure helpers (_heading, _markers). No index math, no token-peeking. Behaviour unchanged: 26 unit tests pass, adapter green, mapper/snoozer skipped, habit-hooks clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012h4Cz9NyGC91eQuxU2XSzb
…unner Researched existing packages before writing more code: - Markdown parsing already delegates to markdown-it-py (no hand-rolled tokenizer). - The bespoke CLI runner (discover/run_file/main/glyph printing) is exactly what a test framework should do. Replace it with a ~50-line pytest conftest (pytest_collect_file) that turns each leaf spec case into a pytest item, so `uv run pytest` reports the harness unit tests AND the specs with native pass/skip/fail and 🟡 skips. - Sybil (the docs-testing library) was evaluated and rejected: it evaluates regions once top-to-bottom in one shared namespace with no filesystem isolation, which conflicts with the contract's per-leaf temp-dir isolation and ancestor-preamble re-running. cram/prysk only parse their own .t format, not our Markdown+emoji grammar. So the marker/context engine is irreducible; only the runner was removable. harness.py is now the pure engine (parse_spec + execute). deptry config declares `harness` first-party and allows pytest in conftest glue. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012h4Cz9NyGC91eQuxU2XSzb
parse_spec carried two responsibilities — build the heading tree, then turn leaf nodes into test cases. Split on that boundary: _build_tree (headings → nodes owning their elements) and _leaf_case (a leaf's ancestry → its ordered steps). parse_spec is now one sentence: "the markdown's leaf contexts, each as a runnable test case." Folded the single-use _ancestry walk into _leaf_case. Behaviour unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012h4Cz9NyGC91eQuxU2XSzb
…ngth sensor harness.py did several things in one 351-line file. Split it into a cohesive package (tests/harness/): glyphs, errors, text, markdown (md → elements), steps (Context + step classes), parser (TreeBuilder/StepBuilder classes → SpecCases), runner. Every module is < 200 lines and every function/method is ≤ 12 body lines. Replaced the nested flush() closure with StepBuilder/TreeBuilder classes (stateful accumulators as methods, no nested functions). Added a function/method length sensor (tests/tools/oversized_function.py) that emits the habit-hooks finding contract for the `oversized-function` smell — the gap habit-hooks' Python preset can't see (ruff has no per-function line rule; it's ESLint-only). 0.2.0 has no consumer-sensor config hook, so it is enforced via pytest (test_function_length.py) over all our Python, including the test helpers. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012h4Cz9NyGC91eQuxU2XSzb
devill
force-pushed
the
claude/spec-harness-unit-tests-blxsm1
branch
from
June 28, 2026 06:38
5176f85 to
65770d8
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds the Python harness that runs the executable specs (
docs/**/*.spec.md) per the grammar indocs/executable_spec.md— the test gate the rest of thesimplifiedrebuild depends on. It's a dev/test tool, not a shippedhabit-*command. No product behaviour is implemented here.How it works
📄<path>write (+@<src>copy form),✏️<VAR>env,⌨️stdin,```bashcommand,🖥️ ✅/🖥️ ❌ <N>exit assertion,🚨stderr assertion.```bashblocks are never consumed as a marker payload; bare non-bash blocks are cosmetic.🖥️/🚨must equal the stream after normalising (ANSI stripped, trailing whitespace per line trimmed, trailing blank lines dropped). No block → unchecked.🟡on a heading marks the test skipped.Files
pyproject.toml—uvproject, zero runtime deps,pytestdev groupspecharness/harness.py— parser + executorspecharness/__main__.py— discovery runner and pass/skip/fail summarytests/test_harness.py— unit suiteVerification
uv run pytest→ 23 passed (one per marker, plus context isolation, ancestor-preamble accumulation, skip reporting, output normalisation, exit-code defaults).docs/adapter.spec.md→ 2 passed, 0 skipped, 0 failed (purejq, fully green).docs/mapper.spec.md(11) anddocs/snoozer.spec.md(3) → all🟡cases reported skipped, no errors.Notes
docs/building-a-plugin.spec.md→## 4. Try itreports a real fail: it pipeshabit-sensors | habit-mapper, which don't exist yet and isn't marked🟡. Left as an honest failure rather than masked — implementing that product code and reconciling the docs are out of scope for this phase.📄 @<src>copy resolves<src>relative to the repo root (the harness invocation cwd). Untested by any current spec example.🤖 Generated with Claude Code
https://claude.ai/code/session_012h4Cz9NyGC91eQuxU2XSzb
Generated by Claude Code