8000 Move the competitor harness out of the ASV suite (#64) by rafatahmed · Pull Request #65 · rafatahmed/Flahax · GitHub
[go: up one dir, main page]
Skip to content
{"props":{"contextRegion":{"crumbs":[{"crumb_type":"user","label":"rafatahmed","is_root":false,"href":"/rafatahmed"},{"crumb_type":"repository","label":"Flahax","is_root":false,"href":"/rafatahmed/Flahax"}],"localNavigation":[{"id":"code","icon":"code","label":"Code","href":"/rafatahmed/Flahax","selectedLinks":["repo_source","repo_downloads","repo_commits","repo_releases","repo_tags","repo_branches","repo_packages","repo_deployments","repo_attestations"],"popoverTarget":false,"commandId":"repositories:go-to-code","reactNav":{"appTarget":"code-view","anchor":"code-view-repo-link"},"turboNav":{"frame":"repo-content-turbo-frame"}},{"id":"issues","icon":"issue-opened","label":"Issues","href":"/rafatahmed/Flahax/issues","selectedLinks":["repo_issues","repo_labels","repo_milestones"],"count":1,"popoverTarget":false,"commandId":"repositories:go-to-issues","reactNav":{"appTarget":"repo","anchor":null},"turboNav":{"frame":"repo-content-turbo-frame"}},{"id":"pull-requests","icon":"git-pull-request","label":"Pull requests","href":"/rafatahmed/Flahax/pulls","selectedLinks":["repo_pulls","checks"],"count":0,"popoverTarget":false,"commandId":"repositories:go-to-pull-requests","reactNav":{"appTarget":null,"anchor":null},"turboNav":{"frame":"repo-content-turbo-frame"}},{"id":"discussions","icon":"comment-discussion","label":"Discussions","href":"/rafatahmed/Flahax/discussions","selectedLinks":["repo_discussions"],"popoverTarget":false,"commandId":"repositories:go-to-discussions","reactNav":{"appTarget":null,"anchor":null},"turboNav":{"frame":"repo-content-turbo-frame"}},{"id":"actions","icon":"play","label":"Actions","href":"/rafatahmed/Flahax/actions","selectedLinks":["repo_actions"],"popoverTarget":false,"commandId":"repositories:go-to-actions","reactNav":{"appTarget":"repo","anchor":null},"turboNav":{"frame":"repo-content-turbo-frame"}},{"id":"security-and-quality","icon":"shield","label":"Security and quality","href":"/rafatahmed/Flahax/security","selectedLinks":["security","overview","alerts","policy","token_scanning","code_scanning"],"count":0,"popoverTarget":false,"commandId":"repositories:go-to-security","reactNav":{"appTarget":null,"anchor":null},"turboNav":{"frame":"repo-content-turbo-frame"}},{"id":"insights","icon":"graph","label":"Insights","href":"/rafatahmed/Flahax/pulse","selectedLinks":["repo_graphs","repo_contributors","dependency_graph","dependabot_updates","pulse","people","community"],"popoverTarget":false,"commandId":"repositories:go-to-insights","reactNav":{"appTarget":null,"anchor":null},"turboNav":{"frame":"repo-content-turbo-frame"}}],"localNavigationUpdateChannel":null,"selectedLink":"repo_pulls"},"owner":null,"headerLogo":{"href":"/","aria-label":"Homepage "},"notifications":{"indicatorMode":"disabled","websocketChannel":null,"fetchIndicatorSrc":"/notifications/indicator","fetchIndicatorEnabled":false},"issues":{"href":"/issues"},"pulls":{"href":"/pulls"},"contributedRepos":{"href":"/repos"},"copilot":{"show":false,"showAgentsButton":false,"showRelaunchAnnouncement":false,"copilotChatUrl":null,"copilotApiUrl":"https://api.githubcopilot.com"},"search":{"show":true,"showCommandPalette":false,"isSearchPage":false,"isJumpToSearch":false,"searchContext":{"scope":"repo:rafatahmed/Flahax","current_repo_name":"Flahax","current_repo_nwo":"rafatahmed/Flahax","user_id":"rafatahmed"}},"commandPalette":null,"enterpriseBar":{"show":false},"globalTransactionalMessage":[],"payloadsUrl":"/_global-navigation/payloads.json?can_toggle_site_admin_and_employee_status=0\u0026is_admin_mode_on=0\u0026is_ui_opted_out=0\u0026show_ui_opt_out=0\u0026v=8","contextRegionUrl":null}}

Move the competitor harness out of the ASV suite (#64) - #65

Merged
rafatahmed merged 1 commit into
mainfrom
fix/asv-discovery
Oct 20, 2026
Merged

Move the competitor harness out of the ASV suite (#64)#65
rafatahmed merged 1 commit into
mainfrom
fix/asv-discovery

Conversation

@rafatahmed
@rafatahmed rafatahmed commented Oct 20, 2026
Copy link
Copy Markdown
Owner

Closes #64.

Problem

The Benchmark v1 workflow on main at 61dda09 (run 39510611814) measured no timings. ASV discovery aborted with ModuleNotFoundError: No module named 'tools' while importing benchmarks/competitors/chemistry_track.py, added in #63.

Root cause

ASV imports every module under benchmark_dir (benchmarks/), and makes only that package importable through its own meta-path importer (asv_runner._aux.update_sys_path); the repository root is not on sys.path. The development-only competitor harness sat inside the ASV suite and imported tools at module level. The local test run puts the root on sys.path, so it did not show.

Fix

  • Move the harness to tools/competitors/ (it is a development tool, not an ASV workload). Run commands become python -m tools.competitors.<track>.
  • tests/test_asv_discovery.py imports the suite the way ASV does: only the benchmark package importable, every submodule imported. It fails on 61dda09 with the CI error and passes here.
  • tools/publish_competitor_snapshot.py also checks tools/competitors for changes since the measured commit.
  • The published competitor snapshot is unchanged. The docs page names the new path and the path at the measured commit.

Verification

  • 367 tests pass (1 skipped, as on main).
  • asv check --python=same: no problems.
  • Benchmark v1 on this branch (run 39511069471): success. All 11 ASV workloads measured, including the equilibrium workload that timed out in the previously published run.

@rafatahmed
Move the competitor harness out of the ASV suite (#64)
2eb908f 
ASV imports every module under benchmark_dir with only the benchmarks
package importable (asv_runner's meta-path importer; the repository root is
not on sys.path). benchmarks/competitors/chemistry_track.py imported tools
at module level, so discovery aborted and Benchmark v1 run 39510611814
measured no timings. The development-only harness moves to
tools/competitors/. test_asv_discovery imports the suite the same way;
it fails on 61dda09 with the CI error and passes here, and asv check
finds no problems.
@read-the-docs-community
read-the-docs-community Bot commented Oct 20, 2026
Copy link
Copy Markdown

Documentation build overview

📚 Flahax | 🛠️ Build #35283272 | 📁 Comparing 2eb908f against latest (61dda09)

  🔍 Preview build  

2 files changed
± changelog.html
± evidence/competitor-comparison.html

@rafatahmed
rafatahmed merged commit 4dffdf8 into main Oct 20, 2026
7 checks passed
@rafatahmed
rafatahmed deleted the fix/asv-discovery branch October 20, 2026 22:54
This was referenced Oct 20, 2026
Merged
Closed
@rafatahmed rafatahmed self-assigned this Oct 23, 2026
rafatahmed added a commit that referenced this pull request Nov 7, 2026
@rafatahmed
Merge pull request #65 from rafatahmed/fix/asv-discovery
8bc4723 
Move the competitor harness out of the ASV suite (#64)
rafatahmed added a commit that referenced this pull request Nov 7, 2026
@rafatahmed
Publish Benchmark v1 snapshot 2026-10-20-linux-4dffdf8; warn only on …
3620d60 
…failed gates (#66)

Hosted run 39511463809 on main at 4dffdf8 (after #62 and #65) is the first
capture with every correctness gate passing: 7/7 controlled NNLS cases,
21/21 stored chemistry comparisons and 17/17 timing workloads.

The publisher wrote 'The numerical scorecard is not fully passing'
unconditionally. scorecard_notice() now gives that warning only when an NNLS
case or chemistry comparison fails, a timing workload is missing or the
workflow did not succeed, and a note otherwise; both branches are tested.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@rafatahmed rafatahmed

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Benchmark v1 ASV discovery fails: competitor harness inside the ASV benchmark directory

1 participant

@rafatahmed
0