8000 feat: comprehensive real-world solver benchmark by dominicusin · Pull Request #95 · dominicusin/package-maximizer · GitHub
[go: up one dir, main page]
Skip to content

feat: comprehensive real-world solver benchmark - #95

Merged
dominicusin merged 1 commit into
mainfrom
feat/0.22.0-benchmark
Sep 8, 2026
Merged

feat: comprehensive real-world solver benchmark#95
dominicusin merged 1 commit into
mainfrom
feat/0.22.0-benchmark

Conversation

@dominicusin
@dominicusin dominicusin commented Sep 8, 2026 •
edited by coderabbitai Bot
Loading
Copy link
Copy Markdown
Owner

What's New

  • benchmark_runner.py: Full solver benchmark on real nixpkgs data
    • Tests all 7 solvers: greedy, enhanced_greedy, z3, pulp, ortools, maxsat, minisat
    • Measures speed, accuracy, correctness, conflicts, unmet deps
    • Real dependency graph from nix-store --references
    • JSON output saved to benchmark_results.json
  • All lint clean, 849 tests, 85.09% coverage

Summary by CodeRabbit

  • New Features

    • Added a benchmark tool for evaluating package selection across multiple solvers using real package and dependency data.
    • Added command-line options to run all solvers or selected solvers, limit package counts, enable verbose output, and save results.
    • Added reporting for runtime, selection rates, conflicts, unmet dependencies, correctness, and accuracy.

  • Documentation

    • Added benchmark results covering seven solvers, including performance and correctness summaries.

@dominicusin
feat: comprehensive real-world solver benchmark
44bb2d6 
- benchmark_runner.py: Full solver benchmark on real nixpkgs data
  - Tests all 7 solvers: greedy, enhanced_greedy, z3, pulp, ortools, maxsat, minisat
  - Measures speed, accuracy, correctness, conflicts, unmet deps
  - Real dependency graph from nix-store --references
  - JSON output saved to benchmark_results.json
- All lint clean (black, isort, mypy, flake8)
- 849 tests, 85.09% coverage
@qodo-code-review
qodo-code-review Bot commented Sep 8, 2026
Copy link
Copy Markdown

Qodo reviews are paused for this user.

Troubleshooting steps vary by plan Learn more →

On a Teams plan?
Reviews resume once this user has a paid seat and their Git account is linked in Qodo.
Link Git account →

Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center?
These require an Enterprise plan - Contact us
Contact us →

@coderabbitai
coderabbitai Bot commented Sep 8, 2026 •
edited
Loading
Copy link
Copy Markdown

Review Change StackReview Change Stack

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: d4457b3c-f6b2-45a5-b47c-2665b0d114ae

📥 Commits

Reviewing files that changed from the base of the PR and between 1aecbeb and 44bb2d6.

📒 Files selected for processing (2)
  • benchmark_results.json
  • benchmark_runner.py

Walkthrough

Adds a CLI benchmark runner that builds real package data from nixpkgs and the Nix store, executes registered solvers, reports metrics, and writes results to benchmark_results.json.

Changes

Benchmark workflow

La 8000 yer / File(s) Summary
Package and dependency discovery
benchmark_runner.py		 Parses nixpkgs package declarations and Nix store entries. Resolves dependencies and builds a bounded package set.
Solver execution and metrics
benchmark_runner.py		 Runs selected solvers through PackageMaximizer. Records timing, selection, conflicts, unmet dependencies, and correctness metrics.
Reporting and CLI output
benchmark_runner.py, benchmark_results.json		 Prints benchmark summaries, supports command-line options, sets the process exit code, and stores per-solver results in JSON.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Operator
  participant BenchmarkRunner
  participant NixEnvironment
  participant PackageMaximizer
  participant Solvers
  participant ResultsFile
  Operator->>BenchmarkRunner: invoke benchmark CLI
  BenchmarkRunner->>NixEnvironment: read nixpkgs and Nix store packages
  NixEnvironment-->>BenchmarkRunner: return package data and dependencies
  BenchmarkRunner->>PackageMaximizer: run benchmark with package set
  PackageMaximizer->>Solvers: execute registered solvers
  Solvers-->>PackageMaximizer: return selections
  PackageMaximizer-->>BenchmarkRunner: return solver metrics
  BenchmarkRunner->>ResultsFile: write benchmark_results.json
  BenchmarkRunner-->>Operator: print report and exit status
Loading
✨ Finishing Touches 💡 1
🛠️ Fix failing CI checks 💡
  • Create stacked PR
  • Commit on current branch
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/0.22.0-benchmark

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

A rabbit checked each package line,
Then timed the solvers, row by row.
Dependencies formed a tidy chain,
Results now rest where JSONs grow.
Seven runners raced in light,
The benchmark closed with status right.

Comment @coderabbitai help to get the list of available commands.

@dominicusin
dominicusin merged commit f95e6ab into main Sep 8, 2026
15 of 17 checks passed
@dominicusin
dominicusin deleted the feat/0.22.0-benchmark branch September 8, 2026 06:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@dominicusin
0