Bite asks one question: if this change disappeared, would a test notice? So we tried to make it answer wrong.
Each trick below is a tiny made-up repo whose newest commit is the trick. We ran bite --free on it and wrote down
what Bite said next to the honest answer.
Before: 18 of 28 tricks fooled Bite (2 more were plain controls, and it got those right). After this round: 2 still do, and one of them (the flaky test) gets through half as often. Each fixed one has a test in tests/test_tricks.py that failed before its fix.
"Fooled" means Bite gave the wrong answer: it said a test noticed when none really did, it skipped a real fix, or it flagged something that isn't a fix claim.
Can you sneak a "fix" past Bite? Make a tiny repo whose newest commit says it fixed something, and get Bite to answer wrong:
- Fool it one way: no test really checks the fix, but Bite says ✅ a test noticed.
- Or the other way: a test really does check the fix, but Bite says
⚠️ no test noticed.
Run bite --free in your repo (no key, free). If Bite gets it wrong, open an issue with the
I fooled Bite form. The two tricks under
"Still fools Bite" below are already known, so they don't count; a new twist on one does.
Every trick that works goes in the tables below with your name, gets a test in tests/test_tricks.py, and gets fixed if it can be. Tricks Bite catches are welcome too: they go under "Not fooled".
| Trick | Honest answer | Bite before | Bite now |
|---|---|---|---|
One commit fixes total() (no test) and tweaks the tested label() |
the fix is unguarded | ✅ noticed by test_label |
total() |
The fix adds a helper cents(); the new test only checks cents exists |
the fix is unguarded | ✅ noticed | total() |
Same, but the test calls cents(1.2) and checks nothing |
the fix is unguarded | ✅ tests crash without it | total() |
Move shop.py to store.py and slip the fix in |
the fix is unguarded | ✅ tests crash without it | |
A fix whose title starts chore: |
checked | ➖ skipped as a tidy-up | |
| "Refactor total and fix rounding bug" | checked | ➖ skipped as a tidy-up | |
| "Make total faster and correct its rounding" | checked | ➖ skipped as a speed-up | |
| "Fix rounding (reported by a Windows user)", no Windows-only code | checked | ➖ skipped: built for Windows | |
| "Fix rounding, first seen on Python 3.8", no version check in the code | checked | ➖ skipped: built for Python 3.8 | |
The change sits after a # inside a triple-quoted string (a colour #fff) |
checked | not checked (read as a comment) | |
The change sits after // inside a JavaScript template string (a URL) |
checked | not checked (read as a comment) | |
The fix is in a docstring the code reads (__doc__) |
checked | not checked (docstring only) | |
The fix is in a type hint the code reads (get_type_hints) |
checked | not checked (hint only) | |
| Every test is marked skip | no tests ran | ||
Revert "Fix total rounding" |
not a fix claim | read as a fix claim | not a claim |
| The fix's test file is deleted in a later commit | the fix is unguarded today | ✅ noticed (by a test that's gone) | |
| An unrelated test that fails half the time | the fix is unguarded | ✅ noticed by the flaky test, 1 run in 2 | 1 run in 4 (see below) |
How the big one works: when the whole change is noticed, Bite now also takes out each piece of it alone (one piece
per function it touched, up to 6). A piece no test notices on its own is named. If no piece is noticed alone, the
pieces only count together and Bite says nothing new. This costs one more test run per piece, only for commits a
test noticed. When test files were deleted after a noticed commit, Bite also takes the change out of today's
code and runs today's tests (two more runs, only then). A piece that only changes a docstring or type hint isn't named on its own.
To keep runs quick, the piece check gets 6 test runs for a whole Bite run (--part-runs N changes that), and
bite --quick skips it, the flaky-test rerun and the deleted-test check.
To check this doesn't add false alarms, we ran the new and the old Bite on Bite's own last 10 code commits. With Bite's full test suite, every piece of every commit was noticed, the same 10 of 10 as before.
| Trick | Why it still works |
|---|---|
| A flaky test, failing by luck twice in a row | Bite now runs the tests a second time when one fails without the change, and a pass the second time means it didn't really notice. A test that fails half the time still gets through about 1 time in 4. |
A test that reads the code as text (assert "round(" in open("shop.py").read()) |
The test really does fail without the change. Telling "checks the text" from "checks what it does" isn't something Bite can see from a test run. The paid step, which reads the claim, is where this gets caught. |
A test that calls pytest.skip(), a test that swallows its own failure with try/except, a test that compares the
code with itself, a test file pytest never collects, a test that hangs or exits the process when the fix is gone,
an empty commit, a whitespace-only commit, a fix in a language the tests don't run, and a fix split over two
commits (code first, test next: "a later test notices").