Repository navigation
v5: parser hardening, benchmarks and performance foundations - #642
Merged
Merged
Conversation
Makes parsing repeatable and measurable, fixes several code paths that always
failed at runtime, and removes the worst performance costs found by measuring.
Stays on Symfony DomCrawler; the MalClient facade and models are unchanged.
Fixes
- Helper\Parser::removeChildNodes mutated the shared page DOM, so a second
getModel() on the same crawler failed ("The current node list is empty") in
14 parsers. It now works on a copy, shallow-cloning the node and deep-copying
only the children that survive.
- MalClient::getNewsList() read a non-existent $this->ghoutte property.
- PersonImageResource\Jpg and UserImageResource\Jpg::__toString() called a
method that does not exist.
- AnimeCardParser::getMembers() fell off the end (a TypeError that bypassed
catch (\Exception)); it now throws a RuntimeException so it surfaces as a
ParserException.
- HttpClientWrapper kept one BrowserKit history entry per request forever; it
no longer records history.
- Remove MangaReviewer and MangaReviewerParser: they extended classes deleted
earlier and nothing referenced them.
Performance
- Seasonal parse ~1200 -> ~370 ms: AnimeCardParser::isContinuing() re-read the
whole season section for every card (quadratic).
- Anime parse ~-26%: IndexesSidebarLabels looks up the 25 sidebar labels in one
XPath pass instead of 25 full-page scans (8 more in MangaParser).
- MalClient: 56 of 76 methods share one private parse() helper (1580 -> ~1200
lines); behaviour unchanged.
Tooling and tests
- scripts/benchmark.php (composer benchmark): fixture-based parse timings with
--json / --compare.
- scripts/drift_check.php --live: 14 canary pages parsed against the live site,
with null-rate reporting. Not part of the test suite.
- PHPStan level 6 with a baseline (composer phpstan, part of composer quality).
- A parse-twice test and a golden-file model snapshot test in 60 parser test
classes (suite: 495 -> 623 tests). The snapshots live in jikan-fixtures
(snapshots/) and must merge there first.
- PLAN.md records the findings, measurements and decisions.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
code-quality-pr, code-quality-push and quality referenced shivammathur/setup-php@v4, which has no such tag, so those jobs failed at setup before running anything. tests.yaml already uses @v2, the current major; use it everywhere. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
UserProfileParser resolves lastOnline against the current day ("Yesterday,
9:25 PM"), so a snapshot generated on one day failed the next (caught by the
pre-commit run the day after the snapshots were written). Mask dates for this
test like the other now-relative parsers; the snapshot in jikan-fixtures is
refreshed to match. Also regenerates the endpoint manifest from the sorted
traversal added in the previous commit.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The library is pure PHP over recorded HTTP fixtures with no OS-specific code, so the Windows job re-ran the same tests as Ubuntu at about twice the runner cost and nothing in the repo or in master's branch rules depends on it. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Keep the Windows job in the test matrix. This reverts ccff9dc. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Workflows - tests.yaml: add a lowest-dependencies job (composer update --prefer-lowest) and a non-blocking PHP 8.5 job next to Ubuntu and Windows on 8.4; run the suite with paratest; install only the extensions the library, PHPUnit and Composer use (no mongodb/gd/pdo/sqlite) with coverage disabled; drop the extension-cache steps that had nothing to cache. - quality.yaml: run the new `composer quality:static` (formatter, PHPStan, endpoint manifest) instead of `composer quality`, which ran the whole test suite a second time on Ubuntu; no fixtures clone needed; cache Composer. - Remove the code-quality-pr/push workflows and their lint scripts: the formatter, PHPStan and the tests already parse every file, and GrumPHP still runs parallel-lint on commit. - Cancel superseded pull request runs (never master), restrict the token to contents: read, and add job timeouts. - Add dependabot.yml for composer and github-actions, weekly. Config - phpunit.xml: PHPUnit 13 format with a <source> section, fail on deprecations and notices (so the PHP 8.5 job reports new ones), drop the stale phpVersion 8.3 gate; delete the unused old-format phpunit.xml.dist. - composer.json: add the quality:static script; declare symfony/dom-crawler (^7.4 || ^8.0, the range browser-kit already allowed), which the library uses directly but only received transitively. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Under Unreleased: - Changed: symfony/dom-crawler declared as a requirement; seasonal pages parse about 3x faster; a card without a member count now raises ParserException rather than an uncaught TypeError; MangaReviewer and MangaReviewerParser (unloadable) removed. - Fixed: getModel() repeatable on 14 parsers; MalClient::getNewsList() no longer fails before the request; the two Jpg::__toString() methods; the unbounded browser-history growth in HttpClientWrapper. Contributor tooling (benchmark, drift check, PHPStan, tests, CI) is left out, per the changelog's own rules. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
First CI run of the new matrix found three problems, all in test tooling: - Windows: the checkout converts snapshots to CRLF, so all 60 snapshot comparisons failed on line endings. ModelSnapshot now normalises the stored file to LF before comparing. - PHP 8.5: SplObjectStorage::contains/attach/detach are deprecated since 8.5 (12 deprecations, and the new failOnDeprecation made them fail the run). ModelSnapshot and drift_check.php use ArrayAccess instead. - Lowest dependencies: Composer pairs paratest 7.4.8 with PHPUnit 10.5, which paratest does not fully support. That job now runs plain PHPUnit; verified locally on PHPUnit 10.5.36 and Symfony 8.0.0 (623 tests pass). Also mask dates in four more tests whose values depend on the current time (MangaGenre, Magazine, Schedule, Friend): their snapshots changed overnight. The refreshed snapshots are in the jikan-fixtures PR for fix/mask-time-dependent-snapshots, which must merge first. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
irfan-dahir
marked this pull request as ready for review
October 13, 2026 17:14
- Helper\Parser::parseDateMDY/DMY read two-digit years 00-69 as 20xx, so search results and user lists reported 1961 titles as 2061 (and isAiring with them). A year more than 10 years ahead now goes back a century. - Friend::getFriendsSince() declared a non-null date while the parser returns null when MAL shows none, a TypeError on access. It now returns ?DateTimeImmutable. - Require PHP ^8.4.4: 8.4.0-8.4.3 corrupt some 4-byte characters in the HTML5 parser (php-src GH-17481), producing invalid UTF-8 that json_encode rejects. Reproduced on the manga reviews fixture with 8.4.1; clean on 8.4.23. All three were found by the new model invariants (next commit). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…asses - ModelInvariants runs inside every snapshot test: valid UTF-8, JSON- encodable, no NBSP, edge whitespace, empty strings or stray entities and tags, absolute URLs, positive ids, non-negative numbers, score range, sane years, consistent pagination, and every public getter (including __toString) callable. Known violations are kept per class in test/unit/__invariants__ and may only shrink (11 empty strings, 6 pagination mismatches, pending owner decisions). - Parse-twice and snapshot tests for the 28 classes that hold several parsers or a model built through MalClient; ModelSnapshot can keep named snapshots per class. Dates are masked where parsing depends on the clock until the planned clock lands. - The fixture callback no longer creates directories under vendor/ and can log what it serves (JIKAN_FIXTURE_LOG); scripts/fixtures_unused.php lists recordings no test loads; scripts/field_witness.php lists model fields no recorded page fills. - CI: a report-only job with pcov line coverage and both reports. - PLAN.md records the test audit and the phased plan. The new snapshots and the fixture prune are in jikan-me/jikan-fixtures (feat/more-model-snapshots), which must merge first. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 14, 2026
This was referenced Oct 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Makes parsing repeatable, checked and measurable. It fixes several code paths that always failed at runtime, removes the worst performance cost found by measuring, and rebuilds the CI pipeline and the test suite's safety net. Stays on Symfony DomCrawler; the
MalClientfacade is unchanged.PLAN.mdrecords the findings, measurements and decisions;CHANGELOG.md(Unreleased) has the consumer-visible changes.CI status: all green on
6416ffc:composer quality:static, Ubuntu and Windows on PHP 8.4, lowest dependencies, PHP 8.5, and the coverage report. First coverage numbers (pcov): 77.8% of lines, 67.1% of methods, 44.9% of classes. All 98 fixtures are loaded; 38 model fields are never filled by any recorded page.Breaking
^8.4.4. PHP 8.4.0–8.4.3's HTML5 parser (Dom\HTMLDocument, php-src GH-17481) corrupts some 4-byte characters (emoji, rare scripts) depending on where they fall in the page. A real MAL manga review came out as invalid UTF-8, whichjson_encoderejects. Reproduced on 8.4.1 and clean on 8.4.23. Consumers on 8.4.0–8.4.3 must update PHP.Fixes
getModel()failed when called twice.Helper\Parser::removeChildNodesmutated the shared page DOM, so a secondgetModel()on the same crawler failed ("The current node list is empty") in 14 parsers. It now works on a copy, shallow-cloning the node and deep-copying only the children that survive.MalClient::getNewsList()always failed: it read a non-existent$this->ghoutteproperty.Jpg::__toString()methods (person and user images) called a method that does not exist.AnimeCardParser::getMembers()fell off the end. That was aTypeErrorthat bypassedcatch (\Exception); it now surfaces as aParserException.HttpClientWrapperhistory leak: it kept one BrowserKit history entry per request for the lifetime of the client (about 2.5 KB each, unbounded). It no longer records history.isAiring. A year more than 10 years ahead now goes back a century.Friend::getFriendsSince()threw aTypeErrorwhen MAL shows no date. It now returns?DateTimeImmutable.symfony/dom-crawleris now declared (^7.4 || ^8.0, the rangebrowser-kitalready allowed); it was used directly but only received transitively.MangaReviewerandMangaReviewerParserextended classes deleted earlier (d8ec60b), and nothing referenced them.Performance
Parse plus model build only, from fixtures; median of 3 interleaved rounds × 30 iterations; this branch vs
master.AnimeCardParser::isContinuing()re-read the whole season section for every card, which is quadratic.IndexesSidebarLabelsreplaces 25 full-page label scans inAnimeParser(8 inMangaParser) with one pass, but the non-destructiveremoveChildNodescosts about the same back.DOMDocument, in PHP code. Handing it a libxml-parsed DOM is 2.3× faster but broke 193 tests, because the parsers' XPaths depend on the HTML5 tree; not pursued. Memoizing repeated card lookups changed nothing measurable; not kept.MalClient56 of its 76 methods repeated one fetch / parse / wrap block and now share a private
parse($request, Parser\X::class)helper (1,580 → about 1,200 lines). Behaviour is unchanged. The other 20 methods have special handling and were left alone.CI pipeline
shivammathur/setup-php@v4, which doesn't exist, so the quality jobs had never actually run. They are now pinned to@v2.--prefer-lowest, plain PHPUnit; verified on PHPUnit 10.5 and Symfony 8.0.0).composer quality:static(formatter, PHPStan, endpoint manifest). Tests are no longer run twice;composer qualitystill runs everything locally.parallel-lintworkflows. GrumPHP still lints on commit.master), the token is read-only, jobs have timeouts, anddependabot.ymlcovers Composer and GitHub Actions.generate_endpoints.phpread test files in filesystem order, soendpoints:checkdisagreed between machines. It now sorts.phpunit.xml.distis gone.Tests
The suite is at 652 tests.
JIKAN_UPDATE_SNAPSHOTS=1. Dates are masked in the classes where parsing depends on the current clock, until the planned clock lands.ModelInvariantsruns on every snapshot model:__toString, callable.test/unit/__invariants__/(11 empty-string fields, 6 pagination mismatches). A new violation fails, and so does an allowlisted one that disappears.vendor/and can log what it serves (JIKAN_FIXTURE_LOG).ModelSnapshotnormalizes CRLF (Windows checkouts) and avoidsSplObjectStoragemethods deprecated in PHP 8.5.Scripts:
composer benchmark,scripts/drift_check.php --live(14 live canary pages, GET only, 3.5 s apart, not in CI),scripts/fixtures_unused.php,scripts/field_witness.php.Findings recorded, not changed
ETagorLast-Modified(Cache-Control: no-cache).AnimeReviewParser::getAnimeScores()andReviewerParser::getAnimeScores()instantiate the deletedAnimeReviewScoresParser.scripts/field_witness.php), for exampleAnimeCard::typeandMangaCard::authors. Some may be bugs, some fields MAL removed; recording variant pages will tell.hasNextPage=truewithlastVisiblePage=1. They are allowlisted pending a decision.Not in this PR
Phase 2 of the test plan in
PLAN.md: an injectable clock (symfony/clock), a fixture format with HTTP status and recording time, a data-driven corpus over every endpoint, and a one-off recording of about 70 variant pages. Also not included: concurrency, a built-in throttle, a PSR-18 adapter, and the model overhaul (RFC track).🤖 Generated with Claude Code