Repository navigation
Count newly indexed URLs, not submitted results, in crawler stats - #609
Merged
Merged
Conversation
Since the batched results API, a crawler posts every local item not yet marked synced, on every round. Items the main index does not keep are never marked synced, so they are resubmitted indefinitely, and a URL is repeated once for each term page it is retrieved from. The server counted len(results.results), so the daily results count, leaderboard, user stats, crawler-version spread and device totals all over-counted. index_documents now reports which URLs it stored and which of those the index did not hold before (what index_new_documents already computed, which now delegates to it), and POST /crawler/results records the number of new URLs instead. Closes #608 No functional changes to the crawler. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DTbu4oERm8rGd9iRy29EiL
No functional changes to the crawler. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DTbu4oERm8rGd9iRy29EiL
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #608
What changed
index_documentsnow also returnsIndexedURLs: the URLs it stored, and which of those the index didn't hold before. This is the same calculationindex_new_documentsalready did, and that function now delegates to it.POST /crawler/resultsrecordslen(indexed.urls.new)instead oflen(results.results). That number feeds the daily results count, the user and leaderboard counts,UserStats, the crawler-version spread and the device totals.UserStatsrow.Why
Since the batched results API (#598), the crawler posts every local item that isn't
SYNCED_WITH_MAIN_INDEX, every round:The server counted every submitted result, so the same pages were counted over and over.
Things to be aware of
UserStatshistory already includes the inflated numbers. This PR doesn't backfill them.Verification
test/test_indexed_results_count.py:test/test_stats.pyfor a zero-result submission.make test: 1774 passed and 1 failed (a version-spread test that mocked indexing), which I then fixed. The affected test files pass afterwards (140 passed).ruff format --checkandruff checkare clean.tyreports the same 1211 diagnostics with and without this change in this environment, somake checkfails here either way.Generated by Claude Code