Repository navigation
Allow bulk domain submissions - #670
blueheron786 wants to merge 3 commits into
Conversation
No functional changes to the crawler. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
There was a problem hiding this comment.
Independent review, from reading the diff and the code it touches. I did not run anything, so the cost figures below are estimates, not measurements.
DoS: pasting 10k or 100k URLs
- Both are rejected cheaply. The
len(lines) > MAX_DOMAIN_SUBMISSIONScheck runs before any per-line validation, so an oversize paste costs onesplitlines()and one list comprehension. - The body is bounded by Django's default
DATA_UPLOAD_MAX_MEMORY_SIZEof 2.5MB. I found no override insettings_common.py. Past that the request fails with a 400 before the view runs. The worst case that reaches the cap check is about 500k lines ofa.co, which is ms-scale work and tens of MB of transient memory. I'd expect a proxyclient_max_body_sizeto bite first, but I can't see the deployment config. - The weak points are the 1000 that are accepted and the unbounded line length. See the inline comments.
Efficiency: file upload instead?
No. The limit that matters is downstream cost, not transport:
- 1000 lines is about 20KB of typical input, and at most about 300KB at the model's
max_length=300. - A file upload would only matter for tens of thousands of entries. At that size the constraint is moderator capacity and crawler cost, which no upload format fixes.
- If we ever do add a file route, cap it at ~256KB (about the textarea worst case) and reuse this form's
clean_name. Don't raise the 1000 to justify it.
Main concerns, in priority order
- Each created row schedules a crawl (see the inline comment on the loop). That is 1000 queued crawls per POST with no per-user limit, which is much stronger than the old one-per-request path.
- Duplicates are not collapsed within a submission, and the new test asserts that they aren't.
- The HTML view has no email-verified check.
POST /domain-submissions/inplatform/api.pycallscheck_email_verified, so this view is now the weaker door. - A line has no length bound and is echoed back in its error message.
Trust and safety: a bulk path makes it easy to push many link-farm or spam domains into the crawl queue and the moderation queue in one request. Items 1-3 are what limit that.
| from mwmbl.utils import add_term_infos, float_or_none, normalize_domain, parse_url, validate_domain | ||
|
|
||
| MAX_CURATED_SCORE = 1_111_111.0 | ||
| MAX_DOMAIN_SUBMISSIONS = 1000 |
There was a problem hiding this comment.
Is 1000 the right cap? It is cheap for the transport (see the review summary), but each entry costs a background crawl plus a human moderation decision. Something nearer 100-250 per submission would still cover the legitimate "paste my site list" case. Combined with a per-user rate limit (comment on the loop below), that bounds abuse better than the per-request cap alone.
| lines = [ | ||
| (number, line) for number, line in enumerate(self.cleaned_data["name"].splitlines(), 1) if line.strip() | ||
| ] | ||
| if len(lines) > MAX_DOMAIN_SUBMISSIONS: |
There was a problem hiding this comment.
Good that the cap is checked before any per-line validation, so 10k or 100k lines are rejected without running the regexes or building 1000+ ModelForms.
Two gaps:
- The cap counts lines, not bytes. A single line can be up to the 2.5MB body limit, and
validate_domainechoes the whole line back inInvalid domain: %(domain)s. A few MB of garbage on one line (or spread over 1000) comes back as an error page. Truncate the echoed value, or reject lines longer than the model's 300-charnamelimit up front. - The
strip=FalseCharFieldhas nomax_lengtheither. Settingmax_lengthon the field (roughlyMAX_DOMAIN_SUBMISSIONS * 301) gives a clear error before the loop.
| if len(lines) > MAX_DOMAIN_SUBMISSIONS: | ||
| raise ValidationError(f"Enter no more than {MAX_DOMAIN_SUBMISSIONS} domains or URLs.") | ||
| for line_number, line in lines: | ||
| form = DomainSubmissionForm({"name": line}) |
There was a problem hiding this comment.
This builds a full ModelForm per line (instance construction, full_clean, constraint and unique validation) just to call normalize_domain and validate_domain. At 1000 lines that is more work than needed. Call the two functions directly and keep the 300-char length check explicit. That also removes the form.errors["name"] indexing. Low priority next to the loop below, but cheap to fix.
| domain_submission.submitted_by = request.user | ||
| domain_submission.submitted_on = timezone.now() | ||
| domain_submission.save() | ||
| with transaction.atomic(): |
There was a problem hiding this comment.
Amplification, the main concern. DomainSubmission has a post_save receiver, enrich_new_submission in mwmbl/signals.py, that calls enrich_domain_submission(instance.name) for every created row. That is a django-background-task that does three page fetches plus model scoring. .create() in a loop therefore does the following inside one request:
- ~1000 INSERTs into
DomainSubmission - ~1000 INSERTs of background Task rows, all in one
atomic()block, so a worker is held for the duration - 1000 queued crawls
The task's "evidence is still fresh" skip only helps once an earlier crawl has finished, so 1000 copies of the same domain queued together can all crawl. I'm assuming background_task doesn't dedupe identical params by default. Worth confirming.
There is also no per-user limit. One account looping this POST can queue unbounded crawls and flood the moderation queue, where the old form needed one request per domain. Suggestions:
- Dedupe names within the submission (order-preserving) before saving.
- Use
bulk_createplus one explicitenrich_domain_submission(name)per distinct name.bulk_createskipspost_save, so theAPPROVEDreceiver is unaffected because rows startPENDING. - Add a per-user limit, e.g. max N pending submissions or N domains per day, so the 1000 cap isn't the only brake.
- Compute
timezone.now()once outside the loop so one batch shares a timestamp.
Trust and safety: this is how a bad actor gets a large batch of spam or link-farm domains crawled and in front of moderators at once.
| @@ -243,15 +275,14 @@ class Meta: | |||
| @login_required | |||
| def submit_domain(request): | |||
There was a problem hiding this comment.
POST /domain-submissions/ calls check_email_verified(request), but this @login_required view doesn't. With bulk submission, an unverified throwaway account can now submit 1000 domains per request. That gap pre-dates the PR, but the PR makes it matter. Consider applying the same verified-email requirement here, or at least for the bulk path.
| "y.com", | ||
| "z.com", | ||
| "www.example.co.uk", | ||
| "x.com", |
There was a problem hiding this comment.
This test codifies that a repeated x.com creates two rows. If you dedupe within a submission (see the comment on the loop), change this expectation. Also worth adding, for the behaviour this PR introduces:
- a test that a submission at exactly 1000 entries succeeds, to pin the boundary (only 1001 is tested now)
- a test for a very long single line, once a length bound exists
Each created DomainSubmission schedules a crawl through post_save, so a 1000-line paste queued 1000 crawls, repeats included. Dedupe the names within a submission, insert them with bulk_create, and schedule one crawl per distinct domain. Also bound each line to the length of DomainSubmission.name, without echoing an over-long line back in the error, bound the whole field, and validate lines directly rather than through a ModelForm per line. No functional changes to the crawler. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
Pushed 2f80482 addressing the review. Fixed
Not changed, needs a decision
Verified: |
Lower the per-submission cap to 250, cap each user at 500 domains per day, and require a verified email address, as the API endpoint does. No functional changes to the crawler.
|
Pushed c81b95c for the three open items from the review.
Tests cover the 250 boundary, the unverified user, the daily limit, and t
8000
he limit ignoring submissions older than a day. |
Summary
Allow users to submit multiple domains or URLs as newline-separated entries. Existing normalization is reused, with subdomains retained. Validation is all-or-nothing, and submissions are capped at 1000 entries.
Verification
make check2212 passed, 10 skipped, 4 deselected