8000 Allow bulk domain submissions by blueheron786 · Pull Request #670 · mwmbl/mwmbl · GitHub
[go: up one dir, main page]

Skip to content

Allow bulk domain submissions - #670

Open
blueheron786 wants to merge 3 commits into
mainfrom
bulk-domain-submissions
Open

blueheron786 wants to merge 3 commits into
mainfrom
bulk-domain-submissions

Conversation

@blueheron786
Copy link
Copy Markdown
Collaborator

Summary

Allow users to submit multiple domains or URLs as newline-separated entries. Existing normalization is reused, with subdomains retained. Validation is all-or-nothing, and submissions are capped at 1000 entries.

Verification

  • make check
  • Isolated full suite: 2212 passed, 10 skipped, 4 deselected
  • Focused tests pass.

No functional changes to the crawler.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@codecov
codecov Bot commented Nov 16, 2026
Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@blueheron786 blueheron786 left a comment
Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent review, from reading the diff and the code it touches. I did not run anything, so the cost figures below are estimates, not measurements.

DoS: pasting 10k or 100k URLs

  • Both are rejected cheaply. The len(lines) > MAX_DOMAIN_SUBMISSIONS check runs before any per-line validation, so an oversize paste costs one splitlines() and one list comprehension.
  • The body is bounded by Django's default DATA_UPLOAD_MAX_MEMORY_SIZE of 2.5MB. I found no override in settings_common.py. Past that the request fails with a 400 before the view runs. The worst case that reaches the cap check is about 500k lines of a.co, which is ms-scale work and tens of MB of transient memory. I'd expect a proxy client_max_body_size to bite first, but I can't see the deployment config.
  • The weak points are the 1000 that are accepted and the unbounded line length. See the inline comments.

Efficiency: file upload instead?

No. The limit that matters is downstream cost, not transport:

  • 1000 lines is about 20KB of typical input, and at most about 300KB at the model's max_length=300.
  • A file upload would only matter for tens of thousands of entries. At that size the constraint is moderator capacity and crawler cost, which no upload format fixes.
  • If we ever do add a file route, cap it at ~256KB (about the textarea worst case) and reuse this form's clean_name. Don't raise the 1000 to justify it.

Main concerns, in priority order

  1. Each created row schedules a crawl (see the inline comment on the loop). That is 1000 queued crawls per POST with no per-user limit, which is much stronger than the old one-per-request path.
  2. Duplicates are not collapsed within a submission, and the new test asserts that they aren't.
  3. The HTML view has no email-verified check. POST /domain-submissions/ in platform/api.py calls check_email_verified, so this view is now the weaker door.
  4. A line has no length bound and is echoed back in its error message.

Trust and safety: a bulk path makes it easy to push many link-farm or spam domains into the crawl queue and the moderation queue in one request. Items 1-3 are what limit that.

Comment thread mwmbl/views.py Outdated
from mwmbl.utils import add_term_infos, float_or_none, normalize_domain, parse_url, validate_domain

MAX_CURATED_SCORE = 1_111_111.0
MAX_DOMAIN_SUBMISSIONS = 1000
Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is 1000 the right cap? It is cheap for the transport (see the review summary), but each entry costs a background crawl plus a human moderation decision. Something nearer 100-250 per submission would still cover the legitimate "paste my site list" case. Combined with a per-user rate limit (comment on the loop below), that bounds abuse better than the per-request cap alone.

Comment thread mwmbl/views.py
lines = [
(number, line) for number, line in enumerate(self.cleaned_data["name"].splitlines(), 1) if line.strip()
]
if len(lines) > MAX_DOMAIN_SUBMISSIONS:
Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good that the cap is checked before any per-line validation, so 10k or 100k lines are rejected without running the regexes or building 1000+ ModelForms.

Two gaps:

  • The cap counts lines, not bytes. A single line can be up to the 2.5MB body limit, and validate_domain echoes the whole line back in Invalid domain: %(domain)s. A few MB of garbage on one line (or spread over 1000) comes back as an error page. Truncate the echoed value, or reject lines longer than the model's 300-char name limit up front.
  • The strip=False CharField has no max_length either. Setting max_length on the field (roughly MAX_DOMAIN_SUBMISSIONS * 301) gives a clear error before the loop.

Comment thread mwmbl/views.py 8000 Outdated
if len(lines) > MAX_DOMAIN_SUBMISSIONS:
raise ValidationError(f"Enter no more than {MAX_DOMAIN_SUBMISSIONS} domains or URLs.")
for line_number, line in lines:
form = DomainSubmissionForm({"name": line})
Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This builds a full ModelForm per line (instance construction, full_clean, constraint and unique validation) just to call normalize_domain and validate_domain. At 1000 lines that is more work than needed. Call the two functions directly and keep the 300-char length check explicit. That also removes the form.errors["name"] indexing. Low priority next to the loop below, but cheap to fix.

Comment thread mwmbl/views.py Outdated
domain_submission.submitted_by = request.user
domain_submission.submitted_on = timezone.now()
domain_submission.save()
with transaction.atomic():
Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Amplification, the main concern. DomainSubmission has a post_save receiver, enrich_new_submission in mwmbl/signals.py, that calls enrich_domain_submission(instance.name) for every created row. That is a django-background-task that does three page fetches plus model scoring. .create() in a loop therefore does the following inside one request:

  • ~1000 INSERTs into DomainSubmission
  • ~1000 INSERTs of background Task rows, all in one atomic() block, so a worker is held for the duration
  • 1000 queued crawls

The task's "evidence is still fresh" skip only helps once an earlier crawl has finished, so 1000 copies of the same domain queued together can all crawl. I'm assuming background_task doesn't dedupe identical params by default. Worth confirming.

There is also no per-user limit. One account looping this POST can queue unbounded crawls and flood the moderation queue, where the old form needed one request per domain. Suggestions:

  1. Dedupe names within the submission (order-preserving) before saving.
  2. Use bulk_create plus one explicit enrich_domain_submission(name) per distinct name. bulk_create skips post_save, so the APPROVED receiver is unaffected because rows start PENDING.
  3. Add a per-user limit, e.g. max N pending submissions or N domains per day, so the 1000 cap isn't the only brake.
  4. Compute timezone.now() once outside the loop so one batch shares a timestamp.

Trust and safety: this is how a bad actor gets a large batch of spam or link-farm domains crawled and in front of moderators at once.

Comment thread mwmbl/views.py
@@ -243,15 +275,14 @@ class Meta:
@login_required
def submit_domain(request):
Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

POST /domain-submissions/ calls check_email_verified(request), but this @login_required view doesn't. With bulk submission, an unverified throwaway account can now submit 1000 domains per request. That gap pre-dates the PR, but the PR makes it matter. Consider applying the same verified-email requirement here, or at least for the bulk path.

Comment thread test/test_domain_submissions.py Outdated
"y.com",
"z.com",
"www.example.co.uk",
"x.com",
Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test codifies that a repeated x.com creates two rows. If you dedupe within a submission (see the comment on the loop), change this expectation. Also worth adding, for the behaviour this PR introduces:

  • a test that a submission at exactly 1000 entries succeeds, to pin the boundary (only 1001 is tested now)
  • a test for a very long single line, once a length bound exists

Each created DomainSubmission schedules a crawl through post_save, so a
1000-line paste queued 1000 crawls, repeats included. Dedupe the names
within a submission, insert them with bulk_create, and schedule one crawl
per distinct domain.

Also bound each line to the length of DomainSubmission.name, without
echoing an over-long line back in the error, bound the whole field, and
validate lines directly rather than through a ModelForm per line.

No functional changes to the crawler.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@blueheron786
Copy link
Copy Markdown
Collaborator Author

Pushed 2f80482 addressing the review.

Fixed

  • Names are deduped within a submission (order kept; X.com and x.com collapse because they are normalised first).
  • Rows are inserted with bulk_create and one enrich_domain_submission is scheduled per distinct domain, in the same transaction. bulk_create sends no post_save, so the APPROVED receiver is unaffected, and the enrich_new_submission docstring now says where bulk crawls are scheduled.
  • Lines are capped at 300 chars (the name column). The error says Line N: longer than 300 characters. and does not echo the line. The field also has a max_length of 1000 × 302.
  • Lines are validated with validate_domain and normalize_domain directly, instead of one ModelForm per line.
  • One submitted_on per batch.
  • Tests: crawl scheduled once per distinct domain, exactly 1000 accepted, over-long line rejected without echo. The existing bulk test now expects the repeat to collapse.

Not changed, needs a decision

  • Per-user limit (pending submissions or domains per day). The 1000 cap is still the only brake on one account looping the POST.
  • Whether to lower the 1000 cap (I suggested 100–250).
  • The HTML view still has no verified-email check, unlike POST /domain-submissions/. Adding it changes behaviour for existing single submissions too.

Verified: test_domain_submissions.py plus the moderation, blocked-URL and signal tests pass (124 in total), ruff format --check and ruff check are clean, and ty adds only the .objects warning that the existing lines already produce. I didn't run the full suite.

Lower the per-submission cap to 250, cap each user at 500 domains per day,
and require a verified email address, as the API endpoint does.

No functional changes to the crawler.
@blueheron786
Copy link
Copy Markdown
Collaborator Author

Pushed c81b95c for the three open items from the review.

  • Cap lowered from 1000 to 250 domains per submission.
  • Added a per-user limit of 500 domains per rolling 24 hours, checked against the user's existing DomainSubmission rows.
  • The HTML view now requires a verified email address, matching POST /domain-submissions/. Unverified users get a form error.

Tests cover the 250 boundary, the unverified user, the daily limit, and t 8000 he limit ignoring submissions older than a day. test_domain_submissions.py and test_views_curation.py pass, and ruff is clean. I didn't run the full suite.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

0