8000 Common Crawl Foundation · GitHub
[go: up one dir, main page]
Skip to content
@commoncrawl

Common Crawl Foundation

Common Crawl provides an archive of webpages going back to 2007.

Pinned Loading

  1. cc-pyspark cc-pyspark Public

    Process Common Crawl data with Python and Spark

    Python 459 98

  2. cc-crawl-statistics cc-crawl-statistics Public

    Statistics of Common Crawl monthly archives mined from URL index files

    HTML 233 19

  3. cc-index-table cc-index-table Public

    Index Common Crawl archives in tabular format

    Java 132 17

  4. cc-warc-examples cc-warc-examples Public

    Forked from Smerity/cc-warc-examples

    CommonCrawl WARC/WET/WAT examples and processing code for Java + Hadoop

    Java 38 17

  5. cc-citations cc-citations Public

    Scientific articles using or citing Common Crawl data

    Jupyter Notebook 31 4

  6. cc-notebooks cc-notebooks Public

    Various Jupyter notebooks about Common Crawl data

    Jupyter Notebook 69 11

Repositories

Showing 10 of 91 repositories
  • web-languages Public

    Crowd-sourced lists of urls to help Common Crawl crawl under-resourced languages. See https://github.com/commoncrawl/web-languages-code/ for the code

    commoncrawl/web-languages's past year of commit activity
    73 103 6 1 Updated Nov 7, 2026
  • news-crawl Public

    News crawling with StormCrawler - stores content as WARC

    commoncrawl/news-crawl's past year of commit activity
    Java 383 Apache-2.0 43 12 6 Updated Nov 7, 2026
  • cc-index-server Public Forked from ikreymer/cc-index-server

    Common Crawl Index Server

    commoncrawl/cc-index-server's past year of commit activity
    HTML 71 28 7 0 Updated Nov 7, 2026
  • cc-downloader Public

    A polite and user-friendly downloader for Common Crawl data

    commoncrawl/cc-downloader's past year of commit activity
    Rust 99 Apache-2.0 9 11 1 Updated Nov 7, 2026
  • langid-eval Public

    Tuneable fuzzy matching for language identification evaluation

    commoncrawl/langid-eval's past year of commit activity
    Python 0 MIT 0 0 0 Updated Nov 6, 2026
  • cc-index-table Public

    Index Common Crawl archives in tabular format

    commoncrawl/cc-index-table's past year of commit activity
    Java 132 Apache-2.0 17 5 7 Updated Nov 5, 2026
  • warc2zip Public

    Convert WARC files into zip archives that are good for you.

    commoncrawl/warc2zip's past year of commit activity
    Python 7 Apache-2.0 1 3 4 Updated Nov 4, 2026
  • presentations Public

    A collection of public presentations from the Common Crawl Foundation

    commoncrawl/presentations's past year of commit activity
    13 1 0 0 Updated Nov 4, 2026
  • lang 84A2 uage-detection-cld2 Public

    Natural language detection, Java bindings for CLD2

    commoncrawl/language-detection-cld2's past year of commit activity
    Java 20 Apache-2.0 2 3 2 Updated Nov 3, 2026
  • ia-hadoop-tools Public Forked from Aloisius/ia-hadoop-tools

    Web archiving tools on Hadoop

    commoncrawl/ia-hadoop-tools's past year of commit activity
    Java 4 29 2 1 Updated Nov 3, 2026
View all repositories
0