A high-performance, concurrent ETL pipeline designed to ingest LaTeX source archives of millions of arXiv papers directly from AWS S3, extract and parse them using a parallel Rust core, and compile them into structured Parquet dataset partitions uploaded to Hugging Face.
This repository is configured specifically for AWS S3 requester-pays download synchronization, eliminating any dependencies on downloading source archives from Hugging Face Hub.
graph TD
subgraph Initialization
Start([Start Orchestrator]) --> CheckKaggle[Check & Update arXiv Metadata Snapshot from Kaggle]
CheckKaggle --> SyncHF[Sync Parquet Manifest from Hugging Face <br/> scripts/sync_manifest.py]
SyncHF --> SyncS3[Sync Source Manifest from AWS S3 <br/> scripts/sync_s3_manifest.py]
end
subgraph ETL Loop
SyncS3 --> Plan[Plan Batches & Filter Processed Archives]
Plan --> Download[Parallel Download .tar Archives from S3 <br/> scripts/download.py]
Download --> Extract[Parallel Native Parsing & Parquet Compilation <br/> arxiv_etl_rust/src/lib.rs]
Extract --> Upload[Upload Parquet Partitions & Update Manifest on HF <br/> scripts/upload.py]
Upload --> Clean[Clean Downloaded Archives & Parquet Output]
Clean --> LoopNext{More Batches?}
LoopNext -- Yes --> Download
end
LoopNext -- No --> End([Success])
The pipeline is split into a high-speed Rust-based parsing engine and a Python orchestrator managing network transfer operations:
- Rust Core / Orchestrator (main.rs): Computes chronological batches of tar files, runs parallel extraction and formatting of papers via Rayon, writes optimized Parquet files with ZSTD compression, and manages immediate disk cleanups.
- S3 Parallel Downloader (download.py): Downloads source
.tararchives in parallel directly from the officials3://arxiv/src/requester-pays bucket. - AWS S3 Manifest Syncer (sync_s3_manifest.py): Fetches the latest global
arXiv_src_manifest.xmlfrom AWS S3. - Hugging Face Syncer & Uploader (sync_manifest.py, upload.py): Tracks already processed files by downloading the tracking manifest
arxiv_parquet_manifest.xmlfrom the target HF dataset, and uploads newly compiled Parquet files to Hugging Face.
- Rust Toolchain:
stable(for building the native parsing binaries) - Python:
>=3.12 - uv: Python package installer and workflow runner
Initialize the virtual environment and sync Python dependencies:
uv syncCompile and register the Rust native library in the virtual environment: (optional)
cd arxiv_etl_rust
uv run maturin develop --release
cd ..Create a .env file in the root workspace directory:
# Hugging Face write token for uploading target datasets
HF_TOKEN=your_huggingface_token
# AWS Credentials for requester-pays S3 access
# If the pipeline is executed inside of AWS, these can be automatically loaded from the IAM role.
AWS_ACCESS_KEY_ID=your_access_key
AWS_SECRET_ACCESS_KEY=your_secret_key
AWS_DEFAULT_REGION=us-east-1For running the sync pipeline at maximum throughput, the recommended AWS cloud setup is:
- Instance Type:
c5.2xlarge(8 vCPUs, 16 GiB RAM) - Storage Volume:
250 GBgp3 EBS volume configured with:- Throughput:
1000 MB/s - IOPS:
4000 IOPS
- Throughput:
To run the full S3 sync pipeline:
cargo run --release --manifest-path arxiv_etl_rust/Cargo.toml -- --repo "your-hf-username/your-dataset-repo"To ignore the resume manifest and re-process all parts from scratch:
cargo run --release --manifest-path arxiv_etl_rust/Cargo.toml -- --no-resume --repo "your-hf-username/your-dataset-repo"To specify the maximum combined size (in GB) of S3 archives to process in a single batch:
cargo run --release --manifest-path arxiv_etl_rust/Cargo.toml -- --max-download-batch-gb 50.0 --repo "your-hf-username/your-dataset-repo"To reconstruct and repair a specific Parquet partition using the automated repair utility:
cargo run --release --manifest-path arxiv_etl_rust/Cargo.toml --bin repair -- \
--part 11 \
--target-repo "your-hf-username/your-dataset-repo" \
--metadata arxiv-metadata-oai-snapshot.json \
--download-dir downloadsTo sample a specific number of random rows across processed Parquet files:
cargo run --release --manifest-path arxiv_etl_rust/Cargo.toml --bin sample -- \
--data-dir target_parquet_dir \
--total-rows 50000 \
--output sample.parquetTo check if there are new source archives on S3 that have not yet been processed:
uv run python scripts/check_updates.py --repo "your-hf-username/your-dataset-repo" --syncThis utility:
- Syncs the latest tracking manifest from Hugging Face.
- Syncs the latest source manifest from AWS S3.
- Compares both manifests and prints the count of new unprocessed archives.
To convert a gitingest-formatted text file (containing parsed LaTeX code) back into a structured .tar (or .tar.gz) archive:
uv run python scripts/reconstruct_tar.py --input sample.txt --output sample.tarThis utility:
- Parses the GitIngest file structure separators.
- Restores the original internal directories and packages the files back into a
.taror.tar.gzarchive.
When running the main pipeline orchestrator, the following options can be passed (and they carry these defaults):
| Option | Default Value | Description |
|---|---|---|
--src-manifest <path> |
arxiv_src_manifest.xml |
Path to the source archive XML manifest |
--metadata <path> |
arxiv-metadata-oai-snapshot.json |
Path to the metadata snapshot JSON (automatically synced/updated via Kaggle Hub) |
--download-dir <path> |
downloads |
Local temporary directory for download cache |
--repo <name> |
scholarweave/arxiv-test |
Target Hugging Face Hub dataset repository for uploads |
--no-resume |
false (flag) |
Disable resume capability (start fresh by ignoring/deleting manifest) |
--max-download-batch-gb |
200.0 |
Maximum combined size of archives to download in a single batch |
--max-workers |
8 |
Maximum concurrent S3 download worker threads |
--s3 |
true (flag) |
Download source manifest and archives from S3 (always active) |
- Zero-Copy Memory-Mapped Indexing: Memory-maps the ~5.3 GB arXiv metadata JSON file for instant O(1) random-access parsing offsets.
- Rayon Parallel Extraction: Distributes CPU-intensive LaTeX parsing and cleaning across all available cores.
- Immediate Disk Reclamation: Downloads archives in parallel batches, extracts and deletes them in loop segments to limit disk space overhead.
- LargeStringBuilder Support: Uses 64-bit offsets in Arrow string builders to completely prevent indexing overflows.
- Kaggle arXiv Dataset: Cornell-University/arxiv
- arXiv S3 Bulk Data Help: arXiv Bulk Data on Amazon S3
- Hugging Face Dataset: scholarweave/arxiv-latex
- Official arXiv API: arXiv API Documentation