E2D0 GitHub - castorini/orbit: Open Web-Reasoning for Information Retrieval Tasks Β· GitHub
[go: up one dir, main page]

Skip to content
Dismiss alert
castoriniPublic

About

Open Web-Reasoning for Information Retrieval Tasks

Resources

Contributing

Stars

17 stars

Watchers

0 watching

Forks

Latest commit

Β 

History

14 Commits

Folders and files

NameName
Last commit message
Last commit date
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 

Repository files navigation

ORBIT

ORBIT

ORBIT: Open Web-Reasoning for Information Retrieval Tasks

| Paper | Model | Dataset | Data Pipeline | Training | Evaluation |

arXiv GitHub Stars Model Dataset License Python


ORBIT Flowchart


Features

  • 🌐 4-round data pipeline β€” turns raw Wikipedia seeds into a verified multi-hop QA dataset through generation, self-verification, and external judging
  • πŸ’Έ Zero API spend β€” Zero dollars spent on expensive Web Search APIs or search-enabled LLMs; the entire pipeline runs on Selenium + DeepSeek Chat and uses search aggregators such as DDGS
  • πŸ€– GRPO training with web search β€” trains search-augmented LLMs via verl-tool with a parallel multi-backend DDGS retrieval server
  • πŸ“Š 12-benchmark evaluation suite β€” covers single-hop (NQ, TriviaQA, PopQA), multi-hop (HotpotQA, 2WikiMHQA, MuSiQue), and hard reasoning (FRAMES, GAIA, MoNaCo) benchmarks

πŸ“š Contents

Data Pipeline

ORBIT QA pairs are built through four sequential rounds. See data/README.md for the full overview.

Round What it does Key script
1 Wikipedia category pages β†’ seed titles create_seeds.py
2 Seeds β†’ multi-hop reasoning questions + answers deepseek_generate_qa.py
3 Self-verification via DeepSeek Chat + Selenium deepseek_self_verify.py
4 External verification on scraped documents via vLLM external_verification.py
Dataset Example β€” TV Shows & Movies domain (click to expand)

Question: What was the exact runtime (minutes) of the 2017 animated feature set inside a smartphone's messaging application, directed by a filmmaker previously known for sequels to popular children's franchises, featuring a protagonist whose facial expression malfunctions, with voice casting that includes a lead actor from a critically acclaimed sitcom, and produced by a studio that later won an Oscar for Spider-Man animation?

Answer: 86 minutes

Verification / Summary of Supporting Facts:

Clue Supporting Evidence
βœ… Animated Set The Emoji Movie (2017), set inside a smartphone messaging app world called Textopolis
βœ… Filmmaker Tony Leondis, previously directed sequels: Lilo & Stitch 2, Kung Fu Panda: Secrets of the Masters
βœ… Protagonist Gene, the main character in The Emoji Movie, struggles with malfunctioning facial expressions
βœ… Voice Cast T.J. Miller, lead actor from the HBO sitcom Silicon Valley
βœ… Studio Sony Pictures Animation, which later won an Oscar for Spider-Man: Into the Spider-Verse

πŸ’Ύ Installation

Installing uv itself:

curl -LsSf https://astral.sh/uv/install.sh | sh
uv sync
source .venv/bin/activate

Training/Evaluation Data

Prior to training & evaluating search agents, you should prepare the training, eval and test datasets. See models/data_process for the full data processing scripts.

export HF_HOME=/u3/n3thakur/projects/cache
export HF_TOKEN=hf_...

Training data β€” ORBIT + NQ + HotpotQA mixed at a 1:1:1 ratio:

python models/data_process/prepare_train_data.py \
    --datasets 'nq,hotpotqa,orbit-ai/orbit-20k' \
    --local_dir models/train/data/mix-nq-hotpotqa-orbit-ratio-1-1-1 \
    --ratio 1:1:1

Eval data β€” all 12 validation benchmarks:

python models/data_process/prepare_eval_data.py \
    --data_sources nq,triviaqa,popqa,hotpotqa,2wikimultihopqa,musique,bamboogle,frames,gaia,monaco,webwalkerqa,webshaper \
    --local_dir models/eval/data/all-12-val-datasets

Test data β€” 8 Wikipedia test benchmarks:

python models/data_process/prepare_test_data.py \
    --local_dir models/eval/data/all-8-wikipedia-test-datasets

Training Code

To train search agents using ORBIT, please see models/train.

export HF_TOKEN="hf_..."
export WANDB_API_KEY="..."

bash models/train/ddgs_web_search.sh # run ddgs web server
bash models/train/run_grpo.sh # train search agent with verl-tool

Evaluation Code

To evaluate search agents trained using ORBIT, please see models/eval.

bash models/eval/retrieval_server_bge.sh # run BGE index using faiss (conda env required)
bash models/eval/run_eval.sh # evaluate search agent on Wikipedia datasets

Repository Structure

orbit/
β”œβ”€β”€ data/
β”‚   β”œβ”€β”€ round-1-seed-creation/         # Wikipedia pages β†’ seed JSONL
β”‚   β”œβ”€β”€ round-2-qa-generation/         # Seeds β†’ inverted QA pairs (DeepSeek)
β”‚   β”œβ”€β”€ round-3-self-verification/     # QA pairs β†’ self-verified answers (DeepSeek)
β”‚   β”œβ”€β”€ round-4-external-verification/ # Verified pairs β†’ externally judged (vLLM)
β”‚   └── outputs/                       # Intermediate + final JSONL files
β”œβ”€β”€ models/
β”‚   β”œβ”€β”€ train/                         # GRPO training + DDGS retrieval server
β”‚   β”œβ”€β”€ eval/                          # BGE retrieval index + evaluation
β”‚   └── data_process/                  # Prepare train/eval parquet files
β”œβ”€β”€ pyproject.toml
└── README.md

Acknowledgements

We thank the following open-source projects:

  • verl-tool for the GRPO training framework for training search agents.
  • vLLM for LLM inference.
  • DDGS for a web search aggregator tool (used for search agent training with GRPO).
  • Search-R1 for retrieval server design inspiration and initial experiments.

Contact

If you have any questions or suggestions, please contact us at:

Citation

If you find this data generation repository helpful, please cite our preprint work ORBIT: Scalable and Verifiable Data Generation for Search Agents on a Tight Budget:

@misc{thakur2026orbit,
      title={ORBIT: Scalable and Verifiable Data Generation for Search Agents on a Tight Budget}, 
      author={Nandan Thakur and Zijian Chen and Xueguang Ma and Jimmy Lin},
      year={2026},
      eprint={2604.01195},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2604.01195}, 
}

About

Open Web-Reasoning for Information Retrieval Tasks

Resources

Contributing

Stars

17 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages

0