8000 GitHub - jonghwi-kim/milq: Official code and dataset for "MiLQ: Benchmarking IR Models for Bilingual Web Search with Mixed Language Queries" (EMNLP 2025 Main). · GitHub
[go: up one dir, main page]

Skip to content
jonghwi-kimPublic

About

Official code and dataset for "MiLQ: Benchmarking IR Models for Bilingual Web Search with Mixed Language Queries" (EMNLP 2025 Main).

Resources

Stars

18 stars

Watchers

0 watching

Forks

Latest commit

 

History

5 Commits

Folders and files

Repository files navigation

MiLQ: Benchmarking IR Models for Bilingual Web Search with Mixed Language Queries


Official repository for "MiLQ: Benchmarking IR Models for Bilingual Web Search with Mixed Language Queries" (EMNLP 2025 Main).

Paper Dataset

📊 MiLQ

TL;DR: MiLQ is the first benchmark for evaluating retrieval models on realistic mixed-language queries preferred by bilingual speakers.

Experiment Results

Main Results

Key Findings:

  • Existing IR models (Mono-, Cross-Distill) show moderate but unstable MQIR performance across scenarios, underscoring the need for more robust systems for bilinguals’ flexible querying.

  • Mixed-language queries (MQIR) markedly outperform native queries (CLIR) in English document retrieval, offering a clear strategic edge for bilingual users.

For more details, see our paper. To reproduce results or use MiLQ, see Reproduction Guide below.

Dataset

The MiLQ dataset is available through ELRA (European Language Resources Association):

🔗 Dataset Link: ELRA-E0047

  • ISLRN: 200-586-423-805-2
  • License: Evaluation Use - ELRA EVALUATION (Free)
  • Citation: If you use this dataset, please include the following citation in your acknowledgements: "MiLQ:Mixed-Language Query Test Set for Bilingual Web Search - Evaluation Package, ELRA catalogue (http://catalog.elra.info), ISLRN: 317-005-302-361-6, ELRA ID: ELRA-E0047"

Reproduction Guide

1. Environment Setup

# Install uv
curl -LsSf https://astral.sh/uv/install.sh | sh

# Clone repository
git clone https://github.com/jonghwi-kim/milq.git
cd milq

# Install dependencies
uv venv --python 3.10
uv sync

The ColBERT-X retriever (the colbert module used by the training/eval scripts) is installed from the plaid-x PyPI package — no submodule checkout is required.

System dependency: pyserini (used for the BM25 / Contriever baselines) requires a JDK (11 or 21) on PATH. uv does not provide a JVM, so install one separately (e.g. apt install openjdk-21-jdk) if you run those baselines.

2. Test Data Preparation

CLEF Test Suite (ELRA-E0008) - Registration and purchase required

NeuCLIR 2022 (NIST) - Publicly available

bash scripts/preprocess_clef.sh ./data/clef
bash scripts/preprocess_neuclir.sh [--download] ./data/neuclir

3. Model Training (ColBERT)

Download training data:

bash scripts/download_msmarco.sh ./data/msmarco

This downloads MS MARCO (en), mMARCO (fr, de), NeuMARCO (fa, zh, ru), plus teacher scores from an mt5-xxl cross-encoder.

All distillation models are trained on MS MARCO (or its translations); the same English-based teacher scores align across languages because the translated collections/queries preserve the original MS MARCO IDs. The per-language training-data sources are:

Component Source Used by
English queries MS MARCO queries.train.tsv Cross-Distill (EN→XX)
English collection MS MARCO collection.tar.gz Mono-Distill (EN→EN)
French / German queries & collection mMARCO (unicamp-dl/mmarco) Mono-/Cross-Distill
Chinese / Russian queries mMARCO (unicamp-dl/mmarco) Mono-Distill (XX→XX)
Farsi queries tdist (hltcoe/tdist-msmarco-scores, msmarco.train.query.fas.tsv.gz) Mono-Distill (XX→XX)
Farsi / Chinese / Russian collection NeuMARCO (neuclir/neumarco, Sockeye-translated MS MARCO) Mono-/Cross-Distill
Swahili / Somali / Finnish queries generated via NLLB-200 (see Translate additional languages) Mono-/Cross-Distill
Teacher scores tdist mt5xxl-monot5-mmarco-engeng.jsonl (mt5-xxl monoT5 cross-encoder) all distillation models (KD)

For the NeuCLIR document languages (zho, fas, rus), the training collection is the NeuMARCO (translated MS MARCO) version — not the NeuCLIR test documents, which are used only at evaluation. Mono-Distill and Cross-Distill differ solely in the query language: Mono-Distill uses native queries (e.g. Chinese), Cross-Distill uses English queries; both share the same native collection and teacher scores.

Note: training scripts use full language names (chinese, farsi, russian), while the NeuCLIR eval script uses ISO codes (zho, fas, rus). Persian is farsi in training and fas in evaluation.

Translate additional languages:

bash scripts/translate.sh swahili swh_Latn
bash scripts/translate.sh somali som_Latn
bash scripts/translate.sh finnish fin_Latn

For languages without an existing translated MS MARCO version, we directly generate translations using NLLB-200. For language codes, see FLORES-200.

Apply code-switching for Mixed-Distill:

bash scripts/codeswitch.sh english chinese MUSE/en-zh.json
bash scripts/codeswitch.sh english swahili PanLex/eng_swh.json

Mixed-language queries are generated via lexicon-based code-switching. All lexicon files are available under data/lexicon.

Train ColBERT-X:

# Train Cross-Distill (ex. English queries -> Chinese docs)
bash scripts/train_colbert.sh english chinese

# Train Mono-Distill (ex. Chinese queries -> Chinese docs)
bash scripts/train_colbert.sh chinese chinese

# Train Mixed-Distill (ex. English & Chinese Code-switched queries -> Chinese docs)
bash scripts/train_colbert.sh english_chinese chinese

Arguments:

  • <query_lang>: Query language (ex. english_chinese)
  • <doc_lang>: Document language (ex. chinese)

Uses local training data:

  • Queries: data/msmarco/<query_lang>/<query_lang>_queries.tsv
  • Documents: data/msmarco/<doc_lang>/<doc_lang>_collection.tsv
  • Teacher scores: data/msmarco/teacher-scores/mt5xxl-monot5-mmarco-engeng.jsonl

Checkpoints saved to experiments/milq/none/<query_lang>_<doc_lang>/....

4. Evaluation

MiLQ mixed-language queries (MQIR):

The MiLQ mixed-language queries are distributed separately through ELRA (ELRA-E0047); the preprocessing scripts above only produce the native and English queries from CLEF/NeuCLIR. To evaluate the MQIR scenario, obtain the MiLQ package and make its mixed queries available to the eval script below.

The eval script reads queries from <query_lang>_queries_concat.tsv in the document-language directory. So place/rename the MiLQ mixed queries to follow that convention, assigning a <query_lang> label of the form eng_<doc_lang> (e.g. eng_zho) to keep them distinct from the native (<doc_lang>) and English (eng) queries:

data/neuclir/<doc_lang>/eng_<doc_lang>_queries_concat.tsv   # e.g. data/neuclir/zho/eng_zho_queries_concat.tsv

(eng_zho here is the label you assign; the exact filenames inside the ELRA package may differ and may need renaming.)

Index and Search with ColBERT-X:

bash scripts/index_search_colbert.sh <query_lang> <doc_lang>

# CLIR  (English query -> native doc):   query_lang=eng
bash scripts/index_search_colbert.sh eng zho
# MonoIR (native query -> native doc):   query_lang=<doc_lang>
bash scripts/index_search_colbert.sh zho zho
# MQIR  (mixed query -> native doc):     query_lang=eng_<doc_lang>
bash scripts/index_search_colbert.sh eng_zho zho

Arguments:

  • <query_lang>: Query-file label — eng (CLIR), <doc_lang> (MonoIR), or eng_<doc_lang> (MQIR). Selects <query_lang>_queries_concat.tsv.
  • <doc_lang>: Document language (ex. fas, zho, rus)

Note: index_search_colbert.sh currently wires up NeuCLIR paths only (fas, zho, rus). Running MQIR/CLIR/MonoIR for the CLEF languages (sw, so, fi, de, fr) requires adapting the script to the CLEF file layout documented below.

NeuCLIR:

  • Collection: data/neuclir/<doc_lang>/<doc_lang>_passages_xlm-roberta-large_180_90.tsv
  • Queries: data/neuclir/<doc_lang>/<query_lang>_queries_concat.tsv
  • Mapping: data/neuclir/<doc_lang>/<doc_lang>_mapping_xlm-roberta-large_180_90.tsv
  • Qrels: data/neuclir/<doc_lang>/2022-qrels.<doc_lang>

CLEF:

  • Collection: data/clef/docs/<doc_lang>_<year>_passages_bert-base-multilingual-uncased_180_90.tsv
  • Queries: data/clef/queries/<query_lang>_2000-2003_queries_concat.tsv
  • Mapping: data/clef/docs/<doc_lang>_<year>_mapping_bert-base-multilingual-uncased_180_90.tsv
  • Qrels: data/clef/qrels/en_2000-2003_qrels.tsv

BM25 and Contriever:

For BM25 and Contriever baselines, refer to Pyserini. Feel free to contact us with any questions.

📝 Citation

If you use MiLQ in your research, please cite:

@inproceedings{kim-etal-2025-milq,
    title = "{M}i{LQ}: Benchmarking {IR} Models for Bilingual Web Search with Mixed Language Queries",
    author = "Kim, Jonghwi  and Kang, Deokhyung  and Hwang, Seonjeong  andKim, Yunsu  andOk, Jungseul  and Lee, Gary",
    editor = "Christodoulopoulos, Christos  and  Chakraborty, Tanmoy  and Rose, Carolyn  and Peng, Violet",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2025.emnlp-main.1153/",
    doi = "10.18653/v1/2025.emnlp-main.1153",
    pages = "22654--22670",
    ISBN = "979-8-89176-332-6"
    "
}

About

Official code and dataset for "MiLQ: Benchmarking IR Models for Bilingual Web Search with Mixed Language Queries" (EMNLP 2025 Main).

Resources

Stars

18 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages

0