Official repository for "MiLQ: Benchmarking IR Models for Bilingual Web Search with Mixed Language Queries" (EMNLP 2025 Main).
TL;DR: MiLQ is the first benchmark for evaluating retrieval models on realistic mixed-language queries preferred by bilingual speakers.
Key Findings:
-
Existing IR models (Mono-, Cross-Distill) show moderate but unstable MQIR performance across scenarios, underscoring the need for more robust systems for bilinguals’ flexible querying.
-
Mixed-language queries (MQIR) markedly outperform native queries (CLIR) in English document retrieval, offering a clear strategic edge for bilingual users.
For more details, see our paper. To reproduce results or use MiLQ, see Reproduction Guide below.
The MiLQ dataset is available through ELRA (European Language Resources Association):
🔗 Dataset Link: ELRA-E0047
- ISLRN: 200-586-423-805-2
- License: Evaluation Use - ELRA EVALUATION (Free)
- Citation: If you use this dataset, please include the following citation in your acknowledgements: "MiLQ:Mixed-Language Query Test Set for Bilingual Web Search - Evaluation Package, ELRA catalogue (http://catalog.elra.info), ISLRN: 317-005-302-361-6, ELRA ID: ELRA-E0047"
# Install uv
curl -LsSf https://astral.sh/uv/install.sh | sh
# Clone repository
git clone https://github.com/jonghwi-kim/milq.git
cd milq
# Install dependencies
uv venv --python 3.10
uv syncThe ColBERT-X retriever (the colbert module used by the training/eval scripts) is installed from the plaid-x PyPI package — no submodule checkout is required.
System dependency:
pyserini(used for the BM25 / Contriever baselines) requires a JDK (11 or 21) onPATH.uvdoes not provide a JVM, so install one separately (e.g.apt install openjdk-21-jdk) if you run those baselines.
CLEF Test Suite (ELRA-E0008) - Registration and purchase required
NeuCLIR 2022 (NIST) - Publicly available
bash scripts/preprocess_clef.sh ./data/clef
bash scripts/preprocess_neuclir.sh [--download] ./data/neuclirDownload training data:
bash scripts/download_msmarco.sh ./data/msmarcoThis downloads MS MARCO (en), mMARCO (fr, de), NeuMARCO (fa, zh, ru), plus teacher scores from an mt5-xxl cross-encoder.
All distillation models are trained on MS MARCO (or its translations); the same English-based teacher scores align across languages because the translated collections/queries preserve the original MS MARCO IDs. The per-language training-data sources are:
| Component | Source | Used by |
|---|---|---|
| English queries | MS MARCO queries.train.tsv |
Cross-Distill (EN→XX) |
| English collection | MS MARCO collection.tar.gz |
Mono-Distill (EN→EN) |
| French / German queries & collection | mMARCO (unicamp-dl/mmarco) | Mono-/Cross-Distill |
| Chinese / Russian queries | mMARCO (unicamp-dl/mmarco) | Mono-Distill (XX→XX) |
| Farsi queries | tdist (hltcoe/tdist-msmarco-scores, msmarco.train.query.fas.tsv.gz) |
Mono-Distill (XX→XX) |
| Farsi / Chinese / Russian collection | NeuMARCO (neuclir/neumarco, Sockeye-translated MS MARCO) | Mono-/Cross-Distill |
| Swahili / Somali / Finnish queries | generated via NLLB-200 (see Translate additional languages) | Mono-/Cross-Distill |
| Teacher scores | tdist mt5xxl-monot5-mmarco-engeng.jsonl (mt5-xxl monoT5 cross-encoder) |
all distillation models (KD) |
For the NeuCLIR document languages (zho, fas, rus), the training collection is the NeuMARCO (translated MS MARCO) version — not the NeuCLIR test documents, which are used only at evaluation. Mono-Distill and Cross-Distill differ solely in the query language: Mono-Distill uses native queries (e.g. Chinese), Cross-Distill uses English queries; both share the same native collection and teacher scores.
Note: training scripts use full language names (
chinese,farsi,russian), while the NeuCLIR eval script uses ISO codes (zho,fas,rus). Persian isfarsiin training andfasin evaluation.
Translate additional languages:
bash scripts/translate.sh swahili swh_Latn
bash scripts/translate.sh somali som_Latn
bash scripts/translate.sh finnish fin_LatnFor languages without an existing translated MS MARCO version, we directly generate translations using NLLB-200. For language codes, see FLORES-200.
Apply code-switching for Mixed-Distill:
bash scripts/codeswitch.sh english chinese MUSE/en-zh.json
bash scripts/codeswitch.sh english swahili PanLex/eng_swh.jsonMixed-language queries are generated via lexicon-based code-switching. All lexicon files are available under data/lexicon.
Train ColBERT-X:
# Train Cross-Distill (ex. English queries -> Chinese docs)
bash scripts/train_colbert.sh english chinese
# Train Mono-Distill (ex. Chinese queries -> Chinese docs)
bash scripts/train_colbert.sh chinese chinese
# Train Mixed-Distill (ex. English & Chinese Code-switched queries -> Chinese docs)
bash scripts/train_colbert.sh english_chinese chineseArguments:
<query_lang>: Query language (ex.english_chinese)<doc_lang>: Document language (ex.chinese)
Uses local training data:
- Queries:
data/msmarco/<query_lang>/<query_lang>_queries.tsv - Documents:
data/msmarco/<doc_lang>/<doc_lang>_collection.tsv - Teacher scores:
data/msmarco/teacher-scores/mt5xxl-monot5-mmarco-engeng.jsonl
Checkpoints saved to experiments/milq/none/<query_lang>_<doc_lang>/....
MiLQ mixed-language queries (MQIR):
The MiLQ mixed-language queries are distributed separately through ELRA (ELRA-E0047); the preprocessing scripts above only produce the native and English queries from CLEF/NeuCLIR. To evaluate the MQIR scenario, obtain the MiLQ package and make its mixed queries available to the eval script below.
The eval script reads queries from <query_lang>_queries_concat.tsv in the document-language directory. So place/rename the MiLQ mixed queries to follow that convention, assigning a <query_lang> label of the form eng_<doc_lang> (e.g. eng_zho) to keep them distinct from the native (<doc_lang>) and English (eng) queries:
data/neuclir/<doc_lang>/eng_<doc_lang>_queries_concat.tsv # e.g. data/neuclir/zho/eng_zho_queries_concat.tsv
(eng_zho here is the label you assign; the exact filenames inside the ELRA package may differ and may need renaming.)
Index and Search with ColBERT-X:
bash scripts/index_search_colbert.sh <query_lang> <doc_lang>
# CLIR (English query -> native doc): query_lang=eng
bash scripts/index_search_colbert.sh eng zho
# MonoIR (native query -> native doc): query_lang=<doc_lang>
bash scripts/index_search_colbert.sh zho zho
# MQIR (mixed query -> native doc): query_lang=eng_<doc_lang>
bash scripts/index_search_colbert.sh eng_zho zhoArguments:
<query_lang>: Query-file label —eng(CLIR),<doc_lang>(MonoIR), oreng_<doc_lang>(MQIR). Selects<query_lang>_queries_concat.tsv.<doc_lang>: Document language (ex.fas,zho,rus)
Note:
index_search_colbert.shcurrently wires up NeuCLIR paths only (fas,zho,rus). Running MQIR/CLIR/MonoIR for the CLEF languages (sw,so,fi,de,fr) requires adapting the script to the CLEF file layout documented below.
NeuCLIR:
- Collection:
data/neuclir/<doc_lang>/<doc_lang>_passages_xlm-roberta-large_180_90.tsv - Queries:
data/neuclir/<doc_lang>/<query_lang>_queries_concat.tsv - Mapping:
data/neuclir/<doc_lang>/<doc_lang>_mapping_xlm-roberta-large_180_90.tsv - Qrels:
data/neuclir/<doc_lang>/2022-qrels.<doc_lang>
CLEF:
- Collection:
data/clef/docs/<doc_lang>_<year>_passages_bert-base-multilingual-uncased_180_90.tsv - Queries:
data/clef/queries/<query_lang>_2000-2003_queries_concat.tsv - Mapping:
data/clef/docs/<doc_lang>_<year>_mapping_bert-base-multilingual-uncased_180_90.tsv - Qrels:
data/clef/qrels/en_2000-2003_qrels.tsv
BM25 and Contriever:
For BM25 and Contriever baselines, refer to Pyserini. Feel free to contact us with any questions.
If you use MiLQ in your research, please cite:
@inproceedings{kim-etal-2025-milq,
title = "{M}i{LQ}: Benchmarking {IR} Models for Bilingual Web Search with Mixed Language Queries",
author = "Kim, Jonghwi and Kang, Deokhyung and Hwang, Seonjeong andKim, Yunsu andOk, Jungseul and Lee, Gary",
editor = "Christodoulopoulos, Christos and Chakraborty, Tanmoy and Rose, Carolyn and Peng, Violet",
booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
month = nov,
year = "2025",
address = "Suzhou, China",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2025.emnlp-main.1153/",
doi = "10.18653/v1/2025.emnlp-main.1153",
pages = "22654--22670",
ISBN = "979-8-89176-332-6"
"
}