8000 GitHub - DominiqueLoyer/TREC_AP_88-90: TREC 88-90 compétition #NLP #search_engine on Association Press articles (243K) · GitHub
[go: up one dir, main page]

Skip to content

About

TREC 88-90 compétition #NLP #search_engine on Association Press articles (2 10BC8 43K)

Resources

Stars

1 star

Watchers

1 watching

Forks

Latest commit

 

History

93 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

cff-version: 1.2.0
title: "TREC Competition: Information Retrieval on AP 88-90"
authors:
  - family-names: Loyer
    given-names: Dominique S.
    orcid: "https://orcid.org/0009-0003-9713-7109"
version: 1.0.0
date-released: 2025-01-06
url: "https://github.com/DominiqueLoyer/TREC_COMPETITION_88-90_AP"
repository-code: "https://github.com/DominiqueLoyer/TREC_COMPETITION_88-90_AP"
license: MIT
keywords:
  - "Information Retrieval"
  - "TREC"
  - "BM25"

TREC ASSOCIATION PRESS COMPETITION 88-90

Buy me a coffee Sponsor on GitHub DOI SysCRED Hugging Face Dataset TREC AP88-90 Corpus English

TREC Competition: Information Retrieval on AP 88-90

Complete Python implementation of information retrieval models evaluated on TREC AP 88-90 collections.

Overview

This project implements and evaluates multiple information retrieval approaches including BM25, TF-IDF, and Vector Space Models on the TREC AP (Associated Press) 88-90 collection containing 165,000 documents.

Features

  • BM25 ranking model with parameter tuning
  • TF-IDF vector space model
  • Query expansion techniques (RM3, pseudo-relevance feedback)
  • Comprehensive evaluation metrics (MAP, NDCG, Precision@K)
  • Integration with Pyserini for efficient indexing and retrieval
  • Comparative analysis of stemming strategies

Installation

pip install numpy pandas scikit-learn pyserini

Integration with systemFactChecking

This TREC repository has been successfully integrated into the systemFactChecking project (v2.3.0) as the core Information Retrieval engine for evidence gathering in fact-checking and credibility verification.

Key Integration Components:

  • trec_retriever.py: Evidence retrieval module based on this repository
  • trec_dataset.py: TREC AP88-90 dataset loader
  • ir_engine.py: BM25, TF-IDF, and Query Likelihood models
  • eval_metrics.py: MAP, NDCG, Precision@K evaluation metrics

SysCRED

For detailed analysis of the integration, see TREC_INTEGRATION_ANALYSIS.md

Performance Metrics

• Baseline (BM25, long queries): MAP=0.2205 • With Query Expansion (RM3): MAP=0.2948 (+34%) • Best configuration: Long query terms + BM25 + RM3 expansion

Datasets

• Collection: TREC AP 88-90 (165,000 documents) • Queries: TREC topics • Format: Standard TREC format

Related Publications

• Evaluation of Information Retrieval Models and Query Expansion on the TREC AP 88-90 Collection • Evaluation de Modeles de Recherche d’Information • Evaluation de modeles de ponderations pour la recherche d’information sur TREC AP 88-90

Visualizations

BM25 TrainMetrics Configuration Configuration Configuration Configuration

Map and Pat10 Configuration

Training data • Corpus: AP 1988–1990 newswire subset from the TREC collections. • Preprocessing: • Standard TREC tokenization and cleaning (e.g., SGML tag removal, lower‑casing if applicable). • Train/dev/test splits following the official TREC topics and relevance judgments. • For full details and statistics, see the associated dataset:  DomLoyer/trec-ap88-90-corpus . Training procedure • Objective: • Point‑wise, pair‑wise or list‑wise ranking loss (e.g., cross‑entropy over relevant vs. non‑relevant documents). • Hyperparameters (example, à adapter à ta config réelle): • Encoder:  t5-base  (or the actual base model you used). • Batch size, learning rate, number of epochs, max sequence length, optimizer. • Hardware: • Specify GPU(s), training time and important implementation details. Evaluation • Benchmarks: • TREC ad‑hoc tasks on AP 88–90 topics. • Metrics: • Mean Average Precision (MAP). • Normalized Discounted Cumulative Gain (nDCG) at standard cut‑offs. • Report: • Provide a small table summarizing your main scores (e.g., MAP@100, nDCG@20) compared to baselines. Ethical considerations • The dataset contains news articles from a specific historical period and geographical context. • Users should be aware of potential biases and outdated terminology present in the corpus. • Do not deploy this model directly in user‑facing search engines without additional filtering, monitoring and evaluation. Citation If you use this model or the underlying corpus in your research, please cite: • Your paper / preprint describing the model and experiments. • The original TREC collection or any official references required by the dataset license.

About

TREC 88-90 compétition #NLP #search_engine on Association Press articles (243K)

Resources

Stars

1 star

Watchers

1 watching

Forks

Releases

Packages

Contributors

Languages

0