|
|
|
|
|
End-to-end pipeline for language-based 3D indoor localization, introduced in our ECCV 2026 paper, LangLoc: “Tell Me What You See”. Given a free-form description of an observer's surroundings, LangLoc estimates the observer's 2D position and heading within a known 3D environment. Evaluated on ScanNet and 3RScan.
- Dataset creation (Sec. 3.1): keyframe selection by image quality assessment (IQA) plus a two-stage Determinantal Point Process (DPP), followed by description generation via GPT-4o-mini.
- Scene retrieval (Sec. 3.2): dual-branch Graph Attention Network v2 (GATv2) encoder with CLIP features, scoring the text-graph against database scene-graphs.
- Fine localization (Sec. 3.3): dense floor-grid scored by ray-cast object visibility.
- Dialogue disambiguation (Sec. 3.4): Bayesian posterior over poses, refined by targeted yes/no questions.
Numbers below are from rerunning the canonical reproducer scripts on
this repo with seed=42. See REPRODUCE.md for the
full per-table breakdown.
| Table | Metric | Paper | This repo |
|---|---|---|---|
| Tab. 1 (ScanScribe, 10-cand) | Top-1 Recall | 76.70 | 76.60 ± 4.29 |
| Tab. 2 (ScanScribe, full) | Top-10 Recall | 91.60 | 90.70 ± 3.58 |
| Tab. 3 (LLM-from-image, fair) | Top-1 Recall | 76.10 | 59.50 ± 5.26 (corrected protocol) |
| Tab. 4(a) 3RScan-100 (no dialog) | Pos median (m) | 1.551 | 1.470 |
| Tab. 4(b) ScanNet-100 (no dialog) | Pos median (m) | 1.314 | 0.998 |
| Tab. 5 full 1319-scene 3RScan | Pos median (m) | 1.308 | 1.230 |
Tabs. 4(b) and 5 beat the paper on position accuracy. Tab. 4(a) median beats paper and the mean is within 5 cm. Retrieval recall is within noise everywhere except Tab. 2 Top-5 (within the reported standard deviation) and the Tab. 3 corrected-protocol gap documented in REPRODUCE.md.
All three steps below are required in order. Step 3 reads files laid down by step 2.
# Step 1. Clone and install. Full details in INSTALL.md.
git clone https://github.com/<your-org>/langloc.git
cd langloc
conda create -n langloc python=3.10 -y && conda activate langloc
pip install -r requirements.txt && pip install -e .
pip install torch==2.6.0 torchvision==0.21.0 torchaudio==2.6.0 \
--index-url https://download.pytorch.org/whl/cu126
python -m spacy download en_core_web_lg
# PyTorch3D from source: see INSTALL.md.
# Step 2. Get the data. Required, not optional. Full details in DATA.md.
# Easiest path: request the Paper_Dataset Google Drive bundle
# and unpack into ./data/. The reproducer scripts read
# data/3RScan/, data/scans/, data/processed_data/eval_pool/,
# and data/model_checkpoints/graph2graph/paper/ from there.
# Step 3. Reproduce a table. See REPRODUCE.md for the full list.
# Example: scene retrieval (Tabs. 1, 2, 3), about 3 s once
# the bundle is in place.
bash scripts/retrieval/reproduce_paper_tables.shThe README is the entry point. Detailed guides live in five focused files:
| File | What it covers |
|---|---|
| INSTALL.md | Conda env, PyTorch3D from source, spaCy, environment variables, install-time troubleshooting. |
| DATA.md | Google Drive bundle (recommended) and from-scratch alternative; target layout under data/; sanity checks. |
| REPRODUCE.md | Per-table reproducer commands, paper-vs-this-repo numbers, dialog-row caveats, optional re-run of the dataset-creation pipeline. |
| CONFIG.md | Hydra config tree, project layout, performance notes, runtime troubleshooting. |
| LICENSE | Project licence. |
Self-contained tools with their own READMEs:
| Path | Purpose | Guide |
|---|---|---|
tools/annotation_website/ |
FastAPI site for crowdsourcing human descriptions and first-person localizations. | README |
tools/baselines/gpt_vlm/ |
GPT-5.5 vision describer (rebuttal closed-loop-bias defense). | README |
tools/baselines/human/ |
Extract human-written descriptions from the annotation website's SQLite DB. | README |
tools/eval/ |
Recall-at-threshold, error-distribution plots, dialog log statistics. | README |
If you use LangLoc in your research, please cite:
@inproceedings{panwar2026langloc,
title = {LangLoc: “Tell Me What You See”},
author = {Panwar, Shaurya Kishore and Nobari, Roham Zendehdel and
Lau, Shirley Feng Yi and Shaik, Abu Bakr Rahman and
G{"u}nther, Manuel and Pollefeys, Marc and Barath, Daniel},
booktitle = {European Conference on Computer Vision},
pages = {598--615},
year = {2026},
organization = {Springer},
}This repository is released under the licence in LICENSE. ScanNet and 3RScan have separate licences; review their respective terms-of-use documents before downloading data.