[NeurIPS 2026] 👻 GHOST: Geometry-Hierarchical Online Streaming Token Eviction for Efficient 3D Reconstruction
Leyang Chen*, Junyi Wu*, Zhiteng Li*, Yulun Zhang†
Shanghai Jiao Tong University
*Equal contribution | †Corresponding author: yulun100@gmail.com
GHOST is a causal visual geometry transformer that extends VGGT with a training-free rolling KV-cache memory, enabling stable, infinite-horizon streaming 3D reconstruction from continuous image sequences. 🚀
- ♾️ Infinite-horizon streaming inference with bounded GPU memory via importance-based KV cache eviction
- 🧠 Cosine-similarity-guided per-layer budget allocation
- 🔄 Alternating frame/global attention for joint per-frame and cross-frame reasoning
- ⚡ Efficient long-sequence processing without OOM
- 📊 Evaluated on 7-Scenes and NRGBD benchmarks
- 🏗️ Training-free streaming extension built on top of VGGT
Requirements: Python 3.11, CUDA 12.x, PyTorch 2.3.1
# 1️⃣ Clone the repository
git clone https://github.com/lokiniuniu/GHOST.git
cd GHOST
# 2️⃣ Create a conda environment
conda create -n ghost python=3.11 cmake=3.14.0
conda activate ghost
# 3️⃣ Install Python dependencies
pip install -r requirements.txt
# 4️⃣ (macOS / some Linux) Fix OpenMP conflict
conda install 'llvm-openmp<16'💡 Optional: Install
flash-kmeansfor 2–5× faster KV cache quantization encoding:pip install flash-kmeans
Download the pretrained StreamVGGT checkpoint from Hugging Face and place it under ./ckpt/:
# Using huggingface-hub CLI
pip install -U huggingface_hub
huggingface-cli download lch01/StreamVGGT --local-dir ./ckptOr download manually from:
👉 https://huggingface.co/lch01/StreamVGGT
The checkpoint file should be at:
./ckpt/model.pth
(or pass the path explicitly via --checkpoint_path)
Download from the official 7-Scenes page and place under:
./7scenes/
Download from neural-rgbd-surface-reconstruction and place under:
./datasets/nrgbd/
# 🎥 Basic inference on a folder of images
python run_inference.py \
--input_dir path/to/images/ \
--checkpoint_path ./ckpt/model.pth
# ♾️ Long sequences:
# Stream frame-by-frame and write results to disk (avoids OOM)
python run_inference.py \
--input_dir path/to/images/ \
--checkpoint_path ./ckpt/model.pth \
--frame_cache_dir path/to/output_per_frame/ \
--no_cache_resultscd src
python -m accelerate.commands.launch eval/mv_recon/launch.py \
--weights ../ckpt/model.pth \
--model_name StreamVGGT \
--dataset 7scenes \
--scenes_root ../7scenes \
--output_dir ../outputs/7scenes| Argument | Default | Description |
|---|---|---|
--dataset |
7scenes |
7scenes, nrgbd, or all |
--total_budget |
1200000 |
🎯 Total KV token budget across all layers |
--use_cosine_budget |
True |
🧠 Cosine-similarity-based per-layer budget allocation |
--eviction_mode |
importance |
🗑️ KV eviction strategy |
--importance_weights_path |
(default) | ⚖️ Custom importance weights JSON |
--max_frames |
None |
🎞️ Limit frames per sequence |
GHOST/
├── run_inference.py # 🚀 Main inference entry point
├── requirements.txt # 📦 Python dependencies
├── configs/
│ ├── importance_weights_default.json
│ │ # ⚖️ Default importance eviction weights
│ └── kv_budget_proportions_cosine.json
│ # 🧠 Per-layer KV budget proportions
├── scripts/ # 🛠️ Utility shell scripts
└── src/
├── add_ckpt_path.py # 🔗 Checkpoint path helper
├── visual_util.py # 🎨 Visualization utilities
├── streamvggt/
│ ├── models/
│ │ ├── streamvggt.py # 👻 Main model
│ │ └── aggregator.py # 🔄 Frame/global attention aggregator
│ ├── layers/
│ │ ├── attention.py # 🧠 KV cache + eviction attention
│ │ ├── block.py # 🧱 Transformer block
│ │ ├── vision_transformer.py
│ │ ├── rope.py # 🌀 Rotary positional embedding
│ │ ├── patch_embed.py
│ │ ├── mlp.py
│ │ └── swiglu_ffn.py
│ ├── heads/
│ │ ├── camera_head.py # 📷 Camera estimation
│ │ ├── dpt_head.py # 🌍 Depth prediction
│ │ ├── track_head.py # 🎯 Tracking
│ │ └── track_modules/
│ ├── eviction/
│ │ ├── importance_eviction.py
│ │ │ # 🗑️ Importance scoring
│ │ └── importance_weights_from_hyperparams.py
│ ├── quantization/ # ⚡ KV cache quantization
│ ├── kernels/ # 🧩 CUDA/sparse attention kernels
│ └── utils/
├── eval/
│ └── mv_recon/
│ ├── launch.py
│ ├── data.py
│ ├── compute_kv_budget_from_cosine_sim.py
│ └── compute_kv_budget_strategies.py
├── dust3r/ # 🏗️ DUSt3R geometry utilities
├── croco/ # 🐊 CroCo backbone utilities
└── vggt/ # 📚 Original VGGT reference
This project builds upon the following excellent open-source works:
Huge thanks to the open-source community ❤️
Citation information will be released soon.
@article{chen2026ghost,
title={GHOST: Geometry-Hierarchical Online Streaming Token Eviction for Efficient 3D Reconstruction},
author={Chen, Leyang and Wu, Junyi and Li, Zhiteng and Zhang, Yulun},
journal={arXiv preprint arXiv:2605.15852},
year={2026}
}See LICENSE.txt.
Consider giving the repo a ⭐ on GitHub!