This repository contains the official implementation of Latent Video Cache (Latent-VC) for grounded video reasoning.
Latent-VC introduces a recurrent latent visual cache inside the decoder of a large multimodal model to mitigate Visual Anchoring Decay in long-form video reasoning.
Instead of relying on a pure read-once, generate-many pipeline, Latent-VC constructs a compact latent visual memory before answer generation, enabling the model to preserve grounding to visual evidence throughout reasoning.
Our method is built on Qwen3.5-9B-Base and trained with two stages:
- Stage I: Supervised Fine-Tuning (SFT) with contrastive cache alignment.
- Stage II: GRPO with vision-grounded rewards and latent grounding supervision.
Title: Latent Visual Cache for Video Reasoning
Video reasoning requires Large Multimodal Models (LMMs) to remain grounded in dense visual evidence, yet existing systems largely follow a read-once, generate-many paradigm, where visual grounding weakens during generation. We identify this issue as Visual Anchoring Decay and propose Latent Video Cache (Latent-VC), a recurrent latent visual cache inserted into the decoder to preserve compact visual memories throughout reasoning. Latent-VC is trained with supervised contrastive cache alignment and vision-grounded GRPO with a latent grounding reward, while maintaining strict train-inference alignment through native decoder hidden states. Built on Qwen3.5-9B, Latent-VC consistently outperforms strong CoT and SFT+GRPO baselines across six video benchmarks, especially on grounding-intensive and long-video tasks. It also achieves higher accuracy with substantially shorter responses, suggesting that latent visual caching improves video reasoning by preserving visual evidence rather than relying on longer textual chains.
Latent-VC contains two key components:
-
Inference pipeline with recurrent latent cache: the model rolls out latent cache states before answer decoding.
-
Latent prefetch learning:
1. SFT stage aligns latent cache states with key visual moments.
2. GRPO stage optimizes answer quality, format, temporal grounding, and latent grounding jointly.
Latent-VC consistently improves over both CoT and SFT+GRPO baselines across six public video reasoning benchmarks.
latent_code/
├── datasets/
├── material/
├── scripts/
├── src/
├── requirements.txt
└── README.md
Download Qwen3.5-9B-Base from Hugging Face:
Place the model under:
models/Qwen3.5-9B-BaseDownload the video dataset from:
Place the raw video files under:
datasets/video_dataThe training annotation files used by the scripts are already expected at:
datasets/lvc_sft.json
datasets/lvc_grpo.jsonconda create -n videolvc python=3.11 -y
conda activate videolvc
pip install -r requirements.txtRun supervised fine-tuning with latent cache alignment:
bash scripts/train_sft.shThis script uses the following default paths:
- Base model:
models/Qwen3.5-9B-Base - Training data:
datasets/lvc_sft.json - Output checkpoint:
checkpoints/video_lvc_sft_all_image/
Run GRPO training starting from the Stage 1 checkpoint:
bash scripts/train_grpo.shThis script uses the following default paths:
- Base model:
models/Qwen3.5-9B-Base - SFT checkpoint:
checkpoints/video_lvc_sft_all_image/checkpoint-1200 - Training data:
datasets/lvc_grpo.json - Output checkpoint:
checkpoints/video_lvc_grpo/
Download the evaluation set from:
Place the evaluation files under:
datasetsbash scripts/eval_all_benchmarks.shBy default, the evaluation script uses:
- Eval directory:
src/r1-v/Evaluation - Output directory:
eval_results_final
You can also evaluate specific checkpoints by overriding environment variables, for example:
MODEL_TYPE=sft MODEL_PATH=checkpoints/video_lvc_sft_all_image/checkpoint-1200 bash scripts/eval_all_benchmarks.shMODEL_TYPE=grpo MODEL_PATH=checkpoints/video_lvc_grpo/checkpoint-800 bash scripts/eval_all_benchmarks.shPlease create Github issues here or email Yongheng Zhang if you have any questions or suggestions. If you find this work useful, please cite:
@misc{zhang2026latentvisualcachevideo,
title={Latent Visual Cache for Video Reasoning},
author={Yongheng Zhang and Zhipeng Xu and Hao Wu and Yinghui Li and Di Yin and Xing Sun and Philip S. Yu},
year={2026},
eprint={2607.02607},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2607.02607},
}