Tái hiện thực nghiệm bài báo "LEO-MINI: An Efficient Multimodal Large Language Model using Conditional Token Reduction and Mixture of Multi-Modal Experts" (EMNLP 2025).
Yimu Wang, Mozhgan Nasr Azadani, Sean Sedwards, Krzysztof Czarnecki — University of Waterloo
LEO-MINI giải quyết hai vấn đề cốt lõi của MLLM hiện đại:
Hiệu quả (Efficiency): Visual tokens chiếm phần lớn context length của LLM.
LEO-MINI giảm 576 tokens/expert xuống còn 64 tokens bằng module CoTR (Conditional Token Reduction).
Hiệu năng (Effectiveness): Nhiều domain yêu cầu features đặc thù.
LEO-MINI dùng MMoE (Mixture of Multi-Modal Experts) — nhiều vision expert đa dạng và LoRA expert có routing thông minh trong LLM.
| So sánh | LEO-MINI-Llama3-8B | LLaVA-NeXT (2880 tokens) |
|---|---|---|
| Visual tokens | 64 | 2880 |
| FLOPs | −63.83% | baseline |
| CUDA time | −90.69% | baseline |
| MME | 1583 | 1533 |
| POPE | 90.3 | 86.5 |
| TextVQA | 75.1 | 67.6 |
Image(s) + Text Instruction
│
├─ MMoE-Vision (4 experts — tất cả frozen trong Stage 3)
│ ├─ CLIP ViT-L/14-336 → (B, 576, 1024)
│ ├─ EVA-02 CLIP-L-14-336 → (B, 576, 1024)
│ ├─ ConvNeXt-Large-D → (B, 576, 768)
│ └─ Pix2Struct-Large → (B, 576, 2048)
│
├─ CoTR [Stage 3 only] 576×4 → 64 tokens/expert
│ Eq.2 s_QUERY = Q̄_i · Ī_i^T ∈ R^{N^V × N_i}
│ Eq.3 s_SELF = 1·Ī_i · Ī_i^T ∈ R^{1 × N_i}
│ Eq.4 s_CROSS = Σ_{j≠i} 1·Ī_j · Ī_i^T ∈ R^{1 × N_i}
│ Eq.5 s_TEXT = 1·T̂ · Ī_i^T ∈ R^{1 × N_i}
│ Eq.6 α_i = softmax((Σ scores) / √d_i^V)
│ Eq.7 Ī_i = α_i · I_i → concat → (B, 64, 4864)
│
├─ Visual Projector (B, 64, 4864) → (B, 64, d_LLM)
│ Linear → GELU → Linear
│
└─ LLM + MMoE-LLM [Stage 3 only]
concat([Ī_proj, T_embed]) → LLM
down_proj replaced by:
y = f_ORI(x) + f_GEN(x) + f_top1(x) (Eq.8)
Router: 2-layer MLP + GELU ← (Ī, T, x)
Balance loss: λ·Σ_i(fraction_i − 1/E)² (λ=0.05)
| Stage | Trainable | Frozen | Data |
|---|---|---|---|
| 1 — Warmup Projector | Visual Projector | LLM, Vision Experts | EAGLE alignment |
| 2 — Full SFT | Tất cả | — | EAGLE SFT |
| 3 — Token Reduction | CoTR + MMoE-LLM + Projector | LLM backbone, Vision Experts | LLaVA-v1.5 665K |
Repo này hỗ trợ ba LLM backbone thay thế cho cấu hình 1 GPU 48 GB:
| Model | VRAM Stage 2 | VRAM Stage 3 | Optimizer Stage 2 | Chất lượng kỳ vọng |
|---|---|---|---|---|
meta-llama/Llama-3.2-1B-Instruct |
~14 GB | ~8 GB | AdamW chuẩn | Thấp nhất, nhanh nhất |
meta-llama/Llama-3.2-3B-Instruct |
~27 GB | ~14 GB | 8-bit Adam | Trung bình |
microsoft/Phi-3.5-mini-instruct |
~30 GB | ~16 GB | 8-bit Adam | Tốt nhất trong nhóm nhỏ |
meta-llama/Meta-Llama-3-8B-Instruct |
~65 GB | ~32 GB | DeepSpeed Zero2 | Paper (8× GPU) |
Llama-3.2-3B và Phi-3.5-mini cần
adamw_bnb_8bit+ gradient checkpointing ở Stage 2 (đã cấu hình sẵn trong model configs).
| Kịch bản | GPU | Ghi chú |
|---|---|---|
| Paper (full scale) | 8× A6000 48 GB | DeepSpeed Zero2 |
| Demo / đồ án (3B, Phi) | 1× A6000 / A100 48 GB | 8-bit Adam + gradient checkpointing |
| Demo / đồ án (1B) | 1× RTX 3090 24 GB | AdamW chuẩn |
| Quick eval (Colab) | T4 16 GB | 4-bit quantisation |
conda create -n leomini python=3.10 -y
conda activate leomini
# PyTorch 2.5.1 + CUDA 12.4 (pin version — PyTorch>=2.6 chỉ có cu130 yêu cầu driver mới hơn)
# Kiểm tra CUDA version: nvidia-smi | grep "CUDA Version"
# Thay cu124 → cu120/cu121 nếu driver của bạn chỉ hỗ trợ CUDA 12.0/12.1
pip install "torch==2.5.1+cu124" "torchvision==0.20.1+cu124" "torchaudio==2.5.1+cu124" \
--index-url https://download.pytorch.org/whl/cu124
pip install -r requirements.txtLưu ý NumPy:
requirements.txtpinnumpy<2.0vì PyTorch 2.x được compile với NumPy 1.x. Nếu gặp_ARRAY_API not found, chạypip install "numpy<2".
Một số model (Meta Llama, EAGLE) là gated repos — cần accept license trước khi tải:
# Đăng nhập một lần, token lưu vào ~/.cache/huggingface/token
hf loginSau đó vào trang model trên HuggingFace và nhấn "Agree and access repository":
- Llama-3.2: https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct
- Llama-3-8B: https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct
Meta Llama được approve ngay lập tức sau khi accept terms.
# LLM backbone — chọn 1 trong các backbone sau:
hf download meta-llama/Llama-3.2-1B-Instruct # 1B (nhẹ nhất)
hf download meta-llama/Llama-3.2-3B-Instruct # 3B (cân bằng)
hf download microsoft/Phi-3.5-mini-instruct # Phi (hiệu năng tốt)
hf download meta-llama/Meta-Llama-3-8B-Instruct # 8B (paper gốc)
# Vision experts
hf download openai/clip-vit-large-patch14-336
hf download google/pix2struct-large
# EVA-02 CLIP-L-14-336: tải tự động qua open_clip khi khởi tạo (nguồn: QuanSun/EVA-CLIP)
# ConvNeXt-Large-D: tải tự động qua open_clip khi khởi tạo
# EAGLE Stage-2 checkpoint (shortcut: bỏ qua Stage 1+2, chỉ dùng với Llama3-8B)
hf download NVEagle/Eagle-X4-8B-Plusmkdir -p data/eagle data/eagle_sft
# Stage 1 — LLaVA-Pretrain 558K (alignment, ~12 GB)
hf download liuhaotian/LLaVA-Pretrain \
--repo-type dataset \
--local-dir data/eagle
unzip data/eagle/images.zip -d data/eagle/images
# Stage 2 — LLaVA-1.5 SFT 665K (chỉ JSON; ảnh tải riêng — xem bên dưới)
# llava_v1_5_mix665k.json nằm trong repo LLaVA-Instruct-150K (không có repo riêng 665K)
hf download liuhaotian/LLaVA-Instruct-150K \
--repo-type dataset \
--local-dir data/eagle_sftẢnh cho Stage 2 đến từ nhiều nguồn (COCO, GQA, OCR-VQA, TextVQA, VisualGenome). Xem hướng dẫn đầy đủ tại LLaVA Data Preparation. Tất cả ảnh đặt vào
data/eagle_sft/images/.Flag
--repo-type datasetlà bắt buộc — thiếu flag nàyhf downloadsẽ báo "repo not found".
Model-specific configs nằm ở configs/models/. Mỗi file chỉ chứa các key khác với stage config mặc định (llm_path, batch size, optimizer, v.v.).
# Stage 1 — Warmup Projector
bash scripts/run_stage1.sh configs/models/llama3_2_1b.yaml # Llama-3.2-1B
bash scripts/run_stage1.sh configs/models/llama3_2_3b.yaml # Llama-3.2-3B
bash scripts/run_stage1.sh configs/models/phi3_5_mini.yaml # Phi-3.5-mini
# Stage 2 — Full SFT (3B/Phi dùng 8-bit Adam + gradient checkpointing, cấu hình sẵn)
bash scripts/run_stage2.sh configs/models/llama3_2_1b.yaml # Llama-3.2-1B
bash scripts/run_stage2.sh configs/models/llama3_2_3b.yaml # Llama-3.2-3B
bash scripts/run_stage2.sh configs/models/phi3_5_mini.yaml # Phi-3.5-mini
# Stage 3 — Token Reduction (CoTR + MMoE-LLM + Projector)
bash scripts/run_stage3.sh configs/models/llama3_2_1b.yaml # Llama-3.2-1B
bash scripts/run_stage3.sh configs/models/llama3_2_3b.yaml # Llama-3.2-3B
bash scripts/run_stage3.sh configs/models/phi3_5_mini.yaml # Phi-3.5-miniĐể train song song nhiều GPU:
NUM_GPUS=4 bash scripts/run_stage3.sh configs/models/llama3_2_3b.yamlbase_model_path: "meta-llama/Llama-3.2-1B-Instruct" # HuggingFace ID cho Stage 1
output_dir: "checkpoints/llama3_2_1b" # cách ly checkpoint giữa các models
per_device_batch_size: 8
gradient_accumulation: 8
optim: "adamw_torch"
gradient_checkpointing: false
deepspeed: nullbase_model_path: "meta-llama/Llama-3.2-3B-Instruct"
output_dir: "checkpoints/llama3_2_3b"
per_device_batch_size: 2
gradient_accumulation: 32 # effective batch = 64
optim: "adamw_bnb_8bit" # BẮTBUỘC để tránh OOM
gradient_checkpointing: true
deepspeed: nullbase_model_path: "microsoft/Phi-3.5-mini-instruct"
output_dir: "checkpoints/phi3_5_mini"
per_device_batch_size: 2
gradient_accumulation: 32
optim: "adamw_bnb_8bit"
gradient_checkpointing: true
deepspeed: nullLưu ý quan trọng: Dùng
base_model_path(không phảillm_path) trong model configs. Trainer tự động derivellm_pathcho Stage 2/3 từoutput_dir/stage{n-1}/llm_checkpoint/. Nếu dùngllm_pathtrong model config, Stage 2/3 sẽ load lại base model thay vì checkpoint Stage trước.
n_visual: 64 # N^V: 576 → 64 tokens qua CoTR
d_proj_cotr: 256 # chiều p
8000
rojection chung trong CoTR
lora_rank: 16 # LoRA rank cho tất cả experts
num_special: 3 # E=3 special LoRA experts
balance_loss_lambda: 0.05 # λ cho L_balance
learning_rate: 2.0e-5
per_device_batch_size: 4
gradient_accumulation: 16 # effective batch = 64python -m src.eval.evaluator \
--model_path checkpoints/stage3 \
--model_name "Llama-3.2-3B" \
--tasks pope,scienceqa,textvqa,mme,mmmu \
--output_dir results/llama3b# Nhanh — 200 mẫu/task, 4-bit quant (chỉ 3B)
BASE_3B="meta-llama/Llama-3.2-3B-Instruct" \
CKPT_3B="checkpoints/llama3_2_3b/stage3/stage3_adapter_weights.pt" \
TASKS="pope,scienceqa,textvqa" LIMIT=200 LOAD_4BIT=1 \
bash scripts/run_benchmark_compare.sh
# Full benchmark — cả 3 models
BASE_1B="meta-llama/Llama-3.2-1B-Instruct" \
CKPT_1B="checkpoints/llama3_2_1b/stage3/stage3_adapter_weights.pt" \
BASE_3B="meta-llama/Llama-3.2-3B-Instruct" \
CKPT_3B="checkpoints/llama3_2_3b/stage3/stage3_adapter_weights.pt" \
BASE_PHI="microsoft/Phi-3.5-mini-instruct" \
CKPT_PHI="checkpoints/phi3_5_mini/stage3/stage3_adapter_weights.pt" \
bash scripts/run_benchmark_compare.shOutput mẫu:
============================================================
BENCHMARK COMPARISON
============================================================
Task Paper(8B) Llama-1B Llama-3B Phi-3.5
------------------------------------------------------------
pope 90.3 85.x 87.x 88.x
scienceqa 84.5 79.x 81.x 82.x
textvqa 75.1 68.x 71.x 72.x
mme 1583.0 1420.x 1510.x 1530.x
mmmu 38.8 32.x 35.x 36.x
------------------------------------------------------------
python -m src.eval.evaluator \
--model_path checkpoints/stage3 \
--tasks mme,pope,textvqa,scienceqa \
--ablation
# So sánh N^V ∈ {1, 16, 64, 256}| Benchmark | Metric | Target (paper, 8B) |
|---|---|---|
| MME | Perception score | 1583.0 |
| MMBench | Overall acc | 77.0 |
| SEED-Bench | Overall acc | 75.8 |
| GQA | Exact match | 64.5 |
| ScienceQA | Acc | 84.5 |
| MMMU | Acc | 38.8 |
| POPE | Acc | 90.3 |
| AI2D | Acc | 75.7 |
| TextVQA | Acc | 75.1 |
| ChartQA | Relaxed acc | 80.5 |
| OCRBench | Acc | 62.4 |
| VizWiz | VQA acc | 69.3 |
Sau khi training, upload checkpoints/stage3/stage3_adapter_weights.pt lên Google Drive, rồi dùng Colab để load và demo.
# torch>=2.5 yêu cầu của transformers mới nhất; numpy<2 tránh xung đột NumPy 2.x
!pip install "torch==2.5.1+cu124" "torchvision==0.20.1+cu124" "torchaudio==2.5.1+cu124" --index-url https://download.pytorch.org/whl/cu124
!pip install "numpy<2" transformers peft bitsandbytes accelerate timm open-clip-torch lmms-eval huggingface_hub[hf_xet]
from google.colab import drive
drive.mount('/content/drive')
import sys
sys.path.insert(0, '/content/leo_mini') # clone repo vào /content/leo_miniimport torch
from src.models.leo_mini import LeoMini
model = LeoMini.from_pretrained(
llm_path="meta-llama/Llama-3.2-3B-Instruct", # hoặc Phi-3.5-mini
stage3_weights="/content/drive/MyDrive/leomini/stage3_adapter_weights.pt",
load_in_4bit=True,
)
model.eval()from PIL import Image
import torch
image = Image.open("test.jpg")
question = "What is shown in the image?"
# Chuẩn bị visual input
processor = model.vision.experts[0].processor # CLIPImageProcessor
pixel_values = processor(images=image, return_tensors="pt").pixel_values
# Chuẩn bị text input
input_text = f"<image>\nUSER: {question} ASSISTANT:"
enc = model.tokenizer(input_text, return_tensors="pt")
input_ids = enc.input_ids
input_ids[input_ids == model.tokenizer.convert_tokens_to_ids("<image>")] = -200
with torch.no_grad():
output_ids = model.generate(
input_ids=input_ids.cuda(),
pixel_values=pixel_values.cuda(),
max_new_tokens=256,
do_sample=False,
)
answer = model.tokenizer.decode(output_ids[0], skip_special_tokens=True)
print(answer)!python -m src.eval.evaluator \
--model_path meta-llama/Llama-3.2-3B-Instruct \
--model_name "LEO-MINI-3B" \
--tasks pope,textvqa,scienceqa \
--limit 500 \
--load_in_4bit \
--output_dir /content/resultsleo_mini/
├── docs/
│ └── leomini.pdf # Bài báo gốc (EMNLP 2025)
├── src/
│ ├── models/
│ │ ├── cotr.py # CoTR: Eq.2–7 (token reduction)
│ │ ├── mmoe_llm.py # MMoE-LLM: LoRA experts + router + balance loss
│ │ ├── vision_experts.py # CLIP / EVA-02 / ConvNeXt / Pix2Struct
│ │ ├── projector.py # Visual Projector (2-layer MLP)
│ │ └── leo_mini.py # Full pipeline + from_pretrained + set_stage()
│ ├── data/
│ │ ├── dataset.py # EAGLEDataset + LLaVADataset
│ │ └── collator.py # Batch collation
│ ├── train/
│ │ └── trainer.py # 3-stage trainer (HuggingFace Trainer)
│ └── eval/
│ ├── evaluator.py # lmms-eval wrapper + compare_models()
│ └── lmms_adapter.py # lmms-eval @register_model adapter
├── configs/
│ ├── config_stage1.yaml # Stage 1: warmup projector
│ ├── config_stage2.yaml # Stage 2: full SFT
│ ├── config_stage3.yaml # Stage 3: CoTR + MMoE-LLM
│ ├── models/
│ │ ├── llama3_2_1b.yaml # Override: Llama-3.2-1B
│ │ ├── llama3_2_3b.yaml # Override: Llama-3.2-3B (8-bit Adam)
│ │ └── phi3_5_mini.yaml # Override: Phi-3.5-mini (8-bit Adam)
│ └── deepspeed_zero2.json # DeepSpeed config (multi-GPU)
├── scripts/
│ ├── run_stage1.sh # usage: bash run_stage1.sh [model_config]
│ ├── run_stage2.sh
│ ├── run_stage3.sh
│ ├── run_benchmark_compare.sh # so sánh 3 model cùng lúc
│ └── run_eval_leomini.py # wrapper để lmms-eval nhận diện model
├── requirements.txt
└── README.md
Với mỗi expert
Trong đó
Thay thế down_proj trong mọi MLP block của LLM:
-
$f_{\text{ORI}}$ : original frozendown_projweight -
$f_{\text{GEN}}$ : general LoRA adapter (rank=16, luôn active) -
$f_i^E$ :$E=3$ special LoRA experts, top-$k=1$ được chọn bởi router - Router:
$R = \text{softmax}(f^{\text{ROUTING}}(\bar{I}, T, x)) \in \mathbb{R}^E$ - 2-layer MLP + GELU
- Input: hidden state
$x$ ⊕ global visual context ⊕ global text context
Balanced loss (tránh routing collapse):
Tổng loss = cross-entropy +
| Metric | Paper (8B) | Llama-1B (ước tính) | Llama-3B (ước tính) | Phi-3.5 (ước tính) |
|---|---|---|---|---|
| MME | 1583.0 | 1380–1450 | 1480–1530 | 1510–1560 |
| POPE | 90.3 | 83–85 | 86–88 | 87–89 |
| ScienceQA | 84.5 | 77–79 | 80–82 | 81–83 |
| TextVQA | 75.1 | 66–69 | 70–72 | 71–73 |
| Visual Tokens | 64 | 64 | 64 | 64 |
@inproceedings{wang2025leomini,
title = {LEO-MINI: An Efficient Multimodal Large Language Model
using Conditional Token Reduction and Mixture of
Multi-Modal Experts},
author = {Wang, Yimu and Nasr Azadani, Mozhgan and
Sedwards, Sean and Czarnecki, Krzysztof},
booktitle = {Proceedings of EMNLP 2025},
pages = {7246--7261},
year = {2025}
}Các công trình liên quan:
- EAGLE — Multi-encoder baseline (Shi et al., 2024); dùng
NVEagle/Eagle-X4-8B-Pluslàm Stage-2 checkpoint thay thế - LLaVA-1.5 — Visual instruction tuning
- lmms-eval — Benchmark evaluation framework
- bitsandbytes — 8-bit Adam optimizer cho Stage 2 trên single GPU