Repository navigation
Questions about reproducing BEAT on VAB-OmniGibson with Qwen2-VL-7B #2
Description
Dear BEAT authors,
I reproduced BEAT on VAB-OmniGibson with Qwen2-VL-7B (p = 0.5). Our code matches your GitHub release file-by-file, and our dataset is the one released on HuggingFace (uiuc-kang-lab/BEAT), verified by MD5. I have a few questions.
-
Which SFT batch size was used for this row? Appendix B specifies batch size 4 with gradient accumulation over 4 steps, whereas in the released train.py the Qwen2-VL defaults are per-device batch 3 and gradient accumulation 3 (bs = 3 if adapter.family == "qwen2vl", --sft_grad_accumdefaults to 3). Which of the two produced the Qwen2-VL / VAB-OmniGibson row of Table 1?
-
Is data/sft_with_negatives/ used for VAB-OmniGibson? The VAB folder ships both data/sft/ and data/sft_with_negatives/ (the latter = the former plus the 100 trigger-free counterparts of the CTL pairs). Is this file used here at all, and if so — in the SFT stage, or as the neutral part of CTL?
-
Is the training set for this row the same GPT-collected data that is released? In the released data, all 1787 rollouts come from four GPT snapshots (gpt-4o-2024-05-13, -08-06, -11-20, gpt-4o-mini-2024-07-18) — none from InternVL or Qwen2-VL — and the benign success rate is 92/368 = 25.0%, matching the "Original GPT-4o" entry of Table 1. Since §3.3 also lists InternVL 2.5-8B/38B/78B and Qwen2-VL-7B/72B as benign-policy sources: were the Qwen2-VL-7B numbers obtained by fine-tuning on these GPT-4o (off-policy) demonstrations, or on Qwen2-VL's own rollouts?
-
Checkpoint selection. With load_best_model_at_end=True and metric_for_best_model="eval_loss" in train.py, our best checkpoint is step 200 (~0.35 epoch), and after step 200 the eval loss stops decreasing and rises again. Is this expected? (Did you report the best or the final checkpoint?)
-
Did you also see CTL degrade overall performance? When I reproduce with this data at p = 0.5 and the SFT stage configured as in Appendix B (batch size 4 with gradient accumulation over 4 steps), the CTL stage makes the model's overall performance drop, instead of improving as in the paper. Did you observe anything similar during development, and what do you think causes it?
Thank you very much for releasing BEAT and for your time.
Best regards
Activity