- sft_trainer.py: SFT trainer based on model engine, support various backends: fsdp, megatron, veomni, torchtitan. Launched by
torchrunand run in multi-controller mode. - [EXPERIMENTAL] sft_trainer_ray.py: SFT trainer based on model engine with single-controller mode. Launched by ray with a driver process coordinating multiple worker processes.
| trainer | description | sync/async | trainer/rollout | partial rollout |
|---|---|---|---|---|
| main_ppo.py | rollout until a batch is completed, then train | synchronous | colocated | No |
| TBD | kimi-1.5 style trainer: streaming rollout with capped length partial rollout | asynchronous | colocated | Yes |
| TBD | Areal style trainer: fully decoupled trainer and rollout with staleness control | asynchronous | disaggregated | Yes |
- main_generation_server.py: Launch standalone servers and generate responses for a specified prompt dataset.
- main_eval.py: Evaluate the performance of generated responses with reward function on a specified prompt dataset.