8000 GitHub - inkitori/LatentCOT: latent reasoning for language models · GitHub
[go: up one dir, main page]

Skip to content
inkitoriPublic
forked from w-jiaqi/LatentCOT

About

latent reasoning for language models

Resources

Stars

0 stars

Watchers

0 watching

Forks

 
 

Repository files navigation

LatentCOT

chain-of-thought in latent space instead of token space. llama-3.2-1B, trained on gsm8k and synthetic multiplication.

normally the model has to verbalize each reasoning step. we sft the model instead to learn a superposition of embeddings to compress the reasoning chains

running

pip install -r requirements.txt
wandb login
python sft/latent_cot_sft.py -c configs/latent_cot_sft/4x4/<config>.yaml

training scripts are in sft/, eval in eval/, configs in configs/, slurm launchers in run/.

About

latent reasoning for language models

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages

0