FLARE evaluates embedding models on an unlabeled corpus with normalizing-flow information-sufficiency estimates.
The repository is organized around three paths:
flare/: the FLARE normalizing-flow implementation.scripts/: command line tools for embedding generation, data preparation, flow training, scoring, and baselines.scripts/baselines/emir/: the EMIR baseline source used byscripts/baselines/compute_emir_knife.py.
pip install -r requirements.txtFor editable local development:
pip install -e .python scripts/generate_embeddings.py \
--dataset maiammar/Augmented_stsb_multi_mt \
--split validation \
--columns sentence2 \
--model stella \
--output-dir embeddings/maiammar_Augmented_stsb_multi_mt/stella_base_en_v2 \
--output-name val_sentence2_embeddings \
--batch-size 128 \
--max-length 512 \
--gpu-id 0python scripts/prepare/prepare_data.py \
--x_path embeddings/DATASET/SOURCE/abstract_embeddings.npy \
--y_path embeddings/DATASET/TARGET/abstract_embeddings.npy \
--output_dir data/DATASET/SOURCE_to_TARGET_no_pca \
--test_size 0.0 \
--val_size 0.1python scripts/train/train_marginal.py \
--config configs/marginal.yaml \
--data_dir data/DATASET/TARGET_marginal \
--out data/DATASET/TARGET_marginal/marginal_model.pthpython scripts/train/train_conditional.py \
--config configs/conditional.yaml \
--data_dir data/DATASET/SOURCE_to_TARGET_no_pca \
--marginal_model data/DATASET/TARGET_marginal/marginal_model.pth \
--out data/DATASET/SOURCE_to_TARGET_no_pca/cond_flow.ptpython scripts/score/compute_flare_is.py \
--data-dir data/DATASET \
--models gte_Qwen2_7B_instruct Linq_Embed_Mistral SFR_Embedding_Mistral \
Zeta_Alpha_E5_Mistral bge_multilingual_gemma2 GritLM_7B \
all_MiniLM_L6_v2 stella_base_en_v2 \
--target-dims configs/model_dims.json \
--output results/flare_is/DATASET.jsonGeometry baselines:
python scripts/baselines/compute_embedding_metrics.py \
--embedding_dir embeddings/DATASET \
--embedding_file abstract_embeddings.npy \
--output metrics.jsonEMIR/KNIFE baseline:
python scripts/baselines/compute_emir_knife.py \
--source-models gte_Qwen2_7B_instruct Linq_Embed_Mistral \
--target-models all_MiniLM_L6_v2 stella_base_en_v2 \
--embeddings-dir embeddings/DATASET \
--embedding-file abstract_embeddings.npy \
--output-dir results/emir_knife \
--device auto \
--batch-size 256 \
--emir-path scripts/baselines/emir