Appearance-based gaze estimation from face images, trained on MPIIFaceGaze.
Gazelle predicts where a person is looking from a single RGB face image. Given a cropped face, the model regresses a 3D gaze direction, expressed as pitch and yaw angles, without any calibration or special hardware.
The project is trained and evaluated on the MPIIFaceGaze dataset using a leave-one-person-out protocol.
The requires uv to manage Python and dependencies.
git clone https://github.com/sepax/gazelle.git
cd gazelle
uv syncThis project uses MPIIFaceGaze, which contains face images from 15 participants with 3D gaze annotations. It is hosted on DaRUS (University of Stuttgart), DOI 10.18419/DARUS-3240.
The data is released under CC BY-NC-SA 4.0 and may only be used for non-commercial scientific purposes.
From the repository root:
uv run python scripts/download.pyThis downloads and extracts MPIIFaceGaze (about 1 GB). It also fetches the 15 small MPIIGaze evaluation sample lists, which define the 3,000 benchmark samples per participant, without downloading the rest of MPIIGaze. Parts that are already present are skipped, and --force downloads everything again.
After downloading, data/ looks like this:
data/
├── MANIFEST.TXT
├── MPIIFaceGaze/
│ ├── changelog.txt
│ ├── readme.txt
│ └── Data/
│ ├── p00/
│ ├── ...
│ └── p14/
└── MPIIGaze/
└── Evaluation Subset/
└── sample list for eye image/
├── p00.txt
├── ...
└── p14.txt
See data/MPIIFaceGaze/readme.txt for the annotation format.
Faces are normalised to a fixed virtual camera and saved as 224x224 images, following the GazeHub benchmark so that results are comparable with published numbers. The code is the benchmark authors' own, in src/gazelle/data/gazehub/data_processing_core.py and data_processing_mpii.py. The input and output paths are set at the top of the latter.
From the repository root:
uv run python -m gazelle.data.gazehub.data_processing_mpiiThis takes a couple of minutes and writes 3,000 samples per participant (45,000 in total):
data/processed/MPIIFaceGaze/
├── Image/
│ └── p00/{face,left,right}/1.jpg ... 3000.jpg
└── Label/
└── p00.label
Training uses the Face image and the 2DGaze label from each .label file, which holds the gaze as (yaw, pitch) in radians. Samples from right eyes are mirrored, together with their labels, as in the benchmark. The eye crops are not used.
- X. Zhang, Y. Sugano, M. Fritz, A. Bulling. It's Written All Over Your Face: Full-Face Appearance-Based Gaze Estimation. CVPRW 2017.
- X. Zhang, Y. Sugano, M. Fritz, A. Bulling. MPIIGaze: Real-World Dataset and Deep Appearance-Based Gaze Estimation. IEEE TPAMI 41(1), 2019.
- Y. Cheng, H. Wang, Y. Bao, F. Lu. Appearance-based Gaze Estimation With Deep Learning: A Review and Benchmark. arXiv:2104.12668, 2021.
Train a single leave-one-person-out fold:
uv run python -m gazelle.train --config configs/resnet18.yamlAny Config field can be overridden on the command line as field=value, so there is no
separate flag per setting:
uv run python -m gazelle.train --config configs/resnet18.yaml test_subject=3 epochs=10 lr=3e-5Precedence is dataclass defaults, then the YAML file, then overrides. Unknown keys are rejected rather than ignored. Training runs on PyTorch Lightning, which picks the accelerator automatically: CUDA, else Apple MPS, else CPU.
Each run writes a timestamped directory:
runs/p00_resnet18_1014-1347/
├── config.yaml every field resolved, defaults and overrides included
├── checkpoints/
│ ├── best.ckpt lowest validation angular error
│ └── last.ckpt final epoch, and the one that is scored
└── events.out.tfevents.* TensorBoard scalars
Follow along in a second terminal:
uv run --no-sync tensorboard --logdir runsuv run python -m gazelle.evaluate runs/p00_resnet18_1014-1347/checkpoints/last.ckptThe config stored inside the checkpoint rebuilds the matching model and transform, so a
checkpoint is always scored exactly as it was trained. Pass --subject N to score a
participant other than the fold's held-out one.
The reportable protocol is separate from a single run: train on 14 participants, test on the 15th, repeat for all 15, and report the mean angular error.
uv run python -m gazelle.benchmark --config configs/resnet18.yamlUse --folds 0 1 2 to run a subset. Results are grouped per benchmark:
benchmarks/1014-1400/
├── summary.json mean, std, per-fold errors, and the config used
├── p00_resnet18_1014-1400/ each fold is a full run directory
└── ...
Hyperparameters must be frozen before running this. Tuning against these numbers overfits the benchmark and makes them meaningless, so do the tuning on one development fold first, then freeze and run all 15.
Preprocessing follows the GazeHub benchmark, and the metric is the angle between predicted and true 3D gaze directions, so results are comparable with published numbers. Training hyperparameters are our own and will differ.
Run the test suite:
uv run pytestLint and format:
uv run ruff check .
uv run ruff format .