This project involves developing a deep learning model to reconstruct 3D faces from 2D images, with real-time feedback to enhance dynamic visualization. The model processes each frame in a video stream, creating accurate 3D face representations optimized for applications like videoconferencing and interactive visualization.
- Real-time 3D face reconstruction from 2D images.
- Efficient transmission of facial data for low-latency applications.
- Real-time visualization capabilities using WebGL Studio.
The project employs a PCA-based 3D model to represent facial configurations efficiently. The AFLW2000-3D dataset and 3DDFA library were used to create 3D point clouds, with a neural network trained using ResNet for dimensionality reduction. The model’s outputs are used to reconstruct the 3D face in real time.
- Deep Learning Frameworks: PyTorch, TensorFlow, CNNs
- Visualization Platform: WebGL Studio
- Techniques Used: Principal Component Analysis (PCA), ResNet architecture for feature extraction, 3D point cloud generation
- The AFLW2000-3D dataset, 3DDFA, PCA implementations, and trained models are not included.
This project is licensed under the MIT License.
This section documents every component of the repository for a technical interview, covering structure, algorithms, data interfaces, training/inference pipelines, dependencies, setup, and known issues with recommended fixes.
- LICENSE (MIT)
- README.md (overview + deep-dive)
- Train_Model/
- FacialDataset.py — custom Dataset that loads images + 25-D target vector
- FileWalker.py — directory traversal helpers (File/Folder abstraction)
- TrainModel.py — model initialization options and training loop
- pycache/ (build artifacts)
- Use_Model/
- UseModelExample.py — real-time inference demo (webcam, Haar face detect, WebSocket stream)
- haarcascade_frontalface_alt2.xml — OpenCV cascade for face detection
- Task: Regress a compact 3D face representation (25 coefficients) from a 2D face crop.
- Output: 25-D vector (e.g., PCA coefficients) used to reconstruct/animate a 3D face in a downstream renderer (WebGL Studio).
- Modeling: CNN backbone (default ResNet-152 pretrained on ImageNet) with a final Linear layer of size 25; optimized with MSELoss.
Training co 8000 de expects a dataset path relative to Train_Model/ as: ../images/<DATASET_NAME>/. Example for data_list=['300W3D-Train']:
images/ 300W3D-Train/ <subject_or_sequence>/ img/ frame_00001.jpg|png ... annot/ frame_00001.pts (text file with floats)
Notes:
- FacialDataset loads images from img/ and targets from annot/.
- Label files matched by name containing 'pts'. The code skips header/footer lines typical of .pts files and parses floating point rows into a 25-D vector (np.array(...), column-major flatten).
- Step parameter can downsample sequences (step=1 loads every frame).
- Dataset: Train_Model/FacialDataset.py
- Class: ImageDatasets(torch.utils.data.Dataset)
- Inputs: data_list (list[str]), dir_gt (override "annot"), blurLevel (unused), onlyFace (unused), step, transform, image_size
- Returns: (img_tensor, label_tensor_float32)
- Preprocessing: PIL -> RGB, torchvision transforms
- Model initialization: Train_Model/TrainModel.py
- initialize_model(model_name, num_classes, feature_extract, use_pretrained)
- Options: resnet152 (default), alexnet, squeezenet1_0, densenet121, inception_v3
- Replaces classifier head with Linear(..., num_classes)
- Input size: 224 for most; inception branch sets 229 (see Known Issues)
- set_parameter_requires_grad(model, feature_extract)
- Freezes parameters if feature_extract=True
- Training loop: Train_Model/TrainModel.py::train_model
- Criterion: nn.MSELoss()
- Optimizer: SGD(lr=0.01, momentum=0.9)
- Batch size: 6; Epochs: 100
- Device: cuda:0 if available else cpu
- Augmentations: train transform defined but the dataset is instantiated with the 'val' transform by default (see Known Issues)
- Saves best weights by lowest loss to netFL.pt
- Real-time inference demo: Use_Model/UseModelExample.py
- Loads ResNet-152 + Linear(25) and state dict from ./netFL.pt
- Opens webcam, detects face with OpenCV Haar cascade, crops with a small border, pre-processes and infers 25-D vector
- Sends scaled coefficients over WebSocket to wss://tamats.com:55000/lgraph2 as JSON {"type":0, "channel":1, "data":[...]} for live visualization
- Draws face rectangle and shows a live window; ESC to exit
- Input: N x 3 x 224 x 224 float32, normalized to ImageNet stats
- Output: N x 25 float32 (PCA-like coefficients)
- Loss: MSE between predicted 25-D vector and ground-truth 25-D vector
- Python 3.8+ (pyproject not included; virtual env recommended)
- PyTorch, torchvision
- Pillow, numpy
- OpenCV (opencv-python)
- websocket-client
# (Windows PowerShell examples)
python -m venv .venv
./.venv/Scripts/Activate.ps1
pip install --upgrade pip
pip install torch torchvision pillow numpy opencv-python websocket-clientcd Train_Model
# Ensure dataset is available at ../images/300W3D-Train/ as described above
python TrainModel.py
# netFL.pt will be saved in Train_Model/ (or the current working dir)Key training params inside TrainModel.py:
- model_name = 'resnet' (change to 'alexnet' | 'squeezenet' | 'densenet' | 'inception')
- num_classes = 25
- batch_size = 6
- num_epochs = 100
- feature_extract = False
cd Use_Model
# Place the trained weights as ./netFL.pt
python UseModelExample.py
# A window opens; press ESC to quit- FileWalker.py
- walk: uses os.path.isfile(path) instead of os.path.isfile(os.path.join(path, f)) when iterating; this misclassifies entries.
- File.init: uses self.full_Path (typo) but attribute is full_path.
- Fix: use os.path.isfile(os.path.join(path, f)) and rename full_Path -> full_path.
- FacialDataset.py
- Imports FileWalker but uses file_walker.walk (undefined variable). Use FileWalker.walk(...) or from FileWalker import walk.
- blurLevel and onlyFace are currently unused.
- TrainModel.py
- Data transforms: dataset instantiated with 'val' transform; if training, prefer 'train' transform for augmentation.
- Dataloader shuffle is set to False; consider shuffle=True for training.
- Inception input size is set to 229; typical Inception v3 uses 299x299.
- train_model prints references to external "dataloader" variable; better to use local argument consistently.
- No validation split/metrics; only training loss is tracked.
- UseModelExample.py
- Cropping uses x-10/y-10 which can underflow indices near borders; clamp to image bounds to avoid errors.
- WebSocket endpoint is external; ensure connectivity or make it configurable.
- Add a proper train/val split and evaluation metrics (e.g., MAE/MSE on held-out set).
- Log training with TensorBoard.
- Export ONNX/TorchScript for deployment.
- Replace Haar cascade with a modern face detector (e.g., RetinaFace) for robustness.
- Consider 3D-aware losses or 3D supervision if available.
- Fix random seeds for torch, numpy, and Python.
- Log exact package versions and GPU info.
- Save checkpoints and training configs alongside results.
RT3D-FaceRecon/
├── LICENSE
├── README.md
├── Train_Model/
│ ├── FacialDataset.py
│ ├── FileWalker.py
│ ├── TrainModel.py
│ └── __pycache__/
│ ├── FacialDataset.cpython-310.pyc
│ └── FileWalker.cpython-310.pyc
└── Use_Model/
├── UseModelExample.py
└── haarcascade_frontalface_alt2.xml
- Training backbone: ResNet-152 (ImageNet-pretrained), configurable via model_name
- Output dimension: 25
- Loss: MSELoss
- Optimizer: SGD(lr=0.01, momentum=0.9)
- Batch size: 6
- Epochs: 100
- Input size: 224x224 (except inception branch in code uses 229; see Known Issues)
- Normalization: mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]
- Dataset root: Train_Model/ expects ../images// with img/ and annot/
- Inference stream: WebSocket to wss://tamats.com:55000/lgraph2, JSON payload with "data" as 25-length float array