8000 GitHub - imkaleemahmad/RT3D-FaceRecon: A deep learning project focused on reconstructing 3D faces from 2D images, with real-time feedback for dynamic visualization. Using frameworks like PyTorch, TensorFlow, and CNNs, this model processes each video frame to produce real-time 3D facial representations. · GitHub
[go: up one dir, main page]

Skip to content

About

A deep learning project focused on reconstructing 3D faces from 2D images, with real-time feedback for dynamic visualization. Using frameworks like PyTorch, TensorFlow, and CNNs, this model processes each video frame to produce real-time 3D facial representations.

Resources

Stars

0 stars

Watchers

2 watching

Forks

Latest commit

 

History

7 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 

Repository files navigation

3D Face Reconstruction Using Deep Learning Techniques & Real-Time Visualization (RT3D-FaceRecon)

Project Overview

This project involves developing a deep learning model to reconstruct 3D faces from 2D images, with real-time feedback to enhance dynamic visualization. The model processes each frame in a video stream, creating accurate 3D face representations optimized for applications like videoconferencing and interactive visualization.

Key Objectives

  • Real-time 3D face reconstruction from 2D images.
  • Efficient transmission of facial data for low-latency applications.
  • Real-time visualization capabilities using WebGL Studio.

Methodology

The project employs a PCA-based 3D model to represent facial configurations efficiently. The AFLW2000-3D dataset and 3DDFA library were used to create 3D point clouds, with a neural network trained using ResNet for dimensionality reduction. The model’s outputs are used to reconstruct the 3D face in real time.

Tools and Technologies

  • Deep Learning Frameworks: PyTorch, TensorFlow, CNNs
  • Visualization Platform: WebGL Studio
  • Techniques Used: Principal Component Analysis (PCA), ResNet architecture for feature extraction, 3D point cloud generation

Disclaimers

  • The AFLW2000-3D dataset, 3DDFA, PCA implementations, and trained models are not included.

License

This project is licensed under the MIT License.


Full Technical Deep-Dive (Interview-Ready)

This section documents every component of the repository for a technical interview, covering structure, algorithms, data interfaces, training/inference pipelines, dependencies, setup, and known issues with recommended fixes.

Repository Map

  • LICENSE (MIT)
  • README.md (overview + deep-dive)
  • Train_Model/
    • FacialDataset.py — custom Dataset that loads images + 25-D target vector
    • FileWalker.py — directory traversal helpers (File/Folder abstraction)
    • TrainModel.py — model initialization options and training loop
    • pycache/ (build artifacts)
  • Use_Model/
    • UseModelExample.py — real-time inference demo (webcam, Haar face detect, WebSocket stream)
    • haarcascade_frontalface_alt2.xml — OpenCV cascade for face detection

Problem Framing

  • Task: Regress a compact 3D face representation (25 coefficients) from a 2D face crop.
  • Output: 25-D vector (e.g., PCA coefficients) used to reconstruct/animate a 3D face in a downstream renderer (WebGL Studio).
  • Modeling: CNN backbone (default ResNet-152 pretrained on ImageNet) with a final Linear layer of size 25; optimized with MSELoss.

Data Expectations and Layout

Training co 8000 de expects a dataset path relative to Train_Model/ as: ../images/<DATASET_NAME>/. Example for data_list=['300W3D-Train']:

images/ 300W3D-Train/ <subject_or_sequence>/ img/ frame_00001.jpg|png ... annot/ frame_00001.pts (text file with floats)

Notes:

  • FacialDataset loads images from img/ and targets from annot/.
  • Label files matched by name containing 'pts'. The code skips header/footer lines typical of .pts files and parses floating point rows into a 25-D vector (np.array(...), column-major flatten).
  • Step parameter can downsample sequences (step=1 loads every frame).

Core Components

  1. Dataset: Train_Model/FacialDataset.py
  • Class: ImageDatasets(torch.utils.data.Dataset)
  • Inputs: data_list (list[str]), dir_gt (override "annot"), blurLevel (unused), onlyFace (unused), step, transform, image_size
  • Returns: (img_tensor, label_tensor_float32)
  • Preprocessing: PIL -> RGB, torchvision transforms
  1. Model initialization: Train_Model/TrainModel.py
  • initialize_model(model_name, num_classes, feature_extract, use_pretrained)
    • Options: resnet152 (default), alexnet, squeezenet1_0, densenet121, inception_v3
    • Replaces classifier head with Linear(..., num_classes)
    • Input size: 224 for most; inception branch sets 229 (see Known Issues)
  • set_parameter_requires_grad(model, feature_extract)
    • Freezes parameters if feature_extract=True
  1. Training loop: Train_Model/TrainModel.py::train_model
  • Criterion: nn.MSELoss()
  • Optimizer: SGD(lr=0.01, momentum=0.9)
  • Batch size: 6; Epochs: 100
  • Device: cuda:0 if available else cpu
  • Augmentations: train transform defined but the dataset is instantiated with the 'val' transform by default (see Known Issues)
  • Saves best weights by lowest loss to netFL.pt
  1. Real-time inference demo: Use_Model/UseModelExample.py
  • Loads ResNet-152 + Linear(25) and state dict from ./netFL.pt
  • Opens webcam, detects face with OpenCV Haar cascade, crops with a small border, pre-processes and infers 25-D vector
  • Sends scaled coefficients over WebSocket to wss://tamats.com:55000/lgraph2 as JSON {"type":0, "channel":1, "data":[...]} for live visualization
  • Draws face rectangle and shows a live window; ESC to exit

Shapes and I/O

  • Input: N x 3 x 224 x 224 float32, normalized to ImageNet stats
  • Output: N x 25 float32 (PCA-like coefficients)
  • Loss: MSE between predicted 25-D vector and ground-truth 25-D vector

Dependencies

  • Python 3.8+ (pyproject not included; virtual env recommended)
  • PyTorch, torchvision
  • Pillow, numpy
  • OpenCV (opencv-python)
  • websocket-client

Environment Setup

# (Windows PowerShell examples)
python -m venv .venv
./.venv/Scripts/Activate.ps1
pip install --upgrade pip
pip install torch torchvision pillow numpy opencv-python websocket-client

Training

cd Train_Model
# Ensure dataset is available at ../images/300W3D-Train/ as described above
python TrainModel.py
# netFL.pt will be saved in Train_Model/ (or the current working dir)

Key training params inside TrainModel.py:

  • model_name = 'resnet' (change to 'alexnet' | 'squeezenet' | 'densenet' | 'inception')
  • num_classes = 25
  • batch_size = 6
  • num_epochs = 100
  • feature_extract = False

Inference (Real-time demo)

cd Use_Model
# Place the trained weights as ./netFL.pt
python UseModelExample.py
# A window opens; press ESC to quit

Known Issues and Recommended Fixes

  1. FileWalker.py
  • walk: uses os.path.isfile(path) instead of os.path.isfile(os.path.join(path, f)) when iterating; this misclassifies entries.
  • File.init: uses self.full_Path (typo) but attribute is full_path.
  • Fix: use os.path.isfile(os.path.join(path, f)) and rename full_Path -> full_path.
  1. FacialDataset.py
  • Imports FileWalker but uses file_walker.walk (undefined variable). Use FileWalker.walk(...) or from FileWalker import walk.
  • blurLevel and onlyFace are currently unused.
  1. TrainModel.py
  • Data transforms: dataset instantiated with 'val' transform; if training, prefer 'train' transform for augmentation.
  • Dataloader shuffle is set to False; consider shuffle=True for training.
  • Inception input size is set to 229; typical Inception v3 uses 299x299.
  • train_model prints references to external "dataloader" variable; better to use local argument consistently.
  • No validation split/metrics; only training loss is tracked.
  1. UseModelExample.py
  • Cropping uses x-10/y-10 which can underflow indices near borders; clamp to image bounds to avoid errors.
  • WebSocket endpoint is external; ensure connectivity or make it configurable.

Extending the Project

  • Add a proper train/val split and evaluation metrics (e.g., MAE/MSE on held-out set).
  • Log training with TensorBoard.
  • Export ONNX/TorchScript for deployment.
  • Replace Haar cascade with a modern face detector (e.g., RetinaFace) for robustness.
  • Consider 3D-aware losses or 3D supervision if available.

Reproducibility Tips

  • Fix random seeds for torch, numpy, and Python.
  • Log exact package versions and GPU info.
  • Save checkpoints and training configs alongside results.

Complete File Tree (Project Root)

RT3D-FaceRecon/
├── LICENSE
├── README.md
├── Train_Model/
│   ├── FacialDataset.py
│   ├── FileWalker.py
│   ├── TrainModel.py
│   └── __pycache__/
│       ├── FacialDataset.cpython-310.pyc
│       └── FileWalker.cpython-310.pyc
└── Use_Model/
    ├── UseModelExample.py
    └── haarcascade_frontalface_alt2.xml

Quick Configuration Summary

  • Training backbone: ResNet-152 (ImageNet-pretrained), configurable via model_name
  • Output dimension: 25
  • Loss: MSELoss
  • Optimizer: SGD(lr=0.01, momentum=0.9)
  • Batch size: 6
  • Epochs: 100
  • Input size: 224x224 (except inception branch in code uses 229; see Known Issues)
  • Normalization: mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]
  • Dataset root: Train_Model/ expects ../images// with img/ and annot/
  • Inference stream: WebSocket to wss://tamats.com:55000/lgraph2, JSON payload with "data" as 25-length float array

About

A deep learning project focused on reconstructing 3D faces from 2D images, with real-time feedback for dynamic visualization. Using frameworks like PyTorch, TensorFlow, and CNNs, this model processes each video frame to produce real-time 3D facial representations.

Resources

Stars

0 stars

Watchers

2 watching

Forks

Releases

Packages

Contributors

Languages

0