A production-ready file management platform featuring intelligent cross-user deduplication, content-based search, and comprehensive storage management. Built with Django REST Framework backend and React TypeScript frontend.
File Vault Dedupe solves the problem of storage waste by automatically detecting duplicate files across all users. When multiple users upload the same file, only one copy is stored on disk while all users maintain access. The system also provides powerful content-based search capabilities, allowing users to find files by searching within their content, not just filenames.
- SHA-256 Hash-Based Detection: Identifies duplicate files with cryptographic precision
- Cross-User Deduplication: Shares storage across all users while maintaining data isolation
- Reference System: Creates lightweight references instead of storing duplicate files
- Automatic Cleanup: Safely removes files only when all references are deleted
- Keyword Extraction: Automatically extracts searchable keywords from file content
- Multi-Format Support: Indexes text from PDF, DOCX, XLSX, PPTX, TXT, CSV, JSON, XML, and more
- Fast Search: Query files by keywords found within their content
- User-Isolated Results: Search results respect user boundaries
- Per-User Quotas: Configurable storage limits per user
- Real-Time Tracking: Monitor storage usage and savings
- Deduplication Statistics: View system-wide storage savings
- Quota Enforcement: Prevents uploads that exceed limits
- Celery Integration: Background file indexing for non-blocking uploads
- Redis Message Broker: Reliable task queue management
- Content Extraction: Automatic text extraction and keyword indexing
- Search Index Maintenance: Automatic cleanup on file deletion
- File Validation: Size limits, extension checking, and content-type verification
- Rate Limiting: Per-user API rate limiting to prevent abuse
- User Isolation: Complete data separation between users
- Database Indexing: Optimized queries for fast performance
Backend:
- Django 4.x with Django REST Framework
- Celery for asynchronous task processing
- Redis for message brokering and caching
- SQLite (development) / PostgreSQL (production-ready)
- Gunicorn for production deployment
Frontend:
- React 18 with TypeScript
- TanStack Query for efficient data fetching
- Axios for HTTP requests
- Tailwind CSS for modern UI
Infrastructure:
- Docker & Docker Compose for containerization
- GitHub Actions for CI/CD
- WhiteNoise for static file serving
- Docker 20.10+ and Docker Compose 2.x+
- (Optional) Python 3.12+ and Node.js 18+ for local development
# Clone the repository
git clone <repository-url>
cd file-vault-dedupe
# Start all services
docker-compose up --buildThis starts:
- Redis (port 6379) - Message broker for Celery
- Backend API (port 8000) - Django REST API
- Celery Worker - Background task processor
- Frontend (port 3000) - React application
Access the application:
- Frontend: http://localhost:3000
- Backend API: http://localhost:8000/api
- Health Check: http://localhost:8000/health/
cd backend
# Create virtual environment
python -m venv venv
source venv/bin/activate # Windows: venv\Scripts\activate
# Install production dependencies
pip install -r requirements.txt
# Install development dependencies (linting, type checking)
pip install -r requirements-dev.txt
# Create environment file (.env)
cat > .env << EOF
DJANGO_SECRET_KEY=your-secret-key-here
MAX_CALLS=10
TIME_WINDOW=1
STORAGE_QUOTA_PER_USER=10485760
CELERY_BROKER_URL=redis://localhost:6379/0
CELERY_RESULT_BACKEND=redis://localhost:6379/0
EOF
# Run migrations
python manage.py migrate
# Setup Git hooks (optional but recommended)
# This installs pre-push hooks that run linting checks automatically
./scripts/setup-hooks.sh
# Start development server
python manage.py runservercd frontend
# Install dependencies
npm install
# Create environment file (.env.local)
echo "REACT_APP_API_URL=http://localhost:8000/api" > .env.local
# Start development server
npm startAll API requests require a UserId header for user identification:
curl -H "UserId: user123" http://localhost:8000/api/files/File Operations:
GET /api/files/- List files (supports filtering and search)POST /api/files/- Upload fileGET /api/files/{id}/- Get file detailsDELETE /api/files/{id}/- Delete file
Search: 8000
GET /api/files/search/?keyword=contract- Search files by content keywordGET /api/files/search/?keywords=contract,agreement- Multi-keyword search
Statistics:
GET /api/files/storage_stats/- User storage statisticsGET /api/files/deduplication_stats/- System-wide deduplication statsGET /api/files/index_stats/- Search index statistics
Filtering:
GET /api/files/?search=filename- Search by filenameGET /api/files/?file_type=application/pdf- Filter by file typeGET /api/files/?min_size=1000&max_size=5000- Filter by size rangeGET /api/files/?start=2024-01-01&end=2024-12-31- Filter by date range
See API Documentation for complete reference.
cd backend
# Run all tests
python manage.py test
# Run with coverage
coverage run --source='.' manage.py test
coverage report
coverage html
# Run specific test suite
python manage.py test files.tests.test_api
python manage.py test files.tests.test_servicesTest coverage: 82% (72 tests covering core functionality)
The project uses Ruff for formatting, import sorting, and linting, plus MyPy for type checking.
cd backend
# Check code quality (formatting, imports, linting, types)
./lint.sh check
# Auto-fix issues (formatting, imports, linting)
./lint.sh fixDevelopment Dependencies:
ruff- Fast linter, formatter, and import sorter (replaces Black + isort)mypy- Static type checker
Install with: pip install -r requirements-dev.txt
The project includes a pre-push Git hook that automatically runs linting checks before pushing to GitHub. This prevents pushing code that doesn't pass quality checks.
Setup:
# Install Git hooks (run once after cloning)
./scripts/setup-hooks.shAfter setup, the hook will automatically run ./lint.sh check before every git push. If linting fails, the push will be blocked. You can bypass it with git push --no-verify (not recommended).
file-vault-dedupe/
βββ backend/
β βββ core/ # Django project settings
β β βββ settings.py # Configuration
β β βββ celery.py # Celery setup
β β βββ middleware/ # Custom middleware
β βββ files/ # Main application
β β βββ models.py # Data models (File, UserStorage, FileSearchIndex)
β β βββ views.py # API endpoints
β β βββ serializers.py # Data serialization
β β βββ services/ # Business logic
β β β βββ deduplication_service.py
β β β βββ search_service.py
β β β βββ storage_service.py
β β β βββ content_extraction_service.py
β β βββ tasks.py # Celery tasks
β β βββ tests/ # Test suite
β βββ requirements.txt # Production dependencies
β βββ requirements-dev.txt # Development dependencies (linting, type checking)
β βββ lint.sh # Code quality script
βββ frontend/
β βββ src/
β β βββ components/ # React components
β β βββ services/ # API client
β β βββ types/ # TypeScript definitions
β βββ package.json
βββ postman/ # API testing collections
βββ .github/workflows/ # CI/CD pipelines
βββ docker-compose.yml # Container orchestration
Backend (.env):
DJANGO_SECRET_KEY- Django secret key (required)MAX_CALLS- Rate limit: max requests per window (default: 10)TIME_WINDOW- Rate limit: time window in seconds (default: 1)STORAGE_QUOTA_PER_USER- Storage quota in bytes (default: 10MB)CELERY_BROKER_URL- Redis connection URLCELERY_RESULT_BACKEND- Redis result backend URL
Frontend (.env.local):
REACT_APP_API_URL- Backend API URL
The system can extract and index text content from:
- Documents: PDF, DOC, DOCX, ODT
- Spreadsheets: XLS, XLSX, ODS, CSV
- Presentations: PPT, PPTX, ODP
- Text Files: TXT, XML, HTML, RTF, JSON, YAML
Files are automatically indexed in the background after upload.
GitHub Actions workflow automatically:
- Runs tests on push/PR
- Generates coverage reports
- Validates code quality
See .github/workflows/backend.yml for details.
This project is open source and available under the MIT License.
Jubel Ahmed
Built with β€οΈ using Django, React, and modern DevOps practices.