8000 GitHub - jubelAhmed/file-vault-dedupe: A full-stack file management system with intelligent deduplication, content search, and storage quota management. Built with Django REST Framework and React. Β· GitHub
[go: up one dir, main page]

Skip to content

About

A full-stack file management system with intelligent deduplication, content search, and storage quota management. Built with Django REST Framework and React.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

Β 

History

24 Commits

Folders and files

NameName
Last commit message
Last commit date
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 

Repository files navigation

File Vault Dedupe

A production-ready file management platform featuring intelligent cross-user deduplication, content-based search, and comprehensive storage management. Built with Django REST Framework backend and React TypeScript frontend.

Overview

File Vault Dedupe solves the problem of storage waste by automatically detecting duplicate files across all users. When multiple users upload the same file, only one copy is stored on disk while all users maintain access. The system also provides powerful content-based search capabilities, allowing users to find files by searching within their content, not just filenames.

Core Features

πŸ”„ Intelligent Deduplication

  • SHA-256 Hash-Based Detection: Identifies duplicate files with cryptographic precision
  • Cross-User Deduplication: Shares storage across all users while maintaining data isolation
  • Reference System: Creates lightweight references instead of storing duplicate files
  • Automatic Cleanup: Safely removes files only when all references are deleted

πŸ” Content-Based Search

  • Keyword Extraction: Automatically extracts searchable keywords from file content
  • Multi-Format Support: Indexes text from PDF, DOCX, XLSX, PPTX, TXT, CSV, JSON, XML, and more
  • Fast Search: Query files by keywords found within their content
  • User-Isolated Results: Search results respect user boundaries

πŸ’Ύ Storage Management

  • Per-User Quotas: Configurable storage limits per user
  • Real-Time Tracking: Monitor storage usage and savings
  • Deduplication Statistics: View system-wide storage savings
  • Quota Enforcement: Prevents uploads that exceed limits

⚑ Asynchronous Processing

  • Celery Integration: Background file indexing for non-blocking uploads
  • Redis Message Broker: Reliable task queue management
  • Content Extraction: Automatic text extraction and keyword indexing
  • Search Index Maintenance: Automatic cleanup on file deletion

πŸ”’ Security & Performance

  • File Validation: Size limits, extension checking, and content-type verification
  • Rate Limiting: Per-user API rate limiting to prevent abuse
  • User Isolation: Complete data separation between users
  • Database Indexing: Optimized queries for fast performance

Technology Stack

Backend:

  • Django 4.x with Django REST Framework
  • Celery for asynchronous task processing
  • Redis for message brokering and caching
  • SQLite (development) / PostgreSQL (production-ready)
  • Gunicorn for production deployment

Frontend:

  • React 18 with TypeScript
  • TanStack Query for efficient data fetching
  • Axios for HTTP requests
  • Tailwind CSS for modern UI

Infrastructure:

  • Docker & Docker Compose for containerization
  • GitHub Actions for CI/CD
  • WhiteNoise for static file serving

Quick Start

Prerequisites

  • Docker 20.10+ and Docker Compose 2.x+
  • (Optional) Python 3.12+ and Node.js 18+ for local development

Docker Setup (Recommended)

# Clone the repository
git clone <repository-url>
cd file-vault-dedupe

# Start all services
docker-compose up --build

This starts:

  • Redis (port 6379) - Message broker for Celery
  • Backend API (port 8000) - Django REST API
  • Celery Worker - Background task processor
  • Frontend (port 3000) - React application

Access the application:

Local Development

Backend Setup

cd backend

# Create virtual environment
python -m venv venv
source venv/bin/activate  # Windows: venv\Scripts\activate

# Install production dependencies
pip install -r requirements.txt

# Install development dependencies (linting, type checking)
pip install -r requirements-dev.txt

# Create environment file (.env)
cat > .env << EOF
DJANGO_SECRET_KEY=your-secret-key-here
MAX_CALLS=10
TIME_WINDOW=1
STORAGE_QUOTA_PER_USER=10485760
CELERY_BROKER_URL=redis://localhost:6379/0
CELERY_RESULT_BACKEND=redis://localhost:6379/0
EOF

# Run migrations
python manage.py migrate

# Setup Git hooks (optional but recommended)
# This installs pre-push hooks that run linting checks automatically
./scripts/setup-hooks.sh

# Start development server
python manage.py runserver

Frontend Setup

cd frontend

# Install dependencies
npm install

# Create environment file (.env.local)
echo "REACT_APP_API_URL=http://localhost:8000/api" > .env.local

# Start development server
npm start

API Usage

Authentication

All API requests require a UserId header for user identification:

curl -H "UserId: user123" http://localhost:8000/api/files/

Key Endpoints

File Operations:

  • GET /api/files/ - List files (supports filtering and search)
  • POST /api/files/ - Upload file
  • GET /api/files/{id}/ - Get file details
  • DELETE /api/files/{id}/ - Delete file

Search: 8000

  • GET /api/files/search/?keyword=contract - Search files by content keyword
  • GET /api/files/search/?keywords=contract,agreement - Multi-keyword search

Statistics:

  • GET /api/files/storage_stats/ - User storage statistics
  • GET /api/files/deduplication_stats/ - System-wide deduplication stats
  • GET /api/files/index_stats/ - Search index statistics

Filtering:

  • GET /api/files/?search=filename - Search by filename
  • GET /api/files/?file_type=application/pdf - Filter by file type
  • GET /api/files/?min_size=1000&max_size=5000 - Filter by size range
  • GET /api/files/?start=2024-01-01&end=2024-12-31 - Filter by date range

See API Documentation for complete reference.

Testing

cd backend

# Run all tests
python manage.py test

# Run with coverage
coverage run --source='.' manage.py test
coverage report
coverage html

# Run specific test suite
python manage.py test files.tests.test_api
python manage.py test files.tests.test_services

Test coverage: 82% (72 tests covering core functionality)

Code Quality & Linting

The project uses Ruff for formatting, import sorting, and linting, plus MyPy for type checking.

cd backend

# Check code quality (formatting, imports, linting, types)
./lint.sh check

# Auto-fix issues (formatting, imports, linting)
./lint.sh fix

Development Dependencies:

  • ruff - Fast linter, formatter, and import sorter (replaces Black + isort)
  • mypy - Static type checker

Install with: pip install -r requirements-dev.txt

Git Hooks (Pre-Push Linting)

The project includes a pre-push Git hook that automatically runs linting checks before pushing to GitHub. This prevents pushing code that doesn't pass quality checks.

Setup:

# Install Git hooks (run once after cloning)
./scripts/setup-hooks.sh

After setup, the hook will automatically run ./lint.sh check before every git push. If linting fails, the push will be blocked. You can bypass it with git push --no-verify (not recommended).

Project Structure

file-vault-dedupe/
β”œβ”€β”€ backend/
β”‚   β”œβ”€β”€ core/                    # Django project settings
β”‚   β”‚   β”œβ”€β”€ settings.py           # Configuration
β”‚   β”‚   β”œβ”€β”€ celery.py             # Celery setup
β”‚   β”‚   └── middleware/           # Custom middleware
β”‚   β”œβ”€β”€ files/                    # Main application
β”‚   β”‚   β”œβ”€β”€ models.py             # Data models (File, UserStorage, FileSearchIndex)
β”‚   β”‚   β”œβ”€β”€ views.py              # API endpoints
β”‚   β”‚   β”œβ”€β”€ serializers.py        # Data serialization
β”‚   β”‚   β”œβ”€β”€ services/             # Business logic
β”‚   β”‚   β”‚   β”œβ”€β”€ deduplication_service.py
β”‚   β”‚   β”‚   β”œβ”€β”€ search_service.py
β”‚   β”‚   β”‚   β”œβ”€β”€ storage_service.py
β”‚   β”‚   β”‚   └── content_extraction_service.py
β”‚   β”‚   β”œβ”€β”€ tasks.py              # Celery tasks
β”‚   β”‚   └── tests/                # Test suite
β”‚   β”œβ”€β”€ requirements.txt          # Production dependencies
β”‚   β”œβ”€β”€ requirements-dev.txt      # Development dependencies (linting, type checking)
β”‚   └── lint.sh                   # Code quality script
β”œβ”€β”€ frontend/
β”‚   β”œβ”€β”€ src/
β”‚   β”‚   β”œβ”€β”€ components/            # React components
β”‚   β”‚   β”œβ”€β”€ services/             # API client
β”‚   β”‚   └── types/                # TypeScript definitions
β”‚   └── package.json
β”œβ”€β”€ postman/                       # API testing collections
β”œβ”€β”€ .github/workflows/             # CI/CD pipelines
└── docker-compose.yml             # Container orchestration

Configuration

Environment Variables

Backend (.env):

  • DJANGO_SECRET_KEY - Django secret key (required)
  • MAX_CALLS - Rate limit: max requests per window (default: 10)
  • TIME_WINDOW - Rate limit: time window in seconds (default: 1)
  • STORAGE_QUOTA_PER_USER - Storage quota in bytes (default: 10MB)
  • CELERY_BROKER_URL - Redis connection URL
  • CELERY_RESULT_BACKEND - Redis result backend URL

Frontend (.env.local):

  • REACT_APP_API_URL - Backend API URL

Supported File Types for Content Search

The system can extract and index text content from:

  • Documents: PDF, DOC, DOCX, ODT
  • Spreadsheets: XLS, XLSX, ODS, CSV
  • Presentations: PPT, PPTX, ODP
  • Text Files: TXT, XML, HTML, RTF, JSON, YAML

Files are automatically indexed in the background after upload.

CI/CD

GitHub Actions workflow automatically:

  • Runs tests on push/PR
  • Generates coverage reports
  • Validates code quality

See .github/workflows/backend.yml for details.

Documentation

License

This project is open source and available under the MIT License.

Author

Jubel Ahmed


Built with ❀️ using Django, React, and modern DevOps practices.

About

A full-stack file management system with intelligent deduplication, content search, and storage quota management. Built with Django REST Framework and React.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages

0