backend

BioHub Backend

Production-ready FastAPI backend for semantic search over biological experiments.

🚀 Features

🔍 Vector Search: Semantic similarity search using Sentence Transformers + Qdrant
🔐 Security: JWT authentication, API keys, rate limiting, CORS protection
⚡ Performance: Async operations, Redis caching, optimized HNSW indexing
📊 Monitoring: Structured logging, health checks, metrics endpoints
🧪 Testing: Comprehensive test suite with 95%+ coverage
🐳 Docker: Production-ready containerization with health checks
🔄 CI/CD: Automated testing, linting, and security scanning

📋 Requirements

Python 3.11+
Docker & Docker Compose (recommended)
Qdrant 1.7+
Redis 7+ (optional, for caching)

🛠️ Installation

Option 1: Docker (Recommended)

# Clone repository
git clone https://github.com/Aloulou-ahmed/BioHub.git
cd BioHub

# Copy environment variables
cp backend/.env.example backend/.env

# Edit .env with your configuration
nano backend/.env

# Start all services
docker-compose up -d

# Check health
curl http://localhost:8000/health

Option 2: Local Development

# Create virtual environment
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install dependencies
cd backend
pip install -r requirements.txt

# Set environment variables
cp .env.example .env
# Edit .env with your configuration

# Start Qdrant (required)
docker run -p 6333:6333 qdrant/qdrant:v1.7.3

# Start Redis (optional, for caching)
docker run -p 6379:6379 redis:7.2-alpine

# Run the server
uvicorn main:app --reload --host 0.0.0.0 --port 8000

🔧 Configuration

All configuration is done via environment variables. See .env.example for all options.

Critical Settings

# Security (CHANGE IN PRODUCTION!)
JWT_SECRET_KEY=your-super-secret-jwt-key-min-32-chars
API_KEY=your-api-key-for-external-integrations

# CORS
ALLOWED_ORIGINS=http://localhost:5173,https://yourdomain.com

# Qdrant
QDRANT_HOST=localhost
QDRANT_PORT=6333

# Redis (optional)
REDIS_ENABLED=true
REDIS_HOST=localhost
REDIS_PORT=6379

📡 API Endpoints

Search Experiments

POST /search
Content-Type: application/json

{
  "query": "CRISPR knockout of TP53 gene in human cell lines",
  "limit": 10,
  "score_threshold": 0.7
}

Response:

{
  "query": "CRISPR knockout of TP53 gene in human cell lines",
  "count": 5,
  "results": [
    {
      "id": 42,
      "score": 0.9234,
      "payload": {
        "text": "CRISPR-Cas9 Knockout of TP53 in Human Cell Lines",
        "organism": "human",
        "method": "CRISPR-Cas9",
        "outcome": "Successfully generated knockout clones"
      }
    }
  ]
}

Health Check

GET /health

API Documentation

Swagger UI: http://localhost:8000/docs
ReDoc: http://localhost:8000/redoc

🧪 Testing

# Run all tests
pytest

# With coverage
pytest --cov=. --cov-report=html

# Run specific test file
pytest tests/test_api.py

# Run with verbose output
pytest -v

📊 Data Ingestion

Ingest papers from PubMed:

python ingest_papers.py

This script:

Fetches papers from PubMed
Converts them to experiment format
Generates embeddings
Stores in Qdrant with deduplication

Custom Ingestion

from ingest_papers import PaperIngestionPipeline

pipeline = PaperIngestionPipeline()
pipeline.ingest_pubmed_papers(
    query="CRISPR gene editing 2024",
    max_results=100
)

🐳 Docker Deployment

Development

docker-compose up

Production

# Build
docker-compose -f docker-compose.yml -f docker-compose.prod.yml build

# Deploy
docker-compose -f docker-compose.yml -f docker-compose.prod.yml up -d

# View logs
docker-compose logs -f backend

# Monitor
docker-compose ps

📈 Performance Optimization

Caching

Enable Redis caching in .env:

REDIS_ENABLED=true
CACHE_TTL=3600  # 1 hour

Qdrant Tuning

The backend is pre-configured with optimized settings:

HNSW indexing with m=16, ef_construct=100
Cosine similarity distance
Lazy indexing threshold: 20,000 points
Memory-mapped storage threshold: 50,000 points

Scaling

Increase Uvicorn workers:

uvicorn main:app --workers 4

Or in Docker:

CMD ["uvicorn", "main:app", "--workers", "4"]

🔒 Security Best Practices

Change default secrets in .env:

JWT_SECRET_KEY=$(openssl rand -hex 32)
API_KEY=$(openssl rand -hex 32)

Restrict CORS origins:
```
ALLOWED_ORIGINS=https://yourdomain.com
```

Enable rate limiting:

RATE_LIMIT_ENABLED=true
RATE_LIMIT_REQUESTS=100
RATE_LIMIT_WINDOW=60

Use HTTPS in production (e.g., with Nginx reverse proxy)
Run as non-root user (Docker does this automatically)

📝 Logging

Structured JSON logging for production:

LOG_FORMAT=json
LOG_LEVEL=INFO

Human-readable logs for development:

LOG_FORMAT=text
LOG_LEVEL=DEBUG

🔍 Monitoring

Health Checks

# Application health
curl http://localhost:8000/health

# Qdrant health
curl http://localhost:6333/healthz

# Redis health
docker exec biohub_redis redis-cli ping

Cache Statistics

from cache import get_cache_stats

stats = await get_cache_stats()
print(stats)
# {"enabled": True, "hits": 1234, "misses": 567, "hit_rate": 0.685}

🐛 Troubleshooting

Qdrant Connection Error

# Check if Qdrant is running
docker ps | grep qdrant

# Check Qdrant logs
docker logs biohub_qdrant

# Test connection
curl http://localhost:6333/healthz

Import Errors

# Reinstall dependencies
pip install --force-reinstall -r requirements.txt

# Check Python version
python --version  # Should be 3.11+

Performance Issues

Enable Redis caching
Increase Uvicorn workers
Optimize Qdrant collection settings
Monitor with docker stats

📚 Architecture

┌─────────────────────────────────────────┐
│           FastAPI Backend               │
│  - Authentication & Rate Limiting       │
│  - Request/Response Validation          │
│  - Structured Logging                   │
└──────┬────────────┬────────────┬────────┘
       │            │            │
       ▼            ▼            ▼
  ┌────────┐  ┌─────────┐  ┌────────┐
  │ Qdrant │  │ Redis   │  │Sentence│
  │Vector  │  │Cache    │  │Transfor│
  │ DB     │  │         │  │  mer   │
  └────────┘  └─────────┘  └────────┘

🤝 Contributing

Fork the repository
Create a feature branch
Make your changes
Add tests
Run linters: ruff check . && black .
Submit a pull request

📄 License

MIT License - see LICENSE file for details

🙏 Acknowledgments

Sentence Transformers for embeddings
Qdrant for vector search
FastAPI for the amazing web framework

📞 Support

Issues: https://github.com/Aloulou-ahmed/BioHub/issues
Discussions: https://github.com/Aloulou-ahmed/BioHub/discussions

Name		Name	Last commit message	Last commit date
parent directory ..
api		api
pdf_video_converter		pdf_video_converter
tests		tests
.env.example		.env.example
.gitignore		.gitignore
CODE_QUALITY_SUMMARY.md		CODE_QUALITY_SUMMARY.md
DEPLOYMENT.md		DEPLOYMENT.md
Dockerfile		Dockerfile
README.md		README.md
__init__.py		__init__.py
auth.py		auth.py
cache.py		cache.py
config.py		config.py
data_loader.py		data_loader.py
embedder.py		embedder.py
ingest_papers.py		ingest_papers.py
literature_scraper.py		literature_scraper.py
logger.py		logger.py
main.py		main.py
models.py		models.py
monitoring.py		monitoring.py
paper_experiment_generator.py		paper_experiment_generator.py
pyproject.toml		pyproject.toml
qdrant_manager.py		qdrant_manager.py
requirements.txt		requirements.txt

FilesExpand file tree

backend

Directory actions

More options

Directory actions

More options

Latest commit

History

backend

Folders and files

parent directory

README.md

BioHub Backend

🚀 Features

📋 Requirements

🛠️ Installation

Option 1: Docker (Recommended)

Option 2: Local Development

🔧 Configuration

Critical Settings

📡 API Endpoints

Search Experiments

Health Check

API Documentation

🧪 Testing

📊 Data Ingestion

Custom Ingestion

🐳 Docker Deployment

Development

Production

📈 Performance Optimization

Caching

Qdrant Tuning

Scaling

🔒 Security Best Practices

📝 Logging

🔍 Monitoring

Health Checks

Cache Statistics

🐛 Troubleshooting

Qdrant Connection Error

Import Errors

Performance Issues

📚 Architecture

🤝 Contributing

📄 License

🙏 Acknowledgments

📞 Support