Secure RAG system with PII anonymization and decoy generation for medical data protection
DSI-RAG is an advanced RAG (Retrieval-Augmented Generation) system designed to protect sensitive medical data using:
- 🎭 PII Anonymization: Dual-layer detection (Presidio + GLiNER)
- 🎲 Decoy Generation: Creates fake chunks to protect real data
- 🔐 Secure Vault: Local storage for PII mappings and labels
- 💾 Vector Store: Cloud-simulated storage (ChromaDB)
- 🔓 De-anonymization: Restores PII in final answers
INGESTION PIPELINE
------------------
Document → Extract → Anonymize (Presidio + GLiNER) → Chunk
↓
Vault (PII Mapping)
↓
Generate Decoys (Groq) → Mix Real + Decoys → Embed → Vector Store
QUERY PIPELINE
--------------
Question → Semantic Search → Filter Decoys (Vault) → Generate Answer (Groq)
↓
De-anonymize → Final Answer
- ✅ Dual PII Detection: Presidio (regex) + GLiNER (ML-based)
- ✅ Decoy Protection: Fake chunks mixed with real data
- ✅ Local Vault: Trusted zone for sensitive mappings
- ✅ Zero-knowledge Vector Store: No labels exposed
- ⚡ Fast Generation: Groq API for decoys and answers
- 🔄 Parallel Processing: Multi-threaded decoy generation
- 📊 Performance Metrics: Built-in timing indicators
- 🐍 Modern Python: Type hints, async/await
- 📝 Comprehensive Logging: Debug-friendly
- 🧪 Test Pipeline: End-to-end testing script
- 📚 Full Documentation: Multiple guides
- Python 3.10+
- Groq API account (free at console.groq.com)
# Clone the repository
git clone https://github.com/Aloulou-ahmed/dsi_rag.git
cd dsi_rag
# Create virtual environment
python -m venv venv
.\venv\Scripts\activate # Windows
# source venv/bin/activate # Linux/Mac
# Install dependencies
pip install -r requirements.txt
# Download spaCy model
python -m spacy download en_core_web_lg# Copy environment template
copy .env.example .env # Windows
# cp .env.example .env # Linux/Mac
# Edit .env and add your Groq API key
# Get your key at: https://console.groq.com
GROQ_API_KEY=your_actual_key_herepython verify_setup.pypython test_pipeline.py Patient.pdfTested on Patient.pdf (2524 characters):
| Stage | Time | % |
|---|---|---|
| Extraction | 0.16s | 0.6% |
| Anonymization | 19.19s | 73.8% |
| Chunking | 0.00s | 0.0% |
| Decoys (Groq) | 4.35s | 16.8% |
| Vault Storage | 0.09s | 0.3% |
| Embedding | 0.65s | 2.5% |
| Retrieval | 0.03s | 0.1% |
| Answer (Groq) | 1.51s | 5.8% |
| TOTAL | 26.0s | 100% |
POST /documents/upload
Content-Type: multipart/form-data
Response:
{
"doc_id": "abc123",
"pii_count": 41,
"total_chunks": 14,
"real_chunks": 7,
"decoy_chunks": 7,
"processing_time": 26.0
}POST /chat
Content-Type: application/json
{
"doc_id": "abc123",
"question": "What is the diagnosis?"
}
Response:
{
"answer": "The patient has been diagnosed with...",
"sources": [...],
"retrieved_total": 8,
"decoys_filtered": 7
}GET /healthbackend/
├── app/
│ ├── api/ # API endpoints
│ ├── core/ # Core business logic
│ │ ├── anonymizer.py # PII detection
│ │ ├── decoy_generator.py # Groq-based decoys
│ │ ├── rag_chain.py # Answer generation
│ │ ├── retriever.py # Filtered retrieval
│ │ ├── vector_store.py # ChromaDB interface
│ │ └── vault.py # Secure storage
│ ├── models/ # Pydantic models
│ ├── utils/ # Utilities
│ ├── config.py # Configuration
│ └── main.py # FastAPI app
├── data/ # Data storage (gitignored)
├── test_pipeline.py # E2E test script
├── verify_setup.py # Setup verification
└── requirements.txt # Dependencies
- QUICK_START_V2.md - 5-minute guide
- INSTALLATION_GUIDE.md - Complete setup
- CHANGELOG_V2.md - Version history
- ANALYSE_RAG_BACKEND.md - Technical analysis
- Encrypt Vault: Use SQLCipher
- Add Authentication: JWT or API keys
- Enable HTTPS: Reverse proxy with SSL
- Validate Uploads: File type, size limits
- Rate Limiting: Prevent abuse
See ANALYSE_RAG_BACKEND.md for security details.
# Full pipeline test
python test_pipeline.py your_document.pdf
# Verify installation
python verify_setup.py
# Run API server
uvicorn app.main:app --reload
# API documentation
http://localhost:8000/docs- FastAPI - REST API framework
- Groq API - LLM inference (decoys + answers)
- Presidio - PII detection (regex)
- GLiNER - PII detection (ML)
- ChromaDB - Vector database
- Sentence Transformers - Embeddings
- LangChain - Text processing
- 🔐 Vault encryption (SQLCipher)
- 🔑 JWT authentication
- 🧪 Unit tests (pytest)
- 🐳 Docker deployment
- 📊 Monitoring dashboard
- 🌐 Multi-language support
Contributions are welcome! Please:
- Read the documentation
- Follow existing code style
- Add tests for new features
- Update documentation
Proprietary - Internal use only
- Project Lead: Medical AI Security Team
- RAG Architecture: DSI Team
- Version 2.0: Groq API Integration
For issues or questions:
- Check INSTALLATION_GUIDE.md
- Review PROBLEMS_SUMMARY.md
- Examine logs in
data/logs/ - Create an issue on GitHub
Built with ❤️ to protect sensitive medical data
Version: 2.0
Status: Production Ready (with security hardening)
Last Updated: January 2026