Repository navigation
Future Enhancement: Offload image processing to AWS Lambda #53
Description
Overview
Currently, image processing for BAP submissions happens on the Express server using Sharp. For better scalability and reduced server load, we should consider offloading this to AWS Lambda functions.
Current Implementation
- Images uploaded to Express server via multer
- Processed synchronously with Sharp (resize, format conversion, EXIF stripping)
- Multiple sizes generated (thumbnail, medium, original)
- Uploaded to Cloudflare R2
- ~2-5 seconds processing time per image
Proposed Lambda Architecture
Option 1: Direct Lambda Processing
Client → API Gateway → Lambda → R2
↓
Progress via WebSocket/SSE
Option 2: Queue-Based Processing (Recommended)
Client → Express → S3 Raw Bucket → Lambda (via S3 event) → R2
↓ ↓
SQS Queue Process & Store
↓
Progress Updates
Implementation Details
Lambda Function
// image-processor-lambda.js
exports.handler = async (event) => {
// 1. Download raw image from S3
// 2. Process with Sharp Lambda Layer
// 3. Generate multiple sizes
// 4. Upload to R2
// 5. Update database via API
// 6. Send completion notification
};Benefits
- Scalability: Auto-scales with demand
- Cost-effective: Pay only for processing time
- Server relief: Reduces server CPU/memory by 90%
- Parallel processing: Multiple images processed simultaneously
- Resilience: Built-in retry mechanisms
Challenges to Consider
- Cold starts: ~1-2s delay for first invocation
- Progress tracking: Need WebSocket or polling
- Error handling: More complex retry logic
- Local development: Harder to test locally
Migration Path
Phase 1: Hybrid Approach
- Keep current server processing as fallback
- Add feature flag for Lambda processing
- Test with subset of users
Phase 2: Full Migration
- Move all processing to Lambda
- Keep server endpoint for upload coordination only
- Implement comprehensive monitoring
Technical Requirements
AWS Resources Needed
- Lambda function with Sharp layer
- S3 bucket for raw uploads
- SQS queue for reliability
- IAM roles and policies
- CloudWatch for monitoring
Estimated Costs (100 images/day)
- Lambda: ~$0.50/month
- S3 temporary storage: ~$0.10/month
- SQS: ~$0.05/month
- Total: < $1/month
Sharp Lambda Layer
Use pre-built layer: arn:aws:lambda:us-east-1:123456789012:layer:sharp:1
Or build custom layer with specific Sharp version
Success Metrics
- Processing time < 3s per image
- Server CPU usage reduced by 90%
- Error rate < 0.1%
- Cost < $5/month at expected volume
References
Priority: Low (current solution works fine for expected traffic)
Effort: Medium (2-3 days to implement and test)
Activity
feat: add image upload functionality to BAP submissions
feat: add image upload functionality to BAP submissions
jra3 commentedon Oct 7, 2025
Consolidated into master issue #115. See #115 for comprehensive tracking of all image upload enhancements.