Repository navigation
Incident: HTTP 5xx due to OutOfMemoryException in Cart API (Grubify Container App) #6
Description
Incident Report: HTTP 5xx due to OutOfMemoryException in Cart API
- Incident ID:
1ad8abaf-cf41-4c34-b710-6a826eaff000 - Service: Azure Container Apps —
ca-grubify-uybgeqfrc2owo(rg:rg-sre-lab) - Subscription:
33ccd01f-fc97-4fa1-b208-db8eb99a430b - FQDN:
ca-grubify-uybgeqfrc2owo.ashymushroom-a36a19ce.eastus2.azurecontainerapps.io - Active revision:
ca-grubify-uybgeqfrc2owo--0000005(100% traffic)
Summary
The Grubify API container app began returning HTTP 5xx errors due to System.OutOfMemoryException caused by an unbounded in-memory cache in CartController.cs. Every POST /api/cart/{userId}/items request allocates a 10MB byte[] that is appended to a static List<byte[]> (RequestDataCache) and never freed. After 75 add-to-cart requests in ~2 minutes, the cache consumed 750MB, exceeding the container's memory limit and causing widespread OOM failures across all API endpoints.
Impact
- All API endpoints returned HTTP 500 errors during the OOM window (~08:08–08:14 UTC), affecting all users
- Cart operations failed entirely — users could not add items, view carts, or place orders
- Peak error rate: ~53 OOM exceptions per minute at 08:09–08:10 UTC
- Blast radius: 100% of API traffic on the single active revision
Timeline (UTC)
- ~07:52: Revision
ca-grubify-uybgeqfrc2owo--0000005deployed and started receiving traffic - ~08:06:51: First cache entry logged —
Cache size: 10MB(1 add-to-cart request) - ~08:07–08:08: Rapid burst of add-to-cart requests — cache grew from 10MB to 750MB in ~2 minutes (75 requests)
- ~08:08: First
System.OutOfMemoryExceptionerrors appear in Kestrel logs (8 OOM/min) - ~08:09–08:10: Peak OOM errors — 52-53 exceptions per minute, ~1000 total log lines/min
- ~08:12:42: Azure Monitor alert
alert-http-5xx-sre-labfired (Sev3) - ~08:14: OOM errors continue but at reduced rate (3/min) — possible partial recovery via GC pressure
Evidence
Console logs (active revision)
Memory leak cache growth (sampled):
2026-09-28T08:06:51Z Analytics cache: Added request data. Total entries: 1
2026-09-28T08:06:51Z Cache size: 10MB
2026-09-28T08:07:38Z Cache size: 100MB
2026-09-28T08:08:00Z Cache size: 300MB
2026-09-28T08:08:22Z Cache size: 500MB
2026-09-28T08:08:50Z Analytics cache: Added request data. Total entries: 75
2026-09-28T08:08:50Z Cache size: 750MB
OOM exceptions (sampled):
2026-09-28T08:13:19Z fail: Microsoft.AspNetCore.Server.Kestrel[13]
Connection id "0HNOT6LJC9ION", Request id "0HNOT6LJC9ION:0000002C": An unhandled exception was thrown by the application.
System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
2026-09-28T08:14:22Z fail: Microsoft.AspNetCore.Server.Kestrel[13]
Connection id "0HNOT6LJC9IRQ", Request id "0HNOT6LJC9IRQ:00000005": An unhandled exception was thrown by the application.
System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
Traffic and Error Timeline
| Time (UTC) | Total Logs/min | OOM Errors/min | Cache Logs/min |
|---|---|---|---|
| 07:52 | 12 | 0 | 0 |
| 08:06 | 2 | 0 | 1 |
| 08:07 | 56 | 0 | 28 |
| 08:08 | 244 | 8 | 46 |
| 08:09 | 988 | 52 | 0 |
| 08:10 | 1007 | 53 | 0 |
| 08:11 | 266 | 14 | 0 |
| 08:13 | 279 | 15 | 0 |
| 08:14 | 55 | 3 | 0 |
Metrics snapshot (Azure Monitor / Log Analytics)
- Cache growth rate: 10MB → 750MB in ~2 minutes (75 add-to-cart requests)
- OOM exceptions total: ~160+ in the investigation window
- Container memory limit: 1Gi (1024MB) — cache alone consumed 750MB leaving insufficient memory for the runtime
- RestartCount: Container remained in
Runningstate (no OOM kill restart observed yet, but imminent)
Root Cause
Unbounded in-memory cache in CartController.cs (lines 30-31). The RequestDataCache is a static List<byte[]> that allocates a 10MB buffer on every POST /api/cart/{userId}/items call and never evicts entries. A burst of 75 add-to-cart requests within ~2 minutes caused the cache to consume 750MB, exhausting the container's 1Gi memory limit and triggering System.OutOfMemoryException across all request-handling threads in Kestrel.
Code path:
// Controllers/CartController.cs, lines 30-31
var requestData = new byte[10 * 1024 * 1024]; // 10MB buffer for request analytics
RequestDataCache.Add(requestData);
// TODO: Implement cache cleanup mechanism in future sprint <-- never implementedRemediation
- Code: Remove the unbounded
RequestDataCacheentirely, or replace it with a boundedConcurrentQueue<byte[]>with a max entry limit and TTL-based eviction. The 10MB allocation per request is unnecessary for analytics — use a lightweight counter or structured log instead. - Defensive: Add a maximum cache size check before allocation (e.g., reject if cache exceeds 100MB). Implement rate limiting on the
POST /api/cart/{userId}/itemsendpoint. - Platform: Increase container memory limit from 1Gi to 2Gi as a short-term buffer. Add memory-based autoscaling rule (scale out at 75% memory utilization).
- Observability: Add an Azure Monitor alert for container memory percentage > 80% sustained over 5 minutes. Instrument Application Insights SDK for proper distributed tracing and exception tracking.
Action Items
| # | Action | Priority |
|---|---|---|
| 1 | Remove or replace unbounded RequestDataCache in CartController.cs with bounded alternative |
High |
| 2 | Add memory limit check before allocation in AddItemToCart |
High |
| 3 | Increase container memory from 1Gi to 2Gi as short-term mitigation | Medium |
| 4 | Add memory-percentage alert (>80% sustained 5min) in Azure Monitor | Medium |
| 5 | Wire Application Insights SDK into API for exception tracking and distributed tracing | Medium |
| 6 | Add rate limiting on cart add-item endpoint | Low |
| 7 | Add unit test verifying cache does not grow unbounded | Low |
References
- Container App:
/subscriptions/33ccd01f-fc97-4fa1-b208-db8eb99a430b/resourceGroups/rg-sre-lab/providers/Microsoft.App/containerApps/ca-grubify-uybgeqfrc2owo - Log Analytics Workspace ID:
52bafba2-1aa0-4bf8-b789-d0e8aa24cd1f - Log Analytics Resource:
/subscriptions/33ccd01f-fc97-4fa1-b208-db8eb99a430b/resourceGroups/rg-sre-lab/providers/Microsoft.OperationalInsights/workspaces/law-uybgeqfrc2owo - App Insights:
/subscriptions/33ccd01f-fc97-4fa1-b208-db8eb99a430b/resourceGroups/rg-sre-lab/providers/Microsoft.Insights/components/appi-uybgeqfrc2owo - Alert:
/subscriptions/33ccd01f-fc97-4fa1-b208-db8eb99a430b/resourcegroups/rg-sre-lab/providers/Microsoft.Insights/metricAlerts/alert-http-5xx-sre-lab - Source code:
Controllers/CartController.cslines 14, 30-31
This issue was created by sre-agent-uybgeqfrc2owo--dd8769df
Tracked by the SRE agent here
Activity