Repository navigation
Incident: HTTP 5xx due to OutOfMemoryException in Cart API (Grubify Container App) #7
Description
Incident Report: HTTP 5xx due to OutOfMemoryException in Cart API
- Incident ID:
67aa3724-330f-4a34-a559-5f0e12f6f000 - Service: Azure Container Apps —
ca-grubify-uybgeqfrc2owo(rg:rg-sre-lab) - Subscription:
33ccd01f-fc97-4fa1-b208-db8eb99a430b - FQDN:
ca-grubify-uybgeqfrc2owo.ashymushroom-a36a19ce.eastus2.azurecontainerapps.io - Active revision:
ca-grubify-uybgeqfrc2owo--0000005(100% traffic)
Summary
The Grubify API container app experienced repeated System.OutOfMemoryException errors caused by an unbounded in-memory cache in CartController.cs. Each POST /api/cart/{userId}/items request allocates a 10MB byte[] buffer that is never freed, growing the static RequestDataCache from 10MB to 750MB in approximately 2 minutes. Once memory approached the 1Gi container limit, Kestrel began throwing OOM exceptions on every inbound request, resulting in HTTP 5xx responses. This is a recurring incident — the same root cause was identified in a previous alert (ID 1ad8abaf-cf41-4c34-b710-6a826eaff000).
Impact
- All API endpoints became unavailable during OOM saturation windows, returning HTTP 5xx errors to users
- Cart operations (add to cart, view cart, checkout) failed for all users during the error window
- Two distinct error bursts observed: initial burst at ~08:08–08:14 UTC and a second burst at ~09:19 UTC that triggered this alert
Timeline (UTC)
- ~08:06:51: First cache size log entry —
Cache size: 10MB(1 entry). Cache growth begins. - ~08:07:29–08:08:51: Rapid cache growth — 75
AddItemToCartrequests grow cache from 20MB to 750MB in ~90 seconds. - ~08:08: First OOM errors appear (8 OOM exceptions/min, 24 total errors/min).
- ~08:09–08:10: Peak OOM storm — 52-53 OOM exceptions/min, 156-159 total errors/min. All API requests failing.
- ~08:11–08:14: Error rate subsides (14→3 OOM/min) as traffic drops or container partially recovers.
- ~09:19: Second OOM burst — 19 OOM exceptions/min, 57 total errors/min.
- ~09:22:37: Azure Monitor alert
alert-http-5xx-sre-labfires (Sev3).
Evidence
Console logs (active revision)
fail: Microsoft.AspNetCore.Server.Kestrel[13]
Connection id "0HNOT6LJC9K0V", Request id "0HNOT6LJC9K0V:00000016": An unhandled exception was thrown by the application.
System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
fail: Microsoft.AspNetCore.Server.Kestrel[13]
Connection id "0HNOT6LJC9K0V", Request id "0HNOT6LJC9K0V:00000015": An unhandled exception was thrown by the application.
System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
Repeated across dozens of request IDs on connection 0HNOT6LJC9K0V, revision ca-grubify-uybgeqfrc2owo--0000005.
Cache growth log
08:06:51 Cache size: 10MB (1 entry)
08:07:29 Cache size: 20MB (2 entries)
08:07:38 Cache size: 100MB (10 entries)
08:08:00 Cache size: 300MB (30 entries)
08:08:23 Cache size: 500MB (50 entries)
08:08:51 Cache size: 750MB (75 entries) ← peak, near 1Gi limit
Traffic and Error Chart
Metrics snapshot (Azure Monitor)
- OOM Errors (1m bins): 8 → 52 → 53 → 14 → 15 → 3 → 19 (second burst at 09:19)
- Total Errors (1m bins): 24 → 156 → 159 → 42 → 45 → 9 → 57
- RestartCount: Container remained running (no OOM kill restart observed — errors returned to clients instead)
- Container resources: 0.5 vCPU, 1Gi memory
- MemoryPercentage: Cache alone consumed ~73% of container memory (750MB / 1024MB)
Root Cause
CartController.cs (lines 14, 30-31) maintains a static List<byte[]> RequestDataCache. Every POST /api/cart/{userId}/items call allocates a new 10 * 1024 * 1024 byte array (10MB) and appends it to this list with no eviction, cleanup, or size limit. After 75 requests, the cache consumed 750MB — approaching the container's 1Gi memory limit — causing System.OutOfMemoryException on all subsequent requests.
Code path: CartController.AddItemToCart() → new byte[10 * 1024 * 1024] → RequestDataCache.Add(requestData) → never freed.
Remediation
- Code: Remove the
RequestDataCacheentirely or replace with a bounded, time-expiring cache (e.g.,IMemoryCachewith size limit and sliding expiration). The current "analytics" buffer serves no actual purpose. - Defensive: Add request payload size validation and rate limiting on the cart endpoint to prevent rapid memory exhaustion under load.
- Platform: Increase container memory from 1Gi to 2Gi as a short-term buffer. Add memory-percentage-based autoscaling or a memory usage alert threshold at 70%.
- Observability: Add an Azure Monitor alert for
WorkingSetBytesexceeding 70% of container limit. Instrument the application with Application Insights SDK for distributed tracing and exception telemetry.
Action Items
| # | Action | Priority |
|---|---|---|
| 1 | Remove or replace RequestDataCache in CartController.cs with bounded IMemoryCache |
High |
| 2 | Add unit test verifying cart operations don't leak memory | Medium |
| 3 | Increase container memory to 2Gi as interim mitigation | Medium |
| 4 | Add memory usage alert (WorkingSetBytes > 70% of limit) | Medium |
| 5 | Integrate Application Insights SDK for distributed tracing | Low |
| 6 | Add rate limiting on POST /api/cart/{userId}/items endpoint |
Low |
References
- Container App:
/subscriptions/33ccd01f-fc97-4fa1-b208-db8eb99a430b/resourceGroups/rg-sre-lab/providers/Microsoft.App/containerApps/ca-grubify-uybgeqfrc2owo - Log Analytics Workspace ID:
52bafba2-1aa0-4bf8-b789-d0e8aa24cd1f - Log Analytics Resource ID:
/subscriptions/33ccd01f-fc97-4fa1-b208-db8eb99a430b/resourceGroups/rg-sre-lab/providers/Microsoft.OperationalInsights/workspaces/law-uybgeqfrc2owo - App Insights:
/subscriptions/33ccd01f-fc97-4fa1-b208-db8eb99a430b/resourceGroups/rg-sre-lab/providers/Microsoft.Insights/components/appi-uybgeqfrc2owo - Previous related incident: GitHub issue Incident: HTTP 5xx due to OutOfMemoryException in Cart API (Grubify Container App) #6
- Alert rule:
/subscriptions/33ccd01f-fc97-4fa1-b208-db8eb99a430b/resourcegroups/rg-sre-lab/providers/Microsoft.Insights/metricAlerts/alert-http-5xx-sre-lab - Source code:
GrubifyApi/Controllers/CartController.cslines 14, 30-31
This issue was created by sre-agent-uybgeqfrc2owo--dd8769df
Tracked by the SRE agent here
Activity
AlahmadiQ8 commentedon Sep 28, 2026
🤖 Grubify SRE Agent Bot
A fix has been submitted in PR #8 to permanently resolve this memory leak.
Changes: Removed the
static List<byte[]> RequestDataCacheand the 10MB allocation inAddItemToCart()fromCartController.cs. No functional behavior changes — the cache served no actual analytics purpose.PR: #8
🔧 Status: Fix submitted — pending merge