Repository navigation
Incident: HTTP 5xx due to OutOfMemory in Cart API (Grubify Container App) #1
Description
Incident Report: OutOfMemoryException in Cart API caused by unbounded RequestDataCache
- Incident ID:
INC-2026033012-OOM-CART - Service: Azure Container Apps —
ca-grubify-v27gz65bhz4ak(rg:rg-sre-lab) - Subscription:
1b788948-bbc3-4576-80f0-6e0deff15e4b - FQDN:
ca-grubify-v27gz65bhz4ak.calmtree-15ce3d21.eastus2.azurecontainerapps.io - Active revision:
ca-grubify-v27gz65bhz4ak--0000002(100% traffic)
Summary
The Grubify "Add to Cart" API (POST /api/cart/{userId}/items) is failing with System.OutOfMemoryException. Every request to this endpoint allocates a 10MB byte array into a static, unbounded RequestDataCache list in CartController.cs (line 30) that is never cleaned up. After ~75 requests, the cache reached 750MB, exhausting the container's 1Gi memory limit and causing a flood of OOM exceptions (~110 errors in a 5-minute window).
Impact
- All "Add to Cart" operations fail with HTTP 500 once memory is exhausted — users cannot add items to cart
- Cascading failure risk: OOM can cause container restarts, dropping all in-flight requests and clearing the in-memory cart data for all users
- Affected endpoint:
POST /api/cart/{userId}/items— core user-facing functionality
Timeline (UTC)
- ~12:00: Container app starts with baseline memory of ~1.2 MB working set
- ~12:05: Memory jumps to ~20 MB as initial requests come in
- ~12:45: First batch of Cart API requests (12 requests); memory rises to ~35 MB; 1 error logged
- ~12:50:11:
RequestDataCacheat 66 entries (660MB); rapid POST calls continue - ~12:50:26: Cache reaches 75 entries (750MB); approaching 1Gi container limit
- ~12:51:01:
System.OutOfMemoryExceptionflood begins — 110+ OOM errors in 5 minutes - ~12:51:52: OOM errors continuing; API unresponsive for cart operations
Evidence
Console logs (active revision)
OOM Exception flood (12:51 UTC):
System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
(30+ occurrences between 12:51:01 and 12:51:52 UTC — all from grubify-api container in revision ca-grubify-v27gz65bhz4ak--0000002)
Cache growth logs (12:50 UTC):
Analytics cache: Added request data. Total entries: 66
Cache size: 660MB
Analytics cache: Added request data. Total entries: 67
Cache size: 670MB
...
Analytics cache: Added request data. Total entries: 75
Cache size: 750MB
Metrics snapshot (Azure Monitor)
- Requests (5m bins): 0 until 12:45 (12 req), then 120 at 12:50
- OOM Errors: 1 at 12:45, then 110 at 12:50
- RestartCount: 0 (OOM exceptions thrown before container OOM-kill threshold)
- WorkingSetBytes (avg): 1.2MB (12:00) → 35.4MB (12:45) → 47.1MB (12:50) — Azure Monitor averages understate the actual heap usage; application logs show cache at 750MB
- UsageNanoCores: Baseline ~25K → spike to 5.76M at 12:50 (GC pressure from massive allocations)
KQL Queries Used
Error logs:
ContainerAppConsoleLogs_CL
| where Log_s contains "OutOfMemory" or Log_s contains "OOM"
| project TimeGenerated, Log_s, ContainerName_s, RevisionName_s
| order by TimeGenerated descCache growth:
ContainerAppConsoleLogs_CL
| where Log_s contains "Analytics cache" or Log_s contains "Cache size"
| project TimeGenerated, Log_s, ContainerName_s
| order by TimeGenerated descRoot Cause
Memory leak in CartController.cs. The AddItemToCart method (line 26-63) allocates a 10MB byte array on every POST request and appends it to a static List<byte[]> RequestDataCache (line 14) that has no eviction, cleanup, or size limit. With the container's 1Gi memory limit, only ~100 requests are needed to exhaust memory.
Problematic code (lines 29-31):
// Store request data for analytics and performance monitoring
var requestData = new byte[10 * 1024 * 1024]; // 10MB buffer for request analytics
RequestDataCache.Add(requestData);The TODO comment on line 33 confirms this was a known gap: // TODO: Implement cache cleanup mechanism in future sprint
Additionally, the in-memory cart storage itself (static Dictionary<string, Cart> UserCarts, line 11) has no eviction policy, representing a secondary (slower) memory leak for long-running containers.
Remediation
- Code (Critical): Remove the
RequestDataCacheentirely — the 10MB byte array allocation serves no functional purpose (it's a placeholder "analytics" buffer). If analytics tracking is needed, use Application Insights or a lightweight counter instead of raw byte buffers. - Code (Secondary): Replace the in-memory
UserCartsdictionary with a bounded cache (e.g.,MemoryCachewith size limits and sliding expiration) or an external store (Redis, Cosmos DB). - Defensive: Add payload size validation and rate limiting on the Cart API to prevent rapid repeated calls from a single user.
- Platform: Consider increasing the container memory limit from 1Gi to 2Gi as a short-term safety net, and add memory-based autoscale rules.
- Observability: Add an Azure Monitor alert for
WorkingSetBytes > 75%of the container limit, and a log alert forOutOfMemoryExceptionin console logs.
Action Items
| # | Action | Priority |
|---|---|---|
| 1 | Remove RequestDataCache and the 10MB allocation from CartController.AddItemToCart |
Critical |
| 2 | Replace in-memory UserCarts with bounded MemoryCache or external store (Redis) |
High |
| 3 | Add input validation and rate limiting on POST /api/cart/{userId}/items |
Medium |
| 4 | Add Azure Monitor alert for WorkingSetBytes > 768MB and log alert for OOM exceptions | Medium |
| 5 | Increase container memory to 2Gi as short-term safety net | Medium |
| 6 | Add unit/integration tests for cart memory behavior under load | Low |
References
- Container App:
/subscriptions/1b788948-bbc3-4576-80f0-6e0deff15e4b/resourceGroups/rg-sre-lab/providers/Microsoft.App/containerApps/ca-grubify-v27gz65bhz4ak - Log Analytics Workspace ID:
e8dc01f2-c152-4cb5-9458-8662fdbd7418 - App Insights:
/subscriptions/1b788948-bbc3-4576-80f0-6e0deff15e4b/resourceGroups/rg-sre-lab/providers/Microsoft.Insights/components/appi-v27gz65bhz4ak - Source file:
GrubifyApi/Controllers/CartController.cs
This issue was created by sre-agent-v27gz65bhz4ak--9199fdaf
Tracked by the SRE agent here
Activity