Repository navigation
Incident: HTTP 5xx due to OutOfMemory in Cart API (Grubify Container App) — Recurrence #8
Description
Incident Report: HTTP 5xx due to OutOfMemoryException in CartController (Recurrence)
- Incident ID:
8efeae7a-fbfc-47e3-a271-d7fab02af000 - Service: Azure Container Apps —
ca-grubify-ubqf2fbkeowr4(rg:rg-sre-lab) - Subscription:
b32ee9e1-20dc-49b8-a81a-a0fbd25bd27a - FQDN:
ca-grubify-ubqf2fbkeowr4.graybay-c0190080.eastus2.azurecontainerapps.io - Active revision:
ca-grubify-ubqf2fbkeowr4--0000002(100% traffic)
Summary
The Grubify backend API experienced a flood of System.OutOfMemoryException errors starting at ~21:19 UTC on 2026-05-20, triggering the alert-http-5xx-sre-lab alert at 21:22:33 UTC. A total of 125 OOM exceptions were recorded in container console logs over a ~4 minute window. All stack traces point to CartController.AddItemToCart at line 30 in CartController.cs. This is a recurrence of the same root cause identified in Issue #6.
Impact
- API errors (HTTP 5xx) on all endpoints during the OOM window — Kestrel unable to serve any requests when heap is exhausted
- Failed cart operations (
POST /api/cart/{userId}/items) for all users during the spike - Potential data loss: in-memory carts and orders are lost if the container restarts due to OOM
- App Insights telemetry blackout: the SDK could not flush telemetry during OOM, so
requestsandexceptionstables showed zero failed requests — only container console logs captured the errors
Timeline (UTC)
- ~21:19: First OOM exceptions appear in container console logs (25 occurrences in 1 min)
- ~21:20: OOM rate escalates to 43/min
- ~21:21: Peak OOM rate at 44/min — API effectively unresponsive
- ~21:22: OOM rate drops to 13/min (possible partial GC or container restart)
- 21:22:33: Azure Monitor alert
alert-http-5xx-sre-labfires (Sev3) - ~21:25: SRE Agent investigation begins
Evidence
OOM Exception Timeline (1-min bins)
| Time (UTC) | OOM Count |
|---|---|
| 21:19 | 25 |
| 21:20 | 43 |
| 21:21 | 44 |
| 21:22 | 13 |
| Total | 125 |
Console logs (active revision)
fail: Microsoft.AspNetCore.Server.Kestrel[13]
Connection id "0HNLMMIMU7GRR", Request id "0HNLMMIMU7GRR:0000000F": An unhandled exception was thrown by the application.
System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at GrubifyApi.Controllers.CartController.AddItemToCart(String userId, AddCartItemRequest request) in /app/Controllers/CartController.cs:line 30
All 125 OOM exceptions follow this identical stack trace — CartController.AddItemToCart → line 30.
App Insights telemetry
- Exceptions table:
ZERO_ROWS_RETURNED— confirms telemetry loss during OOM (SDK cannot flush when managed heap is exhausted) - Requests table: No failed requests recorded — same telemetry blackout pattern observed in previous incident Incident: HTTP 5xx due to OutOfMemoryException in Cart API (Grubify Container App) #6
KQL Query Used (Container Console Logs)
ContainerAppConsoleLogs_CL
| where ContainerAppName_s == "ca-grubify-ubqf2fbkeowr4"
| where Log_s has "OutOfMemoryException"
| summarize OOMCount=count() by bin(TimeGenerated, 1m)
| order by TimeGenerated ascMetrics snapshot (Azure Monitor)
- Container CPU: 0.5 cores
- Container Memory: 1Gi (limit)
- Image:
acrcagrubifyubqf2fbkeowr4.azurecr.io/grubify-api:latest - Min/Max replicas: 1 / 5
Root Cause
Memory leak in CartController.cs (lines 14, 30–31): A static List<byte[]> RequestDataCache field accumulates a 10 MB byte[] allocation on every POST /api/cart/{userId}/items call. The cache is never cleaned up (the TODO on line 33 was never implemented). With a 1Gi container memory limit, approximately 100 add-to-cart calls exhaust the managed heap, causing System.OutOfMemoryException on all subsequent requests.
Code path:
// Line 14: Static list that grows unbounded
private static readonly List<byte[]> RequestDataCache = new();
// Line 30-31: 10MB allocated per request, added to static list
var requestData = new byte[10 * 1024 * 1024]; // 10MB buffer for request analytics
RequestDataCache.Add(requestData);This is the same unfixed root cause as Issue #6. The code fix was not applied between incidents.
Remediation
- Code: Remove the unbounded
RequestDataCachestatic list and the 10MB allocation inAddItemToCart. If analytics caching is needed, implement a bounded cache (e.g.,CircularBufferor LRU cache with a max size) or offload to an external store (Redis, Application Insights custom events). - Defensive: Add request payload size limits and rate limiting on the cart endpoint to prevent rapid memory accumulation even if a cache is used.
- Platform: Increase container memory limit from 1Gi to 2Gi as a short-term mitigation. Add memory-percentage-based autoscaling to trigger scale-out before OOM.
- Observability: Add a dedicated alert for
OutOfMemoryExceptioninContainerAppConsoleLogs_CLsince App Insights telemetry is unreliable during OOM events.
Action Items
| # | Action | Priority |
|---|---|---|
| 1 | Remove RequestDataCache and the 10MB allocation from CartController.AddItemToCart |
Critical |
| 2 | Increase container memory limit to 2Gi as immediate mitigation | High |
| 3 | Add memory-based autoscale rule to prevent single-replica OOM | High |
| 4 | Add Log Analytics alert for OOM exceptions in container console logs | Medium |
| 5 | Implement bounded analytics cache if request tracking is actually needed | Medium |
| 6 | Add integration test that validates memory usage under repeated cart operations | Low |
References
- Container App:
/subscriptions/b32ee9e1-20dc-49b8-a81a-a0fbd25bd27a/resourceGroups/rg-sre-lab/providers/Microsoft.App/containerApps/ca-grubify-ubqf2fbkeowr4 - Log Analytics Workspace ID:
a97181b0-7057-404a-9a72-e08a7074e0f8 - App Insights:
/subscriptions/b32ee9e1-20dc-49b8-a81a-a0fbd25bd27a/resourceGroups/rg-sre-lab/providers/Microsoft.Insights/components/appi-ubqf2fbkeowr4 - Alert: Portal link
- Previous incident: Issue #6
- SRE Agent thread: Investigation thread
This issue was created by sre-agent-ubqf2fbkeowr4--a4798a0c
Tracked by the SRE agent here
Activity
dohl-ms commentedon May 20, 2026
🤖 Grubify SRE Agent Bot
Code Fix Prepared
A fix has been prepared on branch
fix/cartcontroller-memory-leak-issue-8(commita2c5278) that removes the memory leak fromCartController.cs:Changes:
List<byte[]> RequestDataCachefield (line 14)byte[]allocation and cache add inAddItemToCart(lines 30-35)Diff summary:
Note: The PR could not be created automatically due to git authentication constraints in the SRE Agent environment. A developer should apply this fix manually or the branch can be pushed when credentials are available.
🔧 Status: Fix prepared, awaiting manual PR creation