Production OOMKill Root Cause & Fix

Overview

FastAPI/Gunicorn pods kept getting OOMKilled under production load, with no clear signal pointing to why. I root-caused the memory leak and worker mis-sizing behind it and fixed the underlying configuration.

The problem

FastAPI/Gunicorn pods kept OOMKilling under production load, with no clear signal pointing to why, putting client-facing services at risk.

What I did

1. 1

   Used Grafana dashboards and pod metrics to trace memory growth over time under load.

2. 2

   Root-caused a memory leak plus Gunicorn worker mis-sizing for the traffic pattern.

3. 3

   Corrected resource limits and Gunicorn worker configuration.

4. 4

   Stood up proactive alerts so the same failure mode would be caught before it recurred.

Tech used

Outcomes

• Recurring OOMKills eliminated in production.

• Issue caught and fixed before clients noticed any impact.