Linux OOM Killer: Why It Kills Your Process + Fixes

1. Memory leak in your application

This is the most common reason I see. A process allocates memory and never frees it. Over time, the system runs out of RAM, and the OOM killer picks a victim. Usually it's the one eating the most memory, but not always — the kernel has a heuristic that can kill any process.

What's actually happening here is: your app grows until it hits the system limit. The kernel tries to free memory by swapping, but if swap is full or disabled, it invokes the OOM killer. You'll see a line in dmesg like:

[12345.678901] oom-killer: gfp_mask=0x... order=0, oom_score_adj=0
[12345.678902]  [pid]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[12345.678903]  [1234]     0 1234    567890    12345  ...        0              0 myapp

Fix: Find the leaking process. Use top or htop and sort by memory (%MEM). Restart it. Then patch the leak — typically a C/C++ program forgetting to free() or a Python script holding references. If you can't fix the code, set a per-process memory limit with systemd or cgroups:

# systemd unit drop-in
[Service]
MemoryMax=512M

The reason this works: the process gets killed before it starves the whole system. The OOM killer still runs, but only that unit's processes are affected.

2. Swap is too small or disabled

Swap acts as a safety net. When RAM fills up, the kernel pages out idle memory to disk. If swap is too small (or you turned it off thinking "I have enough RAM"), then when RAM runs out, there's nowhere to go but OOM.

I see this often on cloud VMs with 1–2 GB RAM and no swap. The kernel tries to compact memory, fails, then kills the first victim it finds. Real-world trigger: running a database or a Node.js app with a memory leak. The app grows to 90% RAM, then all new requests cause OOM.

Fix: Add a swap file. On Linux 3.x+, swap files are fast enough for this purpose. Don't bother with a swap partition on modern systems — a file works fine.

# Create a 2GB swap file
sudo fallocate -l 2G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile

# Make it permanent
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab

Set swap size to 20–50% of your RAM on a server, or equal to RAM for a desktop. If you're running memory-intensive stuff like database servers, go for 100% of RAM. The kernel will use swap before OOM. It's slower, but the app doesn't crash.

3. Memory overcommit is on (default)

By default, the Linux kernel lets processes allocate more virtual memory than physical RAM + swap exists. This is called overcommit. It works because most programs don't use all the memory they ask for. But when they do, and the total exceeds actual capacity, the OOM killer wakes up.

What's actually happening: a process calls malloc(1GB), gets a pointer, but hasn't touched any pages yet. The kernel says "sure" without reserving physical RAM. Then the process writes to those pages, and the kernel realizes it can't back them. Boom.

This is especially nasty with Java apps that set high -Xmx values. They reserve huge heaps upfront, then slowly fill them. If multiple JVMs run on the same box, they can all allocate their max heap, but the sum exceeds RAM + swap. When they all actually use that memory, OOM kills something.

Fix: Disable overcommit strictly. This makes malloc() fail if memory isn't available, which prevents the OOM killer from ever activating. Your apps will get a clean error instead of a random kill.

# Check current setting
sysctl vm.overcommit_memory
# 0 = heuristic (default), 1 = always overcommit, 2 = strict

# Set to strict (no overcommit)
echo 'vm.overcommit_memory = 2' | sudo tee -a /etc/sysctl.conf
sudo sysctl -p

The downside: apps that allocate large virtual memory (like databases with memory-mapped files) may fail startup. You'll need to adjust vm.overcommit_ratio (default is 50% of RAM). I keep it at 50% for most servers. If you run PostgreSQL, set it to 100% so it can use shared buffers.

Quick-reference summary

One last thing: if you see OOM in logs and you're on a shared host or container, check cgroup limits first. Docker containers have their own memory limits. The kernel's global OOM killer might not be the culprit — the container's OOM killer (cgroup memory limit) is. Run cat /sys/fs/cgroup/memory/memory.usage_in_bytes and compare to the limit. In that case, the fix is to bump the container's memory limit, not the host's.

I've been using Linux for 15 years. The OOM killer is a last resort, not a feature. If you hit it, fix the root cause — don't just reboot.