Your web server just vanished. Or your database. Or that long-running Python script. You check dmesg and see something like: Killed process 12345 (nginx) total-vm:2048000kB. This happens most often when you're running memory-heavy workloads on a server with limited RAM — think a 2GB VPS running both a Node.js app and PostgreSQL. The trigger is the kernel hitting a hard wall: it needs memory for a new allocation (like a page fault or a new process fork), the system has zero free pages, and swap is either full or non-existent. The OOM killer then picks a victim based on a heuristic, and your process gets the axe.
Root Cause: The Kernel's Last Resort
What's actually happening here is that Linux uses an optimistic memory allocation strategy called overcommit. When a process calls malloc or mmap, the kernel doesn't actually reserve physical RAM for that memory — it just says "sure, you can have it later." This works fine until every process tries to use all the memory they've been promised, and the system runs out. The kernel can't just return NULL to a process that's already been writing to pages — that would crash the process in an unpredictable way. So it picks a victim process, sends it a SIGKILL, and reclaims its memory.
The OOM killer isn't random, but it's not fair either. It uses a badness score that factors in:
- Total virtual memory used
- RSS (resident set size)
- Whether the process is running as root
- How long the process has been running
- Whether it's using huge pages
The worst part? The kernel doesn't guarantee it'll kill the biggest memory hog. It can kill a small process if the big one has a low oom_score_adj. I've seen it kill sshd instead of a runaway Java app.
The Fix: Four Approaches, Pick the Right One
Don't just disable the OOM killer — that'll cause kernel panics or silent hangs. Instead, you want to either stop the system from running out of memory, or protect your critical processes. Here's the order I'd do it:
Step 1: Check what actually happened
Before fixing, confirm the OOM killer was the culprit. Run:
sudo dmesg | grep -i 'killed process'
You'll see lines like Out of memory: Killed process 12345 (mysqld). Also check /var/log/syslog or /var/log/messages for similar entries. The log tells you the total-vm and anon-rss of the killed process — that's how much memory it was using.
Step 2: Add swap space (quick win)
If you have no swap, the OOM killer activates much sooner. Adding swap gives the system a buffer. On a server with 4GB RAM, I'd add at least 2GB swap:
sudo fallocate -l 2G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
To make it permanent, add to /etc/fstab: /swapfile none swap sw 0 0. The reason this works is that swap allows the kernel to page out inactive memory, freeing up RAM for active processes. It won't fix a memory leak but it buys time.
Step 3: Adjust the OOM score for your critical processes
You can tell the kernel "don't kill this process" by setting a negative oom_score_adj. This is the responsible way to protect, say, your database or SSH server. Find the PID of your critical process, then:
echo -1000 > /proc/$(pgrep -f mysqld)/oom_score_adj
The value ranges from -1000 (never kill) to +1000 (always kill first). A value of -1000 means the OOM killer will never select this process. For a systemd service, make it persistent by adding:
[Service]
OOMScoreAdjust=-1000
to the service file under /etc/systemd/system/. Then reload with systemctl daemon-reload.
Step 4: Reduce memory overcommit (last resort)
If you're consistently running near the limit, change the overcommit policy. This forces the kernel to actually check if memory is available before granting allocations. Edit /etc/sysctl.conf and add:
vm.overcommit_memory = 2
vm.overcommit_ratio = 50
Then run sysctl -p. This says: "only allow memory allocations up to 50% of swap + RAM." The downside: malloc can return NULL if there's not enough free pages, which some applications don't handle well. Your Python web server might crash with a memory allocation error instead of being OOM-killed. Pick your poison.
Step 5: Tune the OOM killer to kill faster (not slower)
This sounds backwards, but it helps in production. If you set vm.panic_on_oom=0 (default), the kernel keeps running after killing a process. But sometimes you want it to panic and reboot instead of killing random PIDs. That's vm.panic_on_oom=1. I've found this useful on dedicated database servers: if MySQL gets killed, the whole system reboots and recovers cleanly rather than running in a degraded state.
If It Still Fails
You've added swap, set OOM scores, and the OOM killer is still killing processes. Three things to check:
- Memory leak in your application: Run
top -o %MEMorhtopand watch over time. If a process grows indefinitely, that's your real problem. Fix the leak, not the OOM killer. - cgroup memory limits: If you're using Docker, systemd services with MemoryLimit, or Kubernetes, the OOM killer can fire at the cgroup level. Check
/sys/fs/cgroup/memory/formemory.limit_in_bytes. A Docker container with a 512MB limit will get OOM-killed even if the host has 64GB free. - Too many processes fork()ing: Something like a misconfigured Apache spawning hundreds of workers can exhaust memory. Check
ps aux | wc -land compare to your total memory budget.
One more thing: don't set vm.overcommit_memory = 2 on a server that runs multiple services unless you've verified they all handle allocation failures. I've seen MongoDB crash hard with that setting. The real fix is usually adding more RAM or fixing the leak.