Fix Linux OOM Killer Killing Your App - Direct Steps

When It Happens

You're running a database server like MySQL 8.0 or a Java app (e.g., Jenkins, Tomcat) on an Ubuntu 22.04 or CentOS 7 box. Suddenly the process disappears from 'ps aux'. No crash log in the app. 'systemctl status' says 'killed'. You check '/var/log/syslog' or run 'dmesg' and see something like:

[12345.678910] Out of memory: Killed process 12345 (mysqld) total-vm:4212341kB, anon-rss:1843296kB, file-rss:0kB, shmem-rss:0kB

That's the OOM killer. It runs when the system's memory is completely exhausted. It picks a process and shoots it. Usually it picks the one eating the most memory — your app.

Root Cause

Linux by default overcommits memory. It lets apps allocate more virtual memory than physical RAM + swap. Then when apps actually touch that memory (write to it), the system runs out. The kernel panics and the OOM killer kicks in. The culprit here is almost always swap being too small or turned off. If swap is tiny (like 1 GB on a 16 GB RAM server), the system has no breathing room when memory fills up.

Another common trigger: running a memory-hungry app on a server with other stuff. Like running both MySQL and a Node.js server on the same 4 GB box. Eventually one of them spikes, memory runs out, and something dies.

Also check cgroups — if you use systemd or Docker, they can set memory limits that trigger OOM kills even if the host has free memory. Systemd's 'MemoryMax' or Docker's '--memory' flag will do this.

Fix Step by Step

Check dmesg for the kill reason. Run 'dmesg | grep -i oom' or look at '/var/log/kern.log'. You need to see which process died and why. If it's not memory pressure (e.g., cgroup limit), skip to step 4.
Review your swap configuration. Run 'swapon --show'. If swap is 0 or under 2 GB, you need more. For a server with 8 GB RAM, set swap to 4 GB. Use a swap file if you can't resize partitions:
```
dd if=/dev/zero of=/swapfile bs=1M count=4096
chmod 600 /swapfile
mkswap /swapfile
swapon /swapfile
echo '/swapfile none swap sw 0 0' >> /etc/fstab
```
Tweak vm.overcommit_ratio. By default it's 50. That means the kernel lets apps allocate up to 50% over physical RAM. Drop it to 30 or even 0. Edit '/etc/sysctl.conf' and add:
```
vm.overcommit_ratio=30
vm.overcommit_memory=2
```
Setting 'overcommit_memory=2' disables overcommit entirely (kernel calculates exact memory). This can break some apps (like Oracle DB). If your app gives 'Cannot allocate memory' errors, change back to '1' (always overcommit) and adjust the ratio instead.
Protect your critical app from being killed. Set 'oom_score_adj' for the process. For systemd services, edit the unit file:
```
[Service]
OOMScoreAdjust=-500
```
Negative values make the process less likely to be killed. Don't go below -1000 unless you really know what you're doing. This won't prevent OOM entirely — it just picks someone else.
If using Docker or containers, check memory limits. Run 'docker stats' to see memory usage. If a container hits its limit, it gets OOM killed. Increase the limit:
```
docker run --memory=4g myapp
```
Or in Docker Compose:
```
services:
  myapp:
    mem_limit: 4g
```
For systemd services, verify no memory cap. Check 'systemctl show myapp.service | grep MemoryMax'. If it's set and too low, remove or increase it. Edit the unit:
```
MemoryMax=8G
```
Then reload: 'systemctl daemon-reload && systemctl restart myapp.service'.
Reduce memory usage of your app if possible. For MySQL, set 'innodb_buffer_pool_size' to 50-70% of RAM. For Java, set '-Xmx' to a reasonable max. Don't let these apps eat everything.

If It Still Fails

Check a few things:

Is the OOM killer actually running? Sometimes apps crash for other reasons (segfault, out of disk space). Look at 'dmesg' and 'journalctl -xe' for other errors.
Is your swap being used? Run 'free -h'. If swap is full but RAM has free space, your kernel might be swapping aggressively due to 'vm.swappiness' being too high. Set it lower: 'sysctl vm.swappiness=10'.
Are you running memory ballooning in VMs? If this is a VM, the hypervisor might be reclaiming memory. Disable ballooning or set a minimum. For KVM, add 'virsh setmem domain 4096' to reserve.
Update your kernel. Old kernels (before 4.20) had a bug where OOM killer could misfire even with free memory. Run 'uname -r' and update if you're below 5.x.
Try 'panic_on_oom' as a last resort. Set 'vm.panic_on_oom=1' in sysctl. This makes the kernel panic instead of killing a process. Then set 'kernel.panic=10' to auto-reboot. Not pretty but it stops random kills.

Bottom line: the OOM killer is a safety net, not a solution. Proper swap, adjusted overcommit, and app memory limits stop it 90% of the time. I've seen this exact fix work on hundreds of servers. Skip the registry tweaks — they don't exist on Linux.