Fix Docker Container Exited with Code 137 (OOM)

Symptoms

When a Docker container exits with code 137, it typically indicates that the container was killed by the kernel's Out of Memory (OOM) killer. Common symptoms include:

• Container stops unexpectedly with exit code 137
• Logs may show Killed or OOMKilled in docker inspect
• Application inside container becomes unresponsive before exit
• Host system may show high memory usage

Root Causes

Exit code 137 is the result of the Linux kernel's OOM killer terminating a process (usually the container's main process) when the system or container runs out of memory. This can happen due to:

• Insufficient memory allocated to the container via --memory flag
• Memory leak in the application running inside the container
• Host system memory exhaustion
• Incorrect memory swap settings

Step-by-Step Fix

1. Confirm OOM Kill

Run docker inspect <container_name> | grep -i oom to check if OOMKilled is true. Also check host logs: sudo dmesg | grep -i oom for kernel messages.

2. Check Current Memory Usage

Use docker stats --no-stream to see real-time memory usage of running containers. For the failed container, review logs: docker logs <container_name>.

3. Increase Container Memory Limit

If the container has a memory limit, increase it. For example, if previously run with --memory=512m, try --memory=1g. For unlimited memory, omit the flag (not recommended for production).

docker run -d --memory=1g --memory-swap=1g my_image

Note: --memory-swap should equal --memory to disable swap usage.

4. Adjust Docker Compose (if applicable)

In docker-compose.yml, add:

services:
  my_service:
    deploy:
      resources:
        limits:
          memory: 1g
        reservations:
          memory: 512m

5. Restart the Container

Remove the old container and run a new one with updated memory settings.

Alternative Fixes

• Optimize application memory usage: Profile the application to reduce memory footprint or fix memory leaks.
• Add swap space: Increase host swap to prevent OOM kills, though this may degrade performance.
• Use memory cgroup: Enable cgroup v2 on the host for better memory accounting.
• Set memory reservation: Use --memory-reservation to give the container a soft limit.

Prevention

• Monitor container memory with tools like docker stats, Prometheus, or Grafana.
• Set appropriate memory limits based on application profiling.
• Implement health checks and auto-restart policies (--restart=always or unless-stopped).
• Use orchestration tools (Kubernetes, Docker Swarm) to manage resource limits across nodes.
• Regularly update Docker and host kernel to benefit from OOM handling improvements.

By following these steps, you can resolve exit code 137 and ensure your Docker containers run reliably.