Open Source Help: What is Exit Code 137, and Can You Fix It?

Exit code 137 is triggered when a pod or a container within your Kubernetes environment exceeds the amount of memory that they're assigned. Typically, this exit code is accompanied by or simply known as OOMKilled. Killed in this description refers to the result of this error, which causes a pod to terminate.

The acronym OOM stands for out of memory, which is caused by a pod pushing past their memory limit that is set. If you're unsure why your pod terminated, one of the easiest ways of finding out is to run 'kubectl get pods command', which will then summon a status update on a specific pod. Within this, you'll then find OOMKilled, which will alert you to the fact that exit code 137 has been triggered.

Where does OOMKilled Come from?

While OOMKilled is a response that you're likely to see within your Kubernetes environment, it isn't actually native to this system. In fact, it's a central feature of Linux programming that has been carried over to Kubernetes to help facilitate this system. Within Linux Kernel, OOMKilled is known as the OOM Killer, offering the same warning and response as it does within Kubernetes.

Typically, if a platform is taking up too much memory on a certain system, Linux will move through different nodes and decide which of them to kill, scoring all of them with oom_score on a scale of which nodes are taking up the most memory to the least memory. The nodes that are taking up the most memory are the most likely to be terminated, with exit code 137 being the reasoning given for this.

What are the causes of OOMKilled?

If your Kubernetes ecosystem is returning with 'exited with code 137', then you're likely facing a memory problem within this system. While this can be a frustrating issue, it's not the end of the world as this is a fairly easy concern set to remedy.

Typically, there are a few core causes of OOMKilled within a Kubernetes environment:

• Memory Limitations – When running a Kubernetes environment, there are typically hundreds of nodes all working toward a common good. While this system is effective, memory limits can cause exit code 137 to appear, which will then terminate the pod you're working within. One of the most common causes for this is memory limitations within pods. In each Kubernetes pods, you're able to specify a memory limitation on pods. If this is exceeded, then you'll receive the OOMKilled error.

• Memory Leak – If a container has a certain memory limit placed upon it, then it can sometimes reach the limit and then begin to leak out into other processes. This will be flagged as an error and then terminated.

• Overcommitted Nodes – When a pod uses more memory than that which is assigned to it, you'll receive this specific error.

As this is a memory-based error, any of the causes for this exit code will be related to the poor management or use of memory within your Kubernetes ecosystem.

How to Fix Exit Code 137?

As suggested earlier, exit code 137 is one of the easiest errors to fix as it all boils down to either reducing your processes or increasing the amount of memory that each of the nodes is assigned.

If you're trying to fix exit code 137, try the following three things:

• Increase disk space – Very simply, the easiest way to fix any memory that's connected to memory is to increase the disk space that your Kubernetes environment has to work with. This is a blanket fix, as increasing the amount of space will ensure that your ecosystem is no longer reaching its maximum. However, if you consistently run into this issue, then you should endeavor to try the following two fixes to ensure you create a memory-efficient system.

• Add additional pod volume – Within each pod in Kubernetes, you can set a minimum and maximum amount of memory that a certain pod is allowed to use. If a few pods are consistently getting exit code 137 returned to them, then that is a sign that you need to increase the amount of space you afford to the pod. By increasing the maximum limit manually in the pods that are under the most strain, you'll be able to reduce the frequency with which this problem occurs.

• Reduce parallel runners – Parallel processing is where you run two systems at once to fix or maintain different functions. While this boosts the efficiency of Kubernetes and what you can achieve with it, it also leads to much more strain on the memory of the ecosystem as a whole. By using parallel running less frequently, you'll be able to keep the overall memory usage of your system lower, helping to reduce the amount of exit code 137 you run into.

By moving through these three steps, you will have most likely increased the amount of memory that your system has, as well as optimizing individual pods to ensure they have enough memory to complete all of their functions without terminating unexpectedly.

Final Thoughts

If you're experiencing exit code 137 within Kubernetes, then there is likely an issue with how your Kubernetes environment is managing space across its pods, nodes, and containers. As a baseline, Kubernetes suggests that you give each node in your cluster around 300 MIB of memory, which should be enough for the node to function properly.

However, depending on the complexity of your Kubernetes ecosystem, it's always a better idea to have as much memory as possible. If your system has enough space, then designate a higher amount of storage to help every node run without reaching the need for exit code 137 to appear.