tencent cloud

This document describes how to use exit codes to troubleshoot pod issues.

Querying Pod Exceptions

Run the following command to query pod exceptions:

kubectl describe pod <pod name>

The returned result is as follows:

Containers:
  kubedns:
    Container ID:  docker://5fb8adf9ee62afc6d3f6f3d9590041818750b392dff015d7091eaaf99cf1c945
    Image:         ccr.ccs.tencentyun.com/library/kubedns-amd64:1.14.4
    Image ID:      docker-pullable://ccr.ccs.tencentyun.com/library/kubedns-amd64@sha256:40790881bbe9ef4ae4ff7fe8b892498eecb7fe6dcc22661402f271e03f7de344
    Ports:         10053/UDP, 10053/TCP, 10055/TCP
    Host Ports:    0/UDP, 0/TCP, 0/TCP
    Args:
      --domain=cluster.local.
      --dns-port=10053
      --config-dir=/kube-dns-config
      --v=2
    State:          Running
      Started:      Tue, 27 Aug 2019 10:58:49 +0800
    Last State:     Terminated
      Reason:       Error
      Exit Code:    255
      Started:      Tue, 27 Aug 2019 10:40:42 +0800
      Finished:     Tue, 27 Aug 2019 10:58:27 +0800
    Ready:          True
    Restart Count:  1

Exit Code is the status code of the last container exit. If it is not 0, the container exited due to an exception. You can use the exit code to further troubleshoot the problem.

Exit Codes

• A valid exit code is between 0 and 255.
• 0 means the container exited normally.
• If the container exited due to an external signal, the exit code is between 129 and 255. For example, if the operating system sent kill -9 or ctrl+c as the termination signal, the status is SIGKILL or SIGINT.
• If the container exited due to an internal signal, the exit code is between 1 and 128. However, in some circumstances, the exit code might be between 129 and 255.
• If the specified exit code has a value outside the 0-255 range, such as exit(-1), it is automatically translated to a value in the 0-255 range.
  If the exit code is specified as code, it is translated as follows:
  • If the exit code is negative:

    256 - (|code| % 256)

  • If the exit code is positive:

    code % 256

Typical Exit Codes

• 137: indicates that the process was killed by SIGKILL. Possible reasons are:
  • Pod memory reached resources.limits, such as Out of Memory (OOM). Pod resource limits are implemented by using Linux cgroup. If the memory of a pod reaches its limit, cgroup forces it to stop (with a similar effect to kill -9). If you use describe pod, you can see the value of Reason is OOMKilled.
  • If the host does not have sufficient resources (OOM), the kernel stops some processes to free up the memory.

    Note：

    If the process is stopped due to OOM, cgroup, or the host, you can find relevant records in system logs:
    Ubuntu system logs are stored in /var/log/syslog, whereas CentOS system logs are stored in /var/log/messages. You can run the journalctl -k command to view system logs in both operating systems.

  • livenessProbe failed, which causes kubelet to stop the pod.
  • Pod stopped by a trojan process.
• 1 and 255: indicates common issues. Check container logs for further troubleshooting. For example, this could be the result of exit(1) or exit(-1). -1 is translated to 255.

Standard Linux Interruption Signals

Linux programs send an exit code when they are interrupted by external signals. The value of the exit code is the value of the interrupt signal plus 128. For example, the value of SIGKILL is 9, so the program exit code is 9 + 128 = 137. For more standard interrupt signals, see the following table:

C/C++ Exit Codes

/usr/include/sysexits.h provides standardized exit codes for C and C++. These codes are described in the following table:

Status Code Reference

For the description of more status codes, see the following table: