tencent cloud

PrevNext

Feedback

search by keyword

Recent Pages

Documentation
Tencent Kubernetes Engine
    Documentation
    Download PDF

    Cluster DNS Troubleshooting

    Last updated: 2024-12-13 14:48:39
    Download PDF

      Troubleshooting Approaches

      1. Make sure that the cluster DNS runs normally

      Container DNS passes through cluster DNS (usually CoreDNS). First, make sure that the cluster DNS runs normally. You can see the cluster IP of the DNS in the --cluster-dns startup parameter of kubelet:
      $ ps -ef | grep kubelet
      ... /usr/bin/kubelet --cluster-dns=172.16.14.217 ...
      Find the DNS Service:
      $ kubectl get svc -n kube-system | grep 172.16.14.217
      kube-dns              ClusterIP   172.16.14.217   <none>        53/TCP,53/UDP              47d
      Check for the endpoint:
      $ kubectl -n kube-system describe svc kube-dns | grep -i endpoints
      Endpoints:         172.16.0.156:53,172.16.0.167:53
      Endpoints:         172.16.0.156:53,172.16.0.167:53
      Check whether the Pod of the endpoint is normal:
      $ kubectl -n kube-system get pod -o wide | grep 172.16.0.156
      kube-dns-898dbbfc6-hvwlr            3/3       Running   0          8d        172.16.0.156   10.0.0.3

      2. Make sure that the Pod can communicate with the cluster DNS

      Check whether the Pod can connect to the cluster DNS. You can run the telnet command in the Pod to view port 53 of the DNS:
      # Cluster IP for connecting to the DNS Service
      $ telnet 172.16.14.217 53
      Note:
      If there are no testing tools such as telnet in the container, you can use nsenter to enter netns for packet capturing and use telnet on the host for testing.
      If the network is found to be disconnected, check the following network settings:
      Check the security group settings of the node and the container IP range of the cluster that needs to be opened.
      Check for firewall rules and check the iptables.
      Check whether kube-proxy runs normally. The cluster DNS IP is the cluster IP, which is forwarded through the iptables or IPVS rules generated by kube-proxy.

      3. Capture packets

      If the cluster DNS runs normally and the Pod can communicate with the cluster DNS, capture packets for further checks. If the problem can be easily reproduced, you can use nsenter to enter netns to capture container packets:
      tcpdump -i any port 53 -w dns.pcap
      # tcpdump -i any port 53 -nn -tttt
      If the cause still cannot be identified, you can capture packets at multiple points along the request linkage for analysis, such as Pod container, host cbr0 bridge, primary ENI of the host (eth0), primary ENI of the host of the CoreDNS Pod, cbr0, and container. Wait for the problem to recur and locate the point where the packet is lost.

      Issue and Cause

      Latency of five seconds

      If it often takes five seconds to return a DNS query result, packets are usually lost due to kernel conntrack conflicts. The root cause is the bug in the conntrack module, where some packets are discarded due to resource competition when netfilter is used for NAT.
      It may possibly occur when multiple threads or processes send the same quintuple UDP packet through the same socket concurrently.
      Both glibc and musl (Alpine Linux's libc) use "parallel query", i.e., multiple query requests are sent concurrently, which tends to cause conflicts and request discarding.
      As IPVS also uses conntrack, this problem cannot be avoided in IPVS mode of kube-proxy.

      Workaround

      Use local DNS. DNS requests of the container are sent to the local DNS cache service (dnsmasq, nscd, etc.), without DNAT or conntrack conflicts. In addition, the DNS service will not be a performance bottleneck. You can use local DNS in two ways:
      Each container comes with a DNS cache service.
      Each node runs a DNS cache service, and all containers use the DNS cache of the node as their nameserver.

      Timeout of the resolution of an external domain name

      Possible reasons:
      The upstream DNS fails.
      The ACL or firewall of the upstream DNS blocks the packet.

      Timeout of all resolutions

      If a cluster Pod fails to resolve both Services and external domain names, there is generally a problem with the communication between the Pod and the cluster DNS. Possible reasons:
      The node firewall doesn't open the cluster IP range; therefore, the Pod cannot communicate with that of the cluster DNS when they are on different nodes, and DNS requests cannot be received.
      kube-proxy is abnormal.
      Contact Us

      Contact our sales team or business advisors to help your business.

      Contact Us
      Technical Support

      Open a ticket if you're looking for further assistance. Our Ticket is 7x24 avaliable.

      Submit a Ticket
      7x24 Phone Support
      Copyright © 2013-2024 Tencent Cloud. All Rights Reserved.
      Privacy PolicyLegalCookie Policy