tencent cloud

PrevNext

masukan

cari berdasarkan kata kunci

Recent Pages

Dokumentasi
Tencent Kubernetes Engine
    Dokumentasi
    Download PDF

    Cluster DNS Troubleshooting

    Terakhir diperbarui:2024-12-13 14:48:39
    Unduh PDF

      Troubleshooting Approaches

      1. Make sure that the cluster DNS runs normally

      Container DNS passes through cluster DNS (usually CoreDNS). First, make sure that the cluster DNS runs normally. You can see the cluster IP of the DNS in the --cluster-dns startup parameter of kubelet:
      $ ps -ef | grep kubelet
      ... /usr/bin/kubelet --cluster-dns=172.16.14.217 ...
      Find the DNS Service:
      $ kubectl get svc -n kube-system | grep 172.16.14.217
      kube-dns              ClusterIP   172.16.14.217   <none>        53/TCP,53/UDP              47d
      Check for the endpoint:
      $ kubectl -n kube-system describe svc kube-dns | grep -i endpoints
      Endpoints:         172.16.0.156:53,172.16.0.167:53
      Endpoints:         172.16.0.156:53,172.16.0.167:53
      Check whether the Pod of the endpoint is normal:
      $ kubectl -n kube-system get pod -o wide | grep 172.16.0.156
      kube-dns-898dbbfc6-hvwlr            3/3       Running   0          8d        172.16.0.156   10.0.0.3

      2. Make sure that the Pod can communicate with the cluster DNS

      Check whether the Pod can connect to the cluster DNS. You can run the telnet command in the Pod to view port 53 of the DNS:
      # Cluster IP for connecting to the DNS Service
      $ telnet 172.16.14.217 53
      Note:
      If there are no testing tools such as telnet in the container, you can use nsenter to enter netns for packet capturing and use telnet on the host for testing.
      If the network is found to be disconnected, check the following network settings:
      Check the security group settings of the node and the container IP range of the cluster that needs to be opened.
      Check for firewall rules and check the iptables.
      Check whether kube-proxy runs normally. The cluster DNS IP is the cluster IP, which is forwarded through the iptables or IPVS rules generated by kube-proxy.

      3. Capture packets

      If the cluster DNS runs normally and the Pod can communicate with the cluster DNS, capture packets for further checks. If the problem can be easily reproduced, you can use nsenter to enter netns to capture container packets:
      tcpdump -i any port 53 -w dns.pcap
      # tcpdump -i any port 53 -nn -tttt
      If the cause still cannot be identified, you can capture packets at multiple points along the request linkage for analysis, such as Pod container, host cbr0 bridge, primary ENI of the host (eth0), primary ENI of the host of the CoreDNS Pod, cbr0, and container. Wait for the problem to recur and locate the point where the packet is lost.

      Issue and Cause

      Latency of five seconds

      If it often takes five seconds to return a DNS query result, packets are usually lost due to kernel conntrack conflicts. The root cause is the bug in the conntrack module, where some packets are discarded due to resource competition when netfilter is used for NAT.
      It may possibly occur when multiple threads or processes send the same quintuple UDP packet through the same socket concurrently.
      Both glibc and musl (Alpine Linux's libc) use "parallel query", i.e., multiple query requests are sent concurrently, which tends to cause conflicts and request discarding.
      As IPVS also uses conntrack, this problem cannot be avoided in IPVS mode of kube-proxy.

      Workaround

      Use local DNS. DNS requests of the container are sent to the local DNS cache service (dnsmasq, nscd, etc.), without DNAT or conntrack conflicts. In addition, the DNS service will not be a performance bottleneck. You can use local DNS in two ways:
      Each container comes with a DNS cache service.
      Each node runs a DNS cache service, and all containers use the DNS cache of the node as their nameserver.

      Timeout of the resolution of an external domain name

      Possible reasons:
      The upstream DNS fails.
      The ACL or firewall of the upstream DNS blocks the packet.

      Timeout of all resolutions

      If a cluster Pod fails to resolve both Services and external domain names, there is generally a problem with the communication between the Pod and the cluster DNS. Possible reasons:
      The node firewall doesn't open the cluster IP range; therefore, the Pod cannot communicate with that of the cluster DNS when they are on different nodes, and DNS requests cannot be received.
      kube-proxy is abnormal.
      Hubungi Kami

      Hubungi tim penjualan atau penasihat bisnis kami untuk membantu bisnis Anda.

      Hubungi Kami
      Dukungan Teknis

      Buka tiket jika Anda mencari bantuan lebih lanjut. Tiket kami tersedia 7x24.

      Kirim Tiket
      Dukungan Telepon 7x24
      Copyright © 2013-2024 Tencent Cloud. All Rights Reserved.
      Kebijakan PrivasiLegalKebijakan Cookie