tencent cloud

PrevNext

masukan

cari berdasarkan kata kunci

Recent Pages

Dokumentasi
Tencent Kubernetes Engine
    Dokumentasi
    Download PDF

    Elastic Scaling with EKS for IDC-Based Cluster

    Terakhir diperbarui:2024-12-24 16:44:40
    Unduh PDF

      Use Cases

      As your IDC resources may be limited, if you need to handle business traffic surges, the computing resources in your IDC may be insufficient to meet the requirements. In this case, you can use public cloud resources to handle temporary traffic. Based on custom scheduling policies and by leveraging TKE Serverless Container Service, TKE Resilience Chart adds supernodes to elastically migrate workloads in your IDC cluster to the cloud, so your cluster can get greater elastic scalability and enjoy the following benefits:
      1. The hardware and maintenance costs of your IDC/private cloud do not increase.
      2. You can implement high availability for applications at the IDC/private cloud grade and public cloud grade.
      3. You can use public cloud resources as needed in a pay-as-you-go manner.

      Notes

      1. You have activated TKE Serverless cluster.
      2. Your IDC is connected to a VPC through Direct Connect over the private network.
      3. The address of the API server in the IDC cluster can be accessed over the VPC.
      4. Your own IDC cluster can access the public network, as it needs to call TencentCloud APIs over the public network.

      TKE Resilience Chart Feature Description

      Component description

      TKE Resilience Chart mainly consists of a supernode manager, scheduler, and toleration controller as detailed below:
      Alias
      Component Name
      Description
      eklet
      Supernodes manager
      It manages the lifecycle of PodSandboxes and provides APIs related to native kubelet and nodes.
      tke-scheduler
      Scheduler
      It migrates workloads to the cloud elastically according to scheduling policies and is only installed in non-TKE Kubernetes Distro K8s clusters. TKE Kubernetes Distro is a K8s distribution released by TKE to help you create exactly the same K8s cluster in TKE. Currently, it has been open sourced at GitHub. For more information, please see TKE Kubernetes Distro.
      admission-controller
      Toleration controller
      It adds a toleration to a Pod in pending status to make it able to be scheduled to a supernode.

      Main features

      1. If you want to connect an TKE Serverless Pod to a Pod in your local cluster, the local cluster should be in an underlay network model (where a CNI plugin based on BGP routing instead of SDN encapsulation, such as Calico, is used), and you need to add the local Pod's CIDR block routing information in the VPC. For more information, see Interconnection Between Cluster in GlobalRouter Mode and IDC.
      2. The workload resilience feature switch AUTO_SCALE_EKS=true|false is available in global and local dimensions respectively to control whether workloads in pending status should be elastically scheduled to EKS as detailed below:
      Global switch: AUTO_SCALE_EKS in kubectl get cm -n kube-system eks-config is enabled by default.
      Local switch: spec.template.metadata.annotations ['AUTO_SCALE_EKS']
      Global Switch
      Local Switch
      Behavior
      AUTO_SCALE_EKS=true
      AUTO_SCALE_EKS=false
      Successfully scheduled
      AUTO_SCALE_EKS=true
      Undefined
      Successfully scheduled
      AUTO_SCALE_EKS=true
      AUTO_SCALE_EKS=true
      Successfully scheduled
      AUTO_SCALE_EKS=false
      AUTO_SCALE_EKS=false
      Failed to be scheduled
      AUTO_SCALE_EKS=false
      Undefined
      Failed to be scheduled
      AUTO_SCALE_EKS=false
      AUTO_SCALE_EKS=true
      Successfully scheduled
      Undefined
      AUTO_SCALE_EKS=false
      Successfully scheduled
      Undefined
      Undefined
      Successfully scheduled
      Undefined
      AUTO_SCALE_EKS=true
      Successfully scheduled
      3. If you use K8s community edition, you need to specify the scheduler as tke-scheduler in workloads. In TKE Kubernetes Distro, you don't need to specify the scheduler.
      4. In the workloads, set the number of retained replicas in the local cluster through LOCAL_REPLICAS: N.
      5. Workload scale-out:
      If the local cluster resources are insufficient and the settings of the global and local switches for the "successfully scheduled" behavior are satisfied, workloads in pending status will be scaled out to EKS.
      If the number of actually created workload replicas reaches N and the settings of the global and local switches for the "successfully scheduled" behavior are satisfied, workloads in pending status will be scaled out to TKE Serverless cluster.
      6. Workload scale-in:
      For TKE Kubernetes Distro, instances in TKE Serverless cluster will be scaled in preferentially.
      For K8s community edition, workloads will be scaled in randomly.
      7. Scheduling rule restrictions:
      DaemonSet Pods cannot be scheduled to supernodes. This feature is available only in TKE Kubernetes Distro. In K8s community edition, DaemonSet Pods will be scheduled to supernodes but DaemonsetForbidden will be displayed.
      Pods in kube-system and tke-eni-ip-webhook namespaces cannot be scheduled to supernodes.
      Ports whose securityContext.sysctls ["net.ipv4.ip_local_port_range"] value includes 61000–65534 cannot be scheduled.
      Pods in Pod.Annotations [tke.cloud.tencent.com/vpc-ip-claim-delete-policy] cannot be scheduled.
      Ports whose container (initContainer).ports [].containerPort (hostPort) value includes 61000–65534 cannot be scheduled.
      Ports with a container (initContainer) where the probe points to 61000–65534 cannot be scheduled.
      PersistentVolumes (PVs) except nfs, Cephfs, hostPath, and qcloudcbs cannot be scheduled.
      Pods with fixed IP enabled cannot be scheduled to supernodes.
      8. Supernodes support custom DNS configuration: after you add the eks.tke.cloud.tencent.com/resolv-conf annotation to a supernode, /etc/resolv.conf in the generated CVM instance will be updated to the custom content.
      Note:
      The original DNS configuration on the supernodes will be overwritten, and your custom configuration will prevail.
      eks.tke.cloud.tencent.com/resolv-conf: |
      nameserver 4.4.4.4
      nameserver 8.8.8.8

      Directions

      Getting tke-resilience helm chart

       git clone https://github.com/tkestack/charts.git

      Configuring relevant information

      Edit charts/incubator/tke-resilience/values.yaml and configure the following information:
      cloud:
        appID: "{Tencent Cloud account APPID}"
        ownerUIN: "{Tencent Cloud account ID}"
        secretID: "{Tencent Cloud account secretID}"
        secretKey: "{Tencent Cloud account secretKey}"
        vpcID: "{ID of the VPC where the EKS Pod resides}"
        regionShort: "{Abbreviation of the region where the EKS Pod resides}"
        regionLong: "{Full name of the region where the EKS Pod resides}"
        subnets:
          - id: "{ID of the subnet where the EKS Pod resides}"
            zone: "{AZ where the EKS Pod resides}" 
      eklet:
        PodUsedApiserver: "{API server address of the current cluster}"
      Note:
      For more information on the regions and AZs where TKE Serverless container service is available, please see Regions and AZs.

      Installing TKE Resilience Chart

      Run the following command to use a Helm chart to install TKE Resilience Chart in a third-party cluster:
      helm install tke-resilience --namespace kube-system ./tke-resilience --debug
      Run the following command to check whether the required components in the Helm application are installed. This document uses a TKE Kubernetes Distro cluster with no tke-scheduler installed as an example.
      # kubectl get Pod -n kube-system | grep resilience
      eklet-tke-resilience-5f9dcd99df-rgsmc           1/1     Running   0          43h
      eks-admission-tke-resilience-5bb588dc44-9hvhs   1/1     Running   0          44h
      You can see that one supernode has been deployed in the cluster.
      # kubectl get node
      NAME                    STATUS   ROLES    AGE    VERSION
      10.0.1.xx               Ready    <none>   2d4h   v1.20.4-tke.1
      10.0.1.xx               Ready    master   2d4h   v1.20.4-tke.1
      eklet-subnet-xxxxxxxx   Ready    <none>   43h    v2.4.6

      Creating test case

      Create a demo application nginx-deployment, which has four replicas (three in TKE Serverless cluster and one in the local cluster). Below is the sample YAML configuration:
      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: nginx-deployment
        labels:
          app: nginx
      spec:
        replicas: 4
        strategy:
          type: RollingUpdate
        selector:
          matchLabels:
            app: nginx
        template:
          metadata:
            annotations:
              AUTO_SCALE_EKS: "true"
              LOCAL_REPLICAS: "1" # Set the number of running replicas in the local cluster to 1
            labels:
              app: nginx
          spec:
            #schedulerName: tke-scheduler If it is a third-party cluster, you need to run the scheduler as `tke-scheduler`
            containers:
            - name: nginx
              image: nginx
              imagePullPolicy: IfNotPresent
      Check whether the replica status and distribution meet the expectations.
      # kubectl get Pod -owide
      NAME                                READY   STATUS    RESTARTS   AGE   IP            NODE                    NOMINATED NODE   READINESS GATES
      nginx-deployment-77b9b9bc97-cq9ds   1/1     Running   0          27s   10.232.1.88   10.0.1.xxx              <none>           <none>
      nginx-deployment-77b9b9bc97-s9vzc   1/1     Running   0          27s   10.0.1.118    eklet-subnet-xxxxxxxx   <none>           <none>
      nginx-deployment-77b9b9bc97-sd4z5   1/1     Running   0          27s   10.0.1.7      eklet-subnet-xxxxxxxx   <none>           <none>
      nginx-deployment-77b9b9bc97-z86tx   1/1     Running   0          27s   10.0.1.133    eklet-subnet-xxxxxxxx   <none>           <none>
      Check the scale-in feature. As a TKE Kubernetes Distro cluster is used, TKE Serverless cluster instances will be scaled in preferentially. Here, the number of application replicas is adjusted from 4 to 3.
      # kubectl scale deployment nginx-deployment --replicas=3
      As shown below, replicas in Tencent Cloud are scaled in first, which meets the expectation:
      # kubectl get Pod -owide
      NAME                                READY   STATUS    RESTARTS   AGE     IP            NODE                    NOMINATED NODE   READINESS GATES
      nginx-deployment-77b9b9bc97-cq9ds   1/1     Running   0          7m38s   10.232.1.88   10.0.1.xxx              <none>           <none>
      nginx-deployment-77b9b9bc97-s9vzc   1/1     Running   0          7m38s   10.0.1.118    eklet-subnet-xxxxxxxx   <none>           <none>
      nginx-deployment-77b9b9bc97-sd4z5   1/1     Running   0          7m38s   10.0.1.7      eklet-subnet-xxxxxxxx   <none>           <none>
      
      Hubungi Kami

      Hubungi tim penjualan atau penasihat bisnis kami untuk membantu bisnis Anda.

      Hubungi Kami
      Dukungan Teknis

      Buka tiket jika Anda mencari bantuan lebih lanjut. Tiket kami tersedia 7x24.

      Kirim Tiket
      Dukungan Telepon 7x24
      Copyright © 2013-2025 Tencent Cloud. All Rights Reserved.
      Kebijakan PrivasiLegalKebijakan Cookie