- Release Notes and Announcements
- Release Notes
- Announcements
- Security Vulnerability Fix Description
- Announcement on Authentication Upgrade of Some TKE APIs
- Discontinuing Update of NginxIngress Addon
- qGPU Service Adjustment
- Version Upgrade of Master Add-On of TKE Managed Cluster
- Upgrading tke-monitor-agent
- Discontinuing TKE API 2.0
- Instructions on Cluster Resource Quota Adjustment
- Decommissioning Kubernetes Version
- Deactivation of Scaling Group Feature
- Notice on TPS Discontinuation on May 16, 2022 at 10:00 (UTC +8)
- Basic Monitoring Architecture Upgrade
- Starting Charging on Managed Clusters
- Instructions on Stopping Delivering the Kubeconfig File to Nodes
- Release Notes
- Product Introduction
- Purchase Guide
- Quick Start
- TKE General Cluster Guide
- TKE General Cluster Overview
- Purchase a TKE General Cluster
- High-risk Operations of Container Service
- Deploying Containerized Applications in the Cloud
- Open Source Components
- Permission Management
- Cluster Management
- Cluster Overview
- Cluster Hosting Modes Introduction
- Cluster Lifecycle
- Creating a Cluster
- Changing the Cluster Operating System
- Deleting a Cluster
- Cluster Scaling
- Connecting to a Cluster
- Upgrading a Cluster
- Enabling IPVS for a Cluster
- Custom Kubernetes Component Launch Parameters
- Using KMS for Kubernetes Data Source Encryption
- Images
- Worker node introduction
- Normal Node Management
- Native Node Management
- Overview
- Native Node Parameters
- Purchasing Native Nodes
- Lifecycle of a Native Node
- Creating Native Nodes
- Modifying Native Nodes
- Deleting Native Nodes
- Self-Heal Rules
- Declarative Operation Practice
- Native Node Scaling
- In-place Pod Configuration Adjustment
- Enabling Public Network Access for a Native Node
- Management Parameters
- Enabling SSH Key Login for a Native Node
- FAQs for Native Nodes
- Supernode management
- Registered Node Management
- Memory Compression Instructions
- GPU Share
- Kubernetes Object Management
- Overview
- Namespace
- Workload
- Deployment Management
- StatefulSet Management
- DaemonSet Management
- CronJob Management
- Job Management
- Setting the Resource Limit of Workload
- Setting the Scheduling Rule for a Workload
- Setting the Health Check for a Workload
- Setting the Run Command and Parameter for a Workload
- Using a Container Image in a TCR Enterprise Instance to Create a Workload
- Auto Scaling
- Configuration
- Service Management
- Ingress Management
- Storage Management
- Policy Management
- Application and Add-On Feature Management Description
- Add-On Management
- Add-on Overview
- Add-On Lifecycle Management
- Cluster Autoscaler
- OOMGuard
- NodeProblemDetectorPlus Add-on
- NodeLocalDNSCache
- DNSAutoscaler
- COS-CSI
- CFS-CSI
- CFSTURBO-CSI
- CBS-CSI Description
- UserGroupAccessControl
- TCR Introduction
- TCR Hosts Updater
- DynamicScheduler
- DeScheduler
- Network Policy
- Nginx-ingress
- HPC
- Description of tke-monitor-agent
- tke-log-agent
- GPU-Manager Add-on
- Helm Application
- Application Market
- Network Management
- Container Network Overview
- GlobalRouter Mode
- VPC-CNI Mode
- VPC-CNI Mode
- Multiple Pods with Shared ENI Mode
- Pods with Exclusive ENI Mode
- Static IP Address Mode Instructions
- Non-static IP Address Mode Instructions
- Interconnection Between VPC-CNI and Other Cloud Resources/IDC Resources
- Security Group of VPC-CNI Mode
- Instructions on Binding an EIP to a Pod
- VPC-CNI Component Description
- Limits on the Number of Pods in VPC-CNI Mode
- Cilium-Overlay Mode
- OPS Center
- Log Management
- Backup Center
- Remote Terminals
- TKE Serverless Cluster Guide
- TKE Registered Cluster Guide
- TKE Insight
- TKE Scheduling
- Cloud Native Service Guide
- Practical Tutorial
- Cluster
- Cluster Migration
- Serverless Cluster
- Security
- Service Deployment
- Network
- DNS
- Self-Built Nginx Ingress Practice Tutorial
- Quick Start
- Custom Load Balancer
- Enabling CLB Direct Connection
- Optimization for High Concurrency Scenarios
- High Availability Configuration Optimization
- Observability Integration
- Access to Tencent Cloud WAF
- Installing Multiple Nginx Ingress Controllers
- Migrating from TKE Nginx Ingress Plugin to Self-Built Nginx Ingress
- Complete Example of values.yaml Configuration
- Using Network Policy for Network Access Control
- Deploying NGINX Ingress on TKE
- Nginx Ingress High-Concurrency Practices
- Nginx Ingress Best Practices
- Limiting the bandwidth on pods in TKE
- Directly connecting TKE to the CLB of pods based on the ENI
- Use CLB-Pod Direct Connection on TKE
- Obtaining the Real Client Source IP in TKE
- Using Traefik Ingress in TKE
- Release
- Logs
- Monitoring
- OPS
- Removing and Re-adding Nodes from and to Cluster
- Using Ansible to Batch Operate TKE Nodes
- Using Cluster Audit for Troubleshooting
- Renewing a TKE Ingress Certificate
- Using cert-manager to Issue Free Certificates
- Using cert-manager to Issue Free Certificate for DNSPod Domain Name
- Using the TKE NPDPlus Plug-In to Enhance the Self-Healing Capability of Nodes
- Using kubecm to Manage Multiple Clusters kubeconfig
- Quick Troubleshooting Using TKE Audit and Event Services
- Customizing RBAC Authorization in TKE
- Clearing De-registered Tencent Cloud Account Resources
- Terraform
- DevOps
- Auto Scaling
- Cluster Auto Scaling Practices
- Using tke-autoscaling-placeholder to Implement Auto Scaling in Seconds
- Installing metrics-server on TKE
- Using Custom Metrics for Auto Scaling in TKE
- Utilizing HPA to Auto Scale Businesses on TKE
- Using VPA to Realize Pod Scaling up and Scaling down in TKE
- Adjusting HPA Scaling Sensitivity Based on Different Business Scenarios
- Implementing elasticity based on traffic prediction with EHPA
- Implementing Horizontal Scaling based on CLB monitoring metrics using KEDA in TKE
- Containerization
- Microservice
- Cost Management
- Hybrid Cloud
- Fault Handling
- Disk Full
- High Workload
- Memory Fragmentation
- Cluster DNS Troubleshooting
- Cluster kube-proxy Troubleshooting
- Cluster API Server Inaccessibility Troubleshooting
- Service and Ingress Inaccessibility Troubleshooting
- Common Service & Ingress Errors and Solutions
- Engel Ingres appears in Connechtin Reverside
- CLB Ingress Creation Error
- Troubleshooting for Pod Network Inaccessibility
- Pod Status Exception and Handling
- Authorizing Tencent Cloud OPS Team for Troubleshooting
- CLB Loopback
- API Documentation
- History
- API Category
- Cluster APIs
- DescribeEncryptionStatus
- DisableEncryptionProtection
- EnableEncryptionProtection
- UpdateClusterVersion
- AcquireClusterAdminRole
- CreateClusterEndpoint
- CreateClusterEndpointVip
- DeleteCluster
- DeleteClusterEndpoint
- DeleteClusterEndpointVip
- DescribeAvailableClusterVersion
- DescribeClusterAuthenticationOptions
- DescribeClusterCommonNames
- DescribeClusterEndpointStatus
- DescribeClusterEndpointVipStatus
- DescribeClusterEndpoints
- DescribeClusterKubeconfig
- DescribeClusterLevelAttribute
- DescribeClusterLevelChangeRecords
- DescribeClusterSecurity
- DescribeClusterStatus
- DescribeClusters
- DescribeEdgeAvailableExtraArgs
- DescribeEdgeClusterExtraArgs
- DescribeResourceUsage
- DisableClusterDeletionProtection
- EnableClusterDeletionProtection
- GetClusterLevelPrice
- GetUpgradeInstanceProgress
- ModifyClusterAttribute
- ModifyClusterAuthenticationOptions
- ModifyClusterEndpointSP
- UpgradeClusterInstances
- CreateCluster
- UpdateClusterKubeconfig
- DescribeBackupStorageLocations
- DeleteBackupStorageLocation
- CreateBackupStorageLocation
- Node Pool APIs
- TKE Edge Cluster APIs
- DescribeTKEEdgeScript
- UpdateEdgeClusterVersion
- CreateTKEEdgeCluster
- DescribeTKEEdgeExternalKubeconfig
- DescribeTKEEdgeClusters
- DescribeTKEEdgeClusterStatus
- DescribeTKEEdgeClusterCredential
- DescribeEdgeClusterInstances
- DescribeEdgeCVMInstances
- DescribeECMInstances
- DescribeAvailableTKEEdgeVersion
- DeleteTKEEdgeCluster
- DeleteEdgeClusterInstances
- DeleteEdgeCVMInstances
- DeleteECMInstances
- CreateECMInstances
- CheckEdgeClusterCIDR
- ForwardTKEEdgeApplicationRequestV3
- UninstallEdgeLogAgent
- InstallEdgeLogAgent
- DescribeEdgeLogSwitches
- CreateEdgeLogConfig
- CreateEdgeCVMInstances
- DescribeEdgeClusterUpgradeInfo
- Scaling group APIs
- Introduction
- Making API Requests
- Add-on APIs
- Network APIs
- Node APIs
- Cloud Native Monitoring APIs
- Virtual node APIs
- Other APIs
- Data Types
- Error Codes
- API Mapping Guide
- FAQs
- Service Agreement
- Contact Us
- Glossary
- User Guide(Old)
- Release Notes and Announcements
- Release Notes
- Announcements
- Security Vulnerability Fix Description
- Announcement on Authentication Upgrade of Some TKE APIs
- Discontinuing Update of NginxIngress Addon
- qGPU Service Adjustment
- Version Upgrade of Master Add-On of TKE Managed Cluster
- Upgrading tke-monitor-agent
- Discontinuing TKE API 2.0
- Instructions on Cluster Resource Quota Adjustment
- Decommissioning Kubernetes Version
- Deactivation of Scaling Group Feature
- Notice on TPS Discontinuation on May 16, 2022 at 10:00 (UTC +8)
- Basic Monitoring Architecture Upgrade
- Starting Charging on Managed Clusters
- Instructions on Stopping Delivering the Kubeconfig File to Nodes
- Release Notes
- Product Introduction
- Purchase Guide
- Quick Start
- TKE General Cluster Guide
- TKE General Cluster Overview
- Purchase a TKE General Cluster
- High-risk Operations of Container Service
- Deploying Containerized Applications in the Cloud
- Open Source Components
- Permission Management
- Cluster Management
- Cluster Overview
- Cluster Hosting Modes Introduction
- Cluster Lifecycle
- Creating a Cluster
- Changing the Cluster Operating System
- Deleting a Cluster
- Cluster Scaling
- Connecting to a Cluster
- Upgrading a Cluster
- Enabling IPVS for a Cluster
- Custom Kubernetes Component Launch Parameters
- Using KMS for Kubernetes Data Source Encryption
- Images
- Worker node introduction
- Normal Node Management
- Native Node Management
- Overview
- Native Node Parameters
- Purchasing Native Nodes
- Lifecycle of a Native Node
- Creating Native Nodes
- Modifying Native Nodes
- Deleting Native Nodes
- Self-Heal Rules
- Declarative Operation Practice
- Native Node Scaling
- In-place Pod Configuration Adjustment
- Enabling Public Network Access for a Native Node
- Management Parameters
- Enabling SSH Key Login for a Native Node
- FAQs for Native Nodes
- Supernode management
- Registered Node Management
- Memory Compression Instructions
- GPU Share
- Kubernetes Object Management
- Overview
- Namespace
- Workload
- Deployment Management
- StatefulSet Management
- DaemonSet Management
- CronJob Management
- Job Management
- Setting the Resource Limit of Workload
- Setting the Scheduling Rule for a Workload
- Setting the Health Check for a Workload
- Setting the Run Command and Parameter for a Workload
- Using a Container Image in a TCR Enterprise Instance to Create a Workload
- Auto Scaling
- Configuration
- Service Management
- Ingress Management
- Storage Management
- Policy Management
- Application and Add-On Feature Management Description
- Add-On Management
- Add-on Overview
- Add-On Lifecycle Management
- Cluster Autoscaler
- OOMGuard
- NodeProblemDetectorPlus Add-on
- NodeLocalDNSCache
- DNSAutoscaler
- COS-CSI
- CFS-CSI
- CFSTURBO-CSI
- CBS-CSI Description
- UserGroupAccessControl
- TCR Introduction
- TCR Hosts Updater
- DynamicScheduler
- DeScheduler
- Network Policy
- Nginx-ingress
- HPC
- Description of tke-monitor-agent
- tke-log-agent
- GPU-Manager Add-on
- Helm Application
- Application Market
- Network Management
- Container Network Overview
- GlobalRouter Mode
- VPC-CNI Mode
- VPC-CNI Mode
- Multiple Pods with Shared ENI Mode
- Pods with Exclusive ENI Mode
- Static IP Address Mode Instructions
- Non-static IP Address Mode Instructions
- Interconnection Between VPC-CNI and Other Cloud Resources/IDC Resources
- Security Group of VPC-CNI Mode
- Instructions on Binding an EIP to a Pod
- VPC-CNI Component Description
- Limits on the Number of Pods in VPC-CNI Mode
- Cilium-Overlay Mode
- OPS Center
- Log Management
- Backup Center
- Remote Terminals
- TKE Serverless Cluster Guide
- TKE Registered Cluster Guide
- TKE Insight
- TKE Scheduling
- Cloud Native Service Guide
- Practical Tutorial
- Cluster
- Cluster Migration
- Serverless Cluster
- Security
- Service Deployment
- Network
- DNS
- Self-Built Nginx Ingress Practice Tutorial
- Quick Start
- Custom Load Balancer
- Enabling CLB Direct Connection
- Optimization for High Concurrency Scenarios
- High Availability Configuration Optimization
- Observability Integration
- Access to Tencent Cloud WAF
- Installing Multiple Nginx Ingress Controllers
- Migrating from TKE Nginx Ingress Plugin to Self-Built Nginx Ingress
- Complete Example of values.yaml Configuration
- Using Network Policy for Network Access Control
- Deploying NGINX Ingress on TKE
- Nginx Ingress High-Concurrency Practices
- Nginx Ingress Best Practices
- Limiting the bandwidth on pods in TKE
- Directly connecting TKE to the CLB of pods based on the ENI
- Use CLB-Pod Direct Connection on TKE
- Obtaining the Real Client Source IP in TKE
- Using Traefik Ingress in TKE
- Release
- Logs
- Monitoring
- OPS
- Removing and Re-adding Nodes from and to Cluster
- Using Ansible to Batch Operate TKE Nodes
- Using Cluster Audit for Troubleshooting
- Renewing a TKE Ingress Certificate
- Using cert-manager to Issue Free Certificates
- Using cert-manager to Issue Free Certificate for DNSPod Domain Name
- Using the TKE NPDPlus Plug-In to Enhance the Self-Healing Capability of Nodes
- Using kubecm to Manage Multiple Clusters kubeconfig
- Quick Troubleshooting Using TKE Audit and Event Services
- Customizing RBAC Authorization in TKE
- Clearing De-registered Tencent Cloud Account Resources
- Terraform
- DevOps
- Auto Scaling
- Cluster Auto Scaling Practices
- Using tke-autoscaling-placeholder to Implement Auto Scaling in Seconds
- Installing metrics-server on TKE
- Using Custom Metrics for Auto Scaling in TKE
- Utilizing HPA to Auto Scale Businesses on TKE
- Using VPA to Realize Pod Scaling up and Scaling down in TKE
- Adjusting HPA Scaling Sensitivity Based on Different Business Scenarios
- Implementing elasticity based on traffic prediction with EHPA
- Implementing Horizontal Scaling based on CLB monitoring metrics using KEDA in TKE
- Containerization
- Microservice
- Cost Management
- Hybrid Cloud
- Fault Handling
- Disk Full
- High Workload
- Memory Fragmentation
- Cluster DNS Troubleshooting
- Cluster kube-proxy Troubleshooting
- Cluster API Server Inaccessibility Troubleshooting
- Service and Ingress Inaccessibility Troubleshooting
- Common Service & Ingress Errors and Solutions
- Engel Ingres appears in Connechtin Reverside
- CLB Ingress Creation Error
- Troubleshooting for Pod Network Inaccessibility
- Pod Status Exception and Handling
- Authorizing Tencent Cloud OPS Team for Troubleshooting
- CLB Loopback
- API Documentation
- History
- API Category
- Cluster APIs
- DescribeEncryptionStatus
- DisableEncryptionProtection
- EnableEncryptionProtection
- UpdateClusterVersion
- AcquireClusterAdminRole
- CreateClusterEndpoint
- CreateClusterEndpointVip
- DeleteCluster
- DeleteClusterEndpoint
- DeleteClusterEndpointVip
- DescribeAvailableClusterVersion
- DescribeClusterAuthenticationOptions
- DescribeClusterCommonNames
- DescribeClusterEndpointStatus
- DescribeClusterEndpointVipStatus
- DescribeClusterEndpoints
- DescribeClusterKubeconfig
- DescribeClusterLevelAttribute
- DescribeClusterLevelChangeRecords
- DescribeClusterSecurity
- DescribeClusterStatus
- DescribeClusters
- DescribeEdgeAvailableExtraArgs
- DescribeEdgeClusterExtraArgs
- DescribeResourceUsage
- DisableClusterDeletionProtection
- EnableClusterDeletionProtection
- GetClusterLevelPrice
- GetUpgradeInstanceProgress
- ModifyClusterAttribute
- ModifyClusterAuthenticationOptions
- ModifyClusterEndpointSP
- UpgradeClusterInstances
- CreateCluster
- UpdateClusterKubeconfig
- DescribeBackupStorageLocations
- DeleteBackupStorageLocation
- CreateBackupStorageLocation
- Node Pool APIs
- TKE Edge Cluster APIs
- DescribeTKEEdgeScript
- UpdateEdgeClusterVersion
- CreateTKEEdgeCluster
- DescribeTKEEdgeExternalKubeconfig
- DescribeTKEEdgeClusters
- DescribeTKEEdgeClusterStatus
- DescribeTKEEdgeClusterCredential
- DescribeEdgeClusterInstances
- DescribeEdgeCVMInstances
- DescribeECMInstances
- DescribeAvailableTKEEdgeVersion
- DeleteTKEEdgeCluster
- DeleteEdgeClusterInstances
- DeleteEdgeCVMInstances
- DeleteECMInstances
- CreateECMInstances
- CheckEdgeClusterCIDR
- ForwardTKEEdgeApplicationRequestV3
- UninstallEdgeLogAgent
- InstallEdgeLogAgent
- DescribeEdgeLogSwitches
- CreateEdgeLogConfig
- CreateEdgeCVMInstances
- DescribeEdgeClusterUpgradeInfo
- Scaling group APIs
- Introduction
- Making API Requests
- Add-on APIs
- Network APIs
- Node APIs
- Cloud Native Monitoring APIs
- Virtual node APIs
- Other APIs
- Data Types
- Error Codes
- API Mapping Guide
- FAQs
- Service Agreement
- Contact Us
- Glossary
- User Guide(Old)
When deploying or running business, you may trigger high-risk operations at different levels, leading to service failures to different degrees. To help you estimate and avoid operational risks, this document describes the consequences of the high-risk operations and corresponding solutions. Below you can find the high-risk operations you may trigger when dealing with clusters, networking and load balancing, logs, and cloud disks.
Clusters
| Category | High-risk Operation | Consequence | Solution |
|---|---|---|---|
| Master and etcd nodes | Modifying the security groups of nodes in a cluster | Master node may become unavailable | Configure security groups as recommended by Tencent Cloud |
| Node expires or is terminated | The master node becomes unavailable | Unrecoverable | |
| Reinstalling operating system | Master components get deleted | Unrecoverable | |
| Upgrading master or etcd component version on your own | Cluster may become unavailable | Roll back to the original version | |
Deleting or formatting core directory data such as node /etc/kubernetes |
The master node becomes unavailable | Unrecoverable | |
| Changing node IP | The master node becomes unavailable | Change back to the old IP | |
| Modifying parameters of core components, e.g. etcd, kube-apiserver, docker, etc., on your own | Master node may become unavailable | Configure parameters as recommended by Tencent Cloud | |
| Changing master or etcd certificate on your own | Cluster may become unavailable | Unrecoverable | |
| Worker node | Modifying the security groups of nodes in a cluster | Nodes may become unavailable | Configure security groups as recommended by Tencent Cloud |
| Node expires or is terminated | The node becomes unavailable | Unrecoverable | |
| Reinstalling operating system | Node components get deleted | Remove the node and add it back to the cluster | |
| Upgrading node component version on your own | Node may become unavailable | Roll back to the original version | |
| Changing node IP | Node becomes unavailable | Change back to the old IP | |
| Modifying parameters of core components, e.g. etcd, kube-apiserver, docker, etc., on your own | Node may become unavailable | Configure parameters as recommended by Tencent Cloud | |
| Modifying operating system configuration | Node may become unavailable | Try to restore the configurations or delete the node and purchase a new one | |
| Others | Modifying permissions in CAM | Some cluster resources, such as cloud load balancers, may not be able to be created | Restore the permissions |
Networking and Load Balancing
| High-risk Operation | Consequence | Solution |
|---|---|---|
Modifying kernel parameters net.ipv4.ip_forward=0 |
Network not connected | Modify kernel parameters to net.ipv4.ip_forward=1 |
Modifying kernel parameter net.ipv4.tcp_tw_recycle = 1 |
NAT exception | Modify kernel parameter net.ipv4.tcp_tw_recycle = 0 |
| Container CIDR’s UDP port 53 is not opened to the Internet in the security group configuration of the node | In-cluster DNS cannot work normally | Configure security groups as recommended by Tencent Cloud |
| Modifying or deleting LB tags added in TKE | A new LB is purchased | Restore the LB tags |
| Creating custom listeners in TKE-managed LB through LB console | Modification gets reset by TKE | Automatically create listeners through service YAML |
| Binding custom backend rs in TKE-managed LB through LB console | Prohibit manual binding of backend rs | |
| Modifying certificate of TKE-managed LB through LB console | Automatically manage certificate through ingress YAML | |
| Modifying TKE-managed LB listener name through LB console | Prohibit modification of TKE-managed LB listener name |
Logs
| High-risk Operation | Consequence | Solution | Notes |
|---|---|---|---|
Deleting the /tmp/ccs-log-collector/pos directory of the host |
Log gets collected again | None | Files in Pod record where they are collected |
Deleting the /tmp/ccs-log-collector/buffer directory of the host |
Log gets lost | None | Buffer contains log cache file |
Cloud Disks
| High-risk Operation | Consequence | Solution |
|---|---|---|
| Manually unmounting cloud disks through console | Writing to Pod reports IO errors | Delete the mount directory of the node and reschedule the Pod |
| Unmounting disk mounting path on the node | Pod gets written to the local disk | Re-mount the corresponding directory onto Pod |
| Directly operating CBS block device on the node | Pod gets written to the local disk | None |
Yes
No
Was this page helpful?