- Product Overview
- Purchase Guide
- Quick Start
- Operation Guide
- Fault Action Library
- Compute
- JVM Process CPU at Full Load
- Cross-AZ Experiment in CVM
- CVM DNS Unavailability Experiment
- CVM Domain Name Parsing Tampering Experiment
- CVM System Time Skew
- CVM Disk IO Hang Fault Experiment
- CVM Memory OOM and Disk IO Load
- Experiments on CVM Intra-host Network Disorder
- CVM Kernel Faults
- Experiments on CVM Intra-host Network Latency
- High Utilization of CVM Resources (CPU, Memory, and Disk)
- Experiments on CVM Intra-host Network Corruption
- Experiments on CVM Intra-host Network Duplication
- Experiments on CVM Intra-host Network Occupation
- CVM Network Interruption
- CVM Ping Unreachable
- Cloud API Ban in CVM
- Database
- Primary-secondary Switch in TencentDB for PostgreSQL
- MySQL Instance Overall Unavailable
- Primary-secondary Switch in MySQL
- Setting Maximum Number of Connections in MySQL
- Primary Node Fault Experiment on TencentDB for MySQL
- TencentDB for MySQL Read-only Instance Group Unavailable
- Primary-secondary Switch in TDSQL for MySQM
- Primary-secondary Switch in MariaDB
- Primary-secondary Switch in TDSQL-C
- Practice of TencentDB for Redis Proxy Node Faults
- TencentDB for Redis Primary Node Fault
- Primary-secondary Nodes in TencentDB for Redis Instance Unavailable
- Simulating Primary-secondary Switch in Redis
- Simulating MongoDB Storage Node Fault
- Simulating Self-Built MySQL Crash Through Network Blocking
- Restarting TencentDB for MySQL
- Primary-secondary Switch in SQL Server
- Network
- Container
- Simulating Container Resource Network Faults
- Experiment on Container Resource Pod Operation Faults
- Experiment on Container Resource Node Faults
- Experiment on Container Resource Application Process Faults
- Standard Cluster and Serverless Cluster Super Node Faults
- Serverless Pod Fault Experiment Case
- Cluster Node Resource (CPU, Memory, Disk) Stress Test Faults
- Serverless Pod Virtual Node Shutdown Faults
- Simulating Serverless Cluster Pod Network Faults
- High Cluster Pod Resource (CPU, Memory, and Disk) Utilization Rate
- Cluster Pod TencentCloud API Ban
- Big Data
- Cloud Load Balancer
- Message Queue
- Direct Connect
- Custom Actions
- Cloud Streaming Services (CSS)
- Compute
- Permission Management Guide
- API Documentation
- FAQs
- Related Protocol
- Contact Us
- Product Overview
- Purchase Guide
- Quick Start
- Operation Guide
- Fault Action Library
- Compute
- JVM Process CPU at Full Load
- Cross-AZ Experiment in CVM
- CVM DNS Unavailability Experiment
- CVM Domain Name Parsing Tampering Experiment
- CVM System Time Skew
- CVM Disk IO Hang Fault Experiment
- CVM Memory OOM and Disk IO Load
- Experiments on CVM Intra-host Network Disorder
- CVM Kernel Faults
- Experiments on CVM Intra-host Network Latency
- High Utilization of CVM Resources (CPU, Memory, and Disk)
- Experiments on CVM Intra-host Network Corruption
- Experiments on CVM Intra-host Network Duplication
- Experiments on CVM Intra-host Network Occupation
- CVM Network Interruption
- CVM Ping Unreachable
- Cloud API Ban in CVM
- Database
- Primary-secondary Switch in TencentDB for PostgreSQL
- MySQL Instance Overall Unavailable
- Primary-secondary Switch in MySQL
- Setting Maximum Number of Connections in MySQL
- Primary Node Fault Experiment on TencentDB for MySQL
- TencentDB for MySQL Read-only Instance Group Unavailable
- Primary-secondary Switch in TDSQL for MySQM
- Primary-secondary Switch in MariaDB
- Primary-secondary Switch in TDSQL-C
- Practice of TencentDB for Redis Proxy Node Faults
- TencentDB for Redis Primary Node Fault
- Primary-secondary Nodes in TencentDB for Redis Instance Unavailable
- Simulating Primary-secondary Switch in Redis
- Simulating MongoDB Storage Node Fault
- Simulating Self-Built MySQL Crash Through Network Blocking
- Restarting TencentDB for MySQL
- Primary-secondary Switch in SQL Server
- Network
- Container
- Simulating Container Resource Network Faults
- Experiment on Container Resource Pod Operation Faults
- Experiment on Container Resource Node Faults
- Experiment on Container Resource Application Process Faults
- Standard Cluster and Serverless Cluster Super Node Faults
- Serverless Pod Fault Experiment Case
- Cluster Node Resource (CPU, Memory, Disk) Stress Test Faults
- Serverless Pod Virtual Node Shutdown Faults
- Simulating Serverless Cluster Pod Network Faults
- High Cluster Pod Resource (CPU, Memory, and Disk) Utilization Rate
- Cluster Pod TencentCloud API Ban
- Big Data
- Cloud Load Balancer
- Message Queue
- Direct Connect
- Custom Actions
- Cloud Streaming Services (CSS)
- Compute
- Permission Management Guide
- API Documentation
- FAQs
- Related Protocol
- Contact Us
Standard Cluster and Serverless Cluster Super Node Faults
Last updated: 2024-09-26 15:47:38
Background
Super nodes in container services, compared to regular nodes, support rapid auto scaling. They consolidate what would have been multiple nodes into a single node for management, simplifying the administration of container resources. However, container nodes may encounter hardware faults, resource shortages, network faults, etc., which could lead to container instances not operating correctly.
Through these experiments, you can verify whether the system can operate normally in the event of container node faults, and uncover potential issues in advance for system architecture optimization and emergency planning. Through these experiments, you can verify whether the system can operate normally in the event of container node faults, and uncover potential issues in advance for system architecture optimization and emergency planning.
Experiment Execution
Step 1: Experiment Preparation
Purchase standard cluster container instances and super nodes, and deploy test services.
Purchase Serverless cluster container instances with built-in super nodes and deploy test services.
Step 2: Create an Experiment
1. Log in to Tencent Smart Advisor > Chaotic Fault Generator, enter the Experiment Management page, and click Create a New Experiment.
2. Create a blank experiment. Click Skip and create a blank experiment.
3. Fill in the basic information for the experiment, add related resource tags as needed, then click Next.
4. Create two action groups, select Container as the Resource Type, and add Standard Cluster Super Node and Serverless Cluster Super Node Resource Objects respectively.
5. Find the experiment instances and add them to the action groups.
6. Click Add Now to add experiment actions, then select fault actions: Node lockdown and Node drain (Node Eviction).
7. For global configuration, confirm the Experiment Action Group, select the Experiment Execution Method and configure the Guardrail Policies.
8. Click Submit to complete the experiment creation.
Step 3: Execute the Experiment
1. Go to experiment details, click Go to Action Group Execution.
2. Execute the actions in the sequential execution:
3. Execute Node lockdown and Node lockdown Recovery.
3.1 Execute the Node Block Failure Action, view the execution logs on the action card, and observe the node status to confirm it is locked and no longer schedulable.
3.2 Execute the Node Block Fault Recovery Action to unlock the node and observe its status.
4. Execute Node Drain and Node Drain Recovery.
4.1 View the current Pod list on the node, check the high availability policies for the services, and ensure that there is sufficient capacity on other nodes to restart the Pods after eviction.
4.2 Execute the Node Drain Failure Action, which will evict resources from the node. The action card will show the evicted Pods, making it easier to observe the affected services. The super node will be locked and not schedulable. At the same time, through cluster scheduling, the evicted Pods are reconstructed on other nodes to recover services.
4.3 Execute fault recovery. Unblock the node and recover its schedulable status.
Yes
No
Was this page helpful?