- Product Overview
- Purchase Guide
- Quick Start
- Operation Guide
- Fault Action Library
- Compute
- JVM Process CPU at Full Load
- Cross-AZ Experiment in CVM
- CVM DNS Unavailability Experiment
- CVM Domain Name Parsing Tampering Experiment
- CVM System Time Skew
- CVM Disk IO Hang Fault Experiment
- CVM Memory OOM and Disk IO Load
- Experiments on CVM Intra-host Network Disorder
- CVM Kernel Faults
- Experiments on CVM Intra-host Network Latency
- High Utilization of CVM Resources (CPU, Memory, and Disk)
- Experiments on CVM Intra-host Network Corruption
- Experiments on CVM Intra-host Network Duplication
- Experiments on CVM Intra-host Network Occupation
- CVM Network Interruption
- CVM Ping Unreachable
- Cloud API Ban in CVM
- Database
- Primary-secondary Switch in TencentDB for PostgreSQL
- MySQL Instance Overall Unavailable
- Primary-secondary Switch in MySQL
- Setting Maximum Number of Connections in MySQL
- Primary Node Fault Experiment on TencentDB for MySQL
- TencentDB for MySQL Read-only Instance Group Unavailable
- Primary-secondary Switch in TDSQL for MySQM
- Primary-secondary Switch in MariaDB
- Primary-secondary Switch in TDSQL-C
- Practice of TencentDB for Redis Proxy Node Faults
- TencentDB for Redis Primary Node Fault
- Primary-secondary Nodes in TencentDB for Redis Instance Unavailable
- Simulating Primary-secondary Switch in Redis
- Simulating MongoDB Storage Node Fault
- Simulating Self-Built MySQL Crash Through Network Blocking
- Restarting TencentDB for MySQL
- Primary-secondary Switch in SQL Server
- Network
- Container
- Simulating Container Resource Network Faults
- Experiment on Container Resource Pod Operation Faults
- Experiment on Container Resource Node Faults
- Experiment on Container Resource Application Process Faults
- Standard Cluster and Serverless Cluster Super Node Faults
- Serverless Pod Fault Experiment Case
- Cluster Node Resource (CPU, Memory, Disk) Stress Test Faults
- Serverless Pod Virtual Node Shutdown Faults
- Simulating Serverless Cluster Pod Network Faults
- High Cluster Pod Resource (CPU, Memory, and Disk) Utilization Rate
- Cluster Pod TencentCloud API Ban
- Big Data
- Cloud Load Balancer
- Message Queue
- Direct Connect
- Custom Actions
- Cloud Streaming Services (CSS)
- Compute
- Permission Management Guide
- API Documentation
- FAQs
- Related Protocol
- Contact Us
- Product Overview
- Purchase Guide
- Quick Start
- Operation Guide
- Fault Action Library
- Compute
- JVM Process CPU at Full Load
- Cross-AZ Experiment in CVM
- CVM DNS Unavailability Experiment
- CVM Domain Name Parsing Tampering Experiment
- CVM System Time Skew
- CVM Disk IO Hang Fault Experiment
- CVM Memory OOM and Disk IO Load
- Experiments on CVM Intra-host Network Disorder
- CVM Kernel Faults
- Experiments on CVM Intra-host Network Latency
- High Utilization of CVM Resources (CPU, Memory, and Disk)
- Experiments on CVM Intra-host Network Corruption
- Experiments on CVM Intra-host Network Duplication
- Experiments on CVM Intra-host Network Occupation
- CVM Network Interruption
- CVM Ping Unreachable
- Cloud API Ban in CVM
- Database
- Primary-secondary Switch in TencentDB for PostgreSQL
- MySQL Instance Overall Unavailable
- Primary-secondary Switch in MySQL
- Setting Maximum Number of Connections in MySQL
- Primary Node Fault Experiment on TencentDB for MySQL
- TencentDB for MySQL Read-only Instance Group Unavailable
- Primary-secondary Switch in TDSQL for MySQM
- Primary-secondary Switch in MariaDB
- Primary-secondary Switch in TDSQL-C
- Practice of TencentDB for Redis Proxy Node Faults
- TencentDB for Redis Primary Node Fault
- Primary-secondary Nodes in TencentDB for Redis Instance Unavailable
- Simulating Primary-secondary Switch in Redis
- Simulating MongoDB Storage Node Fault
- Simulating Self-Built MySQL Crash Through Network Blocking
- Restarting TencentDB for MySQL
- Primary-secondary Switch in SQL Server
- Network
- Container
- Simulating Container Resource Network Faults
- Experiment on Container Resource Pod Operation Faults
- Experiment on Container Resource Node Faults
- Experiment on Container Resource Application Process Faults
- Standard Cluster and Serverless Cluster Super Node Faults
- Serverless Pod Fault Experiment Case
- Cluster Node Resource (CPU, Memory, Disk) Stress Test Faults
- Serverless Pod Virtual Node Shutdown Faults
- Simulating Serverless Cluster Pod Network Faults
- High Cluster Pod Resource (CPU, Memory, and Disk) Utilization Rate
- Cluster Pod TencentCloud API Ban
- Big Data
- Cloud Load Balancer
- Message Queue
- Direct Connect
- Custom Actions
- Cloud Streaming Services (CSS)
- Compute
- Permission Management Guide
- API Documentation
- FAQs
- Related Protocol
- Contact Us
Background
Tencent Smart Advisor-Chaotic Fault Generator provides fault actions for simulating a primary-secondary switch scene in the TencentDB for TDSQL-C to verify the overall high availability of TDSQL-C in your business.
Primary-secondary switch experiments are intended to help developers in system tests and experiments in a more complex and realistic environment so that possible problems and risks can be identified. Through experiments and tests in chaos engineering, developers can have a more comprehensive understanding of system operating modes and performance characteristics so that they can develop countermeasures and policies for different fault scenes to improving system stability and availability.
Note:
This fault action only supports injection of primary-secondary switch faults into cross-availability zone instances. If an instance deployed in a single availability zone is selected, injection will fail and a prompt will be given in environmental check. Corresponding instances can be selected through TDSQL-C Console to adjust deployment mode.
Example of Instance Deployment Adjustment:
1. Log in to TDSQL-C Console, click Cluster List, and select the instance in a single AZ.
2. Click Edit Icon in Deployment Method in instance information.
3. Select Multiple Availability Zones (Yes), complete multi-availability zone configuration, and click Confirm.
Experiment Implementation
Step 1: Experiment Preparation
A cross-availability zone TDSQL-C instance. Pay attention to availability zones where primary and secondary nodes are located.
Step 2: Experiment Orchestration
1. Log in to the Tencent Smart Advisor > Chaotic Fault Generator, go to the Experiment Management page, and click Create a New Experiment.
2. Click Skip and create a blank experiment.
3. Fill in experiment information and select Object Type TDSQL-C.
4. Add instances and add fault actions.
5. Add Primary-secondary switch experiment actions. No parameter is to be configured for this fault. Click Next.
6. Click Next, configure the Experiment Orchestration method and optionally configure Guardrail Policy Monitoring Metrics, and then click Submit to create an experiment.
Step 3: Experiment Execution
During fault execution, primary-secondary switch in TDSQL-C instance will be triggered. Changes in primary and secondary node architectures can be observed through Cloud Database TDSQL-C Console.
Go to Experiment Details, click Execute to start executing an experiment.
Execute fault actions. Changes in primary and secondary node architectures can be observed through Cloud Database TDSQL-C Console.
During fault injection, state of switch can be observed through TDSQL-C Console.
After a successful fault execution, you can observe changes in the primary node through Cloud Database TDSQL-C Console, and the primary node has been switched from the original primary availability zone to cross-availability zone replica availability zone node.
Execute fault recovery actions to recover instance deployment to the pre-fault state.
After a successful recovery, it can be seen through the console that the node has been recovered to the pre-fault state.
Yes
No
Was this page helpful?