ckafka-migrate.py
file.# your local broker ip:port list# Broker list of the self-built instance ["broker1:port1","ip2:port2"]bootstrapServers = ["$ip:$port"]# your local zk ip:port list# ZooKeeper list of the self-built instance ["zk1:port1","zk2:port2"]. This is an optional configuration. If it is not configured, the source cluster information will be obtained via `bootstrapServers`.sourceZk = ["$ip:$port"]# This parameter is required if ZooKeeper has enabled authentication. Format: [("digest", "$user:$password")]zkAuthData = []# This parameter is required if the ZooKeeper path has a prefix. Sample value: "/cluster1".zkPathPrefix = ""# your cloud instanceId# CKafka instance ID "ckafka-xxx"instanceId = "$yourinstanceId"# topic regex,just migrate match topics# Regular expression of the topic name. If it is not empty, only matched topics will be migrated.topicRegex = ""# your secretId and secretKey# Account key pairsecretId = "$yoursecretId"secretKey = "$yoursecretKey"# your cloud instance region# CKafka instance region and code:# Guangzhou: ap-guangzhou; Shanghai: ap-shanghai; Nanjing: ap-nanjing; Beijing: ap-beijing; Chengdu: ap-chengdu; Chongqing: ap-chongqing;# Hong Kong (China): ap-hongkong; Singapore: ap-singapore; Mumbai: ap-mumbai; Tokyo: ap-tokyo; Silicon Valley: na-siliconvalley;# Virginia: na-ashburn; Taipei (China): ap-taipei; Tianjin: ap-tianjin; Shanghai Finance: ap-shanghai-fsi;# Shenzhen Finance: ap-shenzhen-fsi; Shenzhen: ap-shenzhen; Frankfurt: eu-frankfurt; Seoul: ap-seoul; Qingyuan: ap-qingyuan;# Beijing Finance: ap-beijing-fsi; Bangkok: ap-bangkok; Changsha: ap-changsha-ec; Jakarta: ap-jakarta;# Hangzhou ec: ap-hangzhou-ec; São Paulo: sa-saopaulo; Shanghai Auto-Driving Cloud: ap-shanghai-adc; Wuhan ec: ap-wuhan-ec, Xi'an ec: ap-xian-ecregion = "ap-tokyo"# If you make sure the migrate topic List, please modify checkFlag = 1# Check flag. If it is 0, only the list of topics to be migrated will be displayed, with no migration performed. You can set it to 0 first to check the list of topics. After confirming that everything is correct, you can change it to 1 to start migration.# 0: List the topics to be migrated and then stop the script# 1: List the topics to be migrated and then start migrationcheckFlag = 0# force transfer your cloud-topic config to migrate# If the value is 0, a local topic will not be migrated to CKafka if its attributes do not match those of CKafka. If the value is 1, the topic attribute values out of the value range of CKafka will be forcibly converted to the closest values within the value range.# For example, if the CKafka topic supports only 1, 2 or 3 replicas, but a local topic has 5 replicas, it will not be migrated to CKafka. If you set the parameter to 1, then a 3-replica topic, which is closest to the original number of replicas (5), will be created for the CKafka topic.# 0: Skip the unmatched topic or topic attributes if the numbers of replicas or attributes of the local and CKafka topics do not match.# 1: Forcibly migrate all topics to CKafka if the numbers of replicas or attributes of the local and CKafka topics do not match. Then, modify CKafka topic attributes to be as close as possible to those self-built ones, without modifying any data in the local self-built Kafka cluster.force = 0
Parameter | Description |
bootstrapServers | Broker list of the self-built instance ["ip1:port1","ip2:port2"] |
sourceZk | ZooKeeper list of the self-built instance ["zk1:port1","zk2:port2"]. This is an optional configuration. If it is not configured, the source cluster information will be obtained via bootstrapServers . |
zkAuthData | This parameter is required if ZooKeeper has enabled authentication. Format: [("digest", "$user:$password")] |
zkPathPrefix | This parameter is required if the ZooKeeper path has a prefix. Sample value: "/cluster1". |
instanceId | ID of the CKafka instance you purchased in Step 1. Purchasing CKafka Instance. You can copy the ID on the Instance List page in the CKafka console. |
secretId | Account key pair – ID |
secretKey | Account key pair – password |
region | The deployment region you selected in Step 1. Purchasing CKafka Instance. The region codes are available in the script annotation. |
checkFlag | Check flag. If it is 0, only the list of topics to be migrated will be displayed and migration will not start; otherwise, the topics will start to be migrated. |
topicRegex | Regular expression of the topic name. If it is empty, all topics will be migrated; otherwise, only matched topics will be migrated. |
force | It specifies whether to forcibly migrate topics. If the value is 0, a local topic will not be migrated to CKafka if its attributes do not match those of CKafka. If the value is 1, the topic attribute values out of the value range of CKafka will be forcibly converted to the closest values within the value range. |
checkFlag
parameter in ckafka-migrate.py
to 0, run python ckafka-migrate.py
to execute the script, and check the list of topics that need to be migrated based on the output result.checkFlag
parameter in ckafka-migrate.py
to 1 and run python ckafka-migrate.py
to execute the script to start migrating the topics.
Was this page helpful?