Check the log of the source Redis database. If it contains the following message, the client-output-buffer-limit configured for the source database is exceeded.
psync scheduled to be closed ASAP for overcoming of output buffer limits
Solution
Run the following command to set the client-output-buffer-limit to infinite and initiate the DTS task again.
config set client-output-buffer-limit 'slave 0 0 0'
Issue 2
Description
The following error message is displayed during migration with DTS:
[launch]state:8 #rdb rdbfile:./tmp1600395232_34851.rdb rdbsize:107994104 rdb_writed_size:107994104 rdb_parsed_size:107994104 rdb_parsed_begin:1600395238 rdb_parsed_time:5 #replication master_replid:995dba8ccffb7cc32a7c85de7b1632b952b74496 repl_offset:23851025 write_command_count:940765 finish_command_count:940763 last_replack_time:1600395298 #queue send_write_pos:440766 send_read_pos:440765 response_write_pos:440765 response_read_pos:440764 errtime:1600395297 errmsg:get rsp error:ERR value is not an integer or out of range command:*2 $4 INCR $35 APP_API_ORDER_CREATION_USER_4260882
Causes
By capturing packets on two DTS Syncer instances in the region, it was found that the value of the key was characters rather than a number, causing the INCR execution to fail.
Solution
Delete the relevant key and initiate the DTS migration again.
Issue 3
Description
The following error message is displayed during migration with DTS:
errmsg:Error reading bulk length while SYNCing:Operation now in progress read rdb length from src fail save rdb fail ready shutdown dts
Causes
The error message of the source instance reveals that the RDB file does not have permission to access the directory.
Solution
Run the following command to set diskless replication and initiate the DTS task again.
config set repl-diskless-sync yes
Issue 4
Description
The following error message is displayed during migration with DTS:
[launch]state:6 #rdb rdbfile:./tmp1597977351_20216.rdb rdbsize:24282193511 rdb_writed_size:18683334200 rdb_parsed_size:0 rdb_parsed_begin:0 rdb_parsed_time:0 #replication master_replid:1b0da9f595cc40b795803eba3c9bea3aad1a1d68 repl_offset:921330115650 write_command_count:0 finish_command_count:0 last_replack_time:0 #queue send_write_pos:0 send_read_pos:0 response_write_pos:0 response_read_pos:0 errtime:1597978778 errmsg:write rdb data fail:456!=1696 error:No space left on device save rdb fail ready shutdown dts
Causes
The disk space in the DTS Syncer instance is insufficient.
Solution
Clear the disk in the DTS Syncer instance or mount a new disk and then initiate the DTS task again.
Issue 5
Description
The following error message is displayed during migration with DTS:
[launch]state:4/5 #rdb rdbfile: rdbsize:0 rdb_writed_size:0 rdb_parsed_size:0 rdb_parsed_begin:0 rdb_parsed_time:0 #replication master_replid:d3e707ec0e72c3908b0ce70dd2460f48086c5386 repl_offset:683087907631 write_command_count:0 finish_command_count:0 last_replack_time:0 #queue send_write_pos:0 send_read_pos:0 response_write_pos:0 response_read_pos:0 errtime:1654369638 errmsg:Error reading bulk length while SYNCing:Operation now in progress read rdb length from src fail save rdb fail ready shutdown dts
Causes
The issue is caused by insufficient memory on the source, leading to RDB generation failure, or an unstable network.
Solution
1. Adjust the kernel connection limits and buffer limits on the source.
Disconnect the source database and adjust the limit on the connections to the source system kernel.
Execute the following command to set the source's client-output-buffer-limit to unlimited.
config set client-output-buffer-limit 'slave 0 0 0'
2. If the issue persists, increase the memory on the machine where the source node is located to ensure that the node has sufficient resources to generate the RDB file.
Issue 6
Description
The following error message is displayed during migration from Redis Memory Edition (Standard Architecture) to Cluster Architecture with DTS:
[launch]state:8 #rdb rdbfile:./tmp1645683629_34614.rdb rdbsize:781035471 rdb_writed_size:781035471 rdb_parsed_size:781035471 rdb_parsed_begin:1645683632 rdb_parsed_time:25 #replication master_replid:5abe7987b1e263582c68835412d2963eeb0a3d60 repl_offset:895807918761 write_command_count:6102523 finish_command_count:6102137 last_replack_time:1645683656 #queue send_write_pos:101832 send_read_pos:101742 response_write_pos:101742 response_read_pos:101357 errtime:1645683657 errmsg:get rsp error:CROSSSLOT Keys in request don't hash to the same slot command:*3 $6 RENAME $16 dispatch:km:pool $34 dispatch:km:tmp-pool:1645683651224 ready shutdown dts send replconf ack to src fail:Bad file descriptor
You can see the following information in the execution log of the source node:
44:M 05 Jun 03:31:06.728 * Starting BGSAVE for SYNC with target: disk
44:M 05 Jun 03:31:06.978 * Background saving started by pid 89
89:C 05 Jun 03:32:08.417 # Error moving temp DB file temp-89.rdb on the final destination 20617.20324.rdb (in server root dir /opt/data/dump): No such file or directory
44:M 05 Jun 03:32:08.698 # Background saving error
44:M 05 Jun 03:32:08.698 # Connection with slave 10.xx.xx.119:<unknown-slave-port> lost.
44:M 05 Jun 03:32:08.698 # SYNC failed. BGSAVE child returned an error
44:M 05 Jun 03:50:24.626 * Slave 10.xx.xx.119:<unknown-slave-port> asks for synchronization
44:M 05 Jun 03:50:24.626 * Full resync requested by slave 10.xx.xx.119:<unknown-slave-port>
44:M 05 Jun 03:50:24.626 * Starting BGSAVE for SYNC with target: disk
44:M 05 Jun 03:50:24.880 * Background saving started by pid 90
90:C 05 Jun 03:51:22.585 * DB saved on disk
90:C 05 Jun 03:51:22.739 * RDB: 280 MB of memory used by copy-on-write
44:M 05 Jun 03:51:23.008 * Background saving terminated with success
44:M 05 Jun 03:51:27.898 * Synchronization with slave 10.xx.xx.119:<unknown-slave-port> succeeded
44:M 05 Jun 03:52:19.531 # Connection with slave client id #317862457 lost.
Causes
This is often because the connection of the DTS task to the source node timed out due to network environment issues, big keys contained in the database, or client-output-buffer-limit overflows on the source node.
Solution
Check the source network environment for any issue. For detailed directions, see Database Connection Check.
Clear big keys in the source database. You can quickly locate, assess, and delete big keys as instructed in Memory Analysis.
Run the following command to set the client-output-buffer-limit to infinite on the source node.
config set client-output-buffer-limit 'slave 0 0 0'
Issue 8
Description
The following error message is displayed during migration with DTS:
The Redis node of the target instance experienced an HA master/replica switch, or the proxy node experienced a failover, causing the sync task to fail.
Solution
Create a new DTS task and configure the new node after the HA switch as the target node for data migration.
Issue 10
Description
When the memory eviction policy of the target instance is set to allkey-lru during migration with DTS, the following error message is displayed:
[launch]state:8 #rdb rdbfile:./tmp1638263556_29975.rdb rdbsize:597343276 rdb_writed_size:597343276 rdb_parsed_size:428299275 rdb_parsed_begin:1638263575 rdb_parsed_time:7 #replication master_replid:ae0dfc45f72f3ee8642c8e31e493b6442179734f repl_offset:34832262785 write_command_count:6811 finish_command_count:6798 last_replack_time:1638263582 #queue send_write_pos:6811 send_read_pos:6811 response_write_pos:6811 response_read_pos:6799 errtime:1638263583 errmsg:get rsp error:OOM command not allowed when used memory > 'maxmemory'. command:*3 $3 SET $26 all_business_newmikoxmsong $1189783 [{"id":3,"label":"AI\\u5e73\\u53f0\\u90e8","node_level":0,"leaf":false,"children":[{"id":"887381","label":"[OMG][\\u4f53\\u80b2\\u641c\\u7d22][CMDB]","node_level":1,"leaf":false,"children":[{"id":"722605","label":"[\\u4e2a\\u6027\\u5316\\u63a8\\u8350\\u4e2d\\u5fc3][\\u817e\\u8baf\\u7f51\\u4f53\\u80b2APP\\u63a8\\u8350]","node_level":2,"leaf":false,"children":[],"collet":0}],"collet":0},{"id":"460871","label":"[OMG][\\u817e\\u8baf\\u89c6\\u9891\\u641c\\u7d22][CMDB]","node_level":1,"leaf":false,"children":[{"id":"383393","label":"[\\u641c\\u7d22\\u4e1a\\u52a1\\u4e2d\\u5fc3][\\u817e\\u8baf\\u89c6\\u9891\\u641c\\u7d22]","node_level":2,"leaf":false,"children":[],"collet":0}],"collet":0}]},{"id":8,"label":"IDC\\u5e73\\u53f0\\u90e8","node_level":0,"leaf":false,"children":[{"id":"452519","label":"IDC\\u5e73\\u53f0\\u90e8\\u81ea\\u7528[CMDB]","node_level":1,"leaf":false,"children":[{"id":"453099","label":"IDC\\u7cfb\\u7edf\\u5f00\\u53d1","node_level":2,"leaf":false,"children":[],"collet":0}],"collet":0}]},{"id":14 ready shutdown dts send replconf ack to src fail:Bad file descriptor
Causes
The memory of the target instance is smaller than the memory used by the data to be migrated from the source database.
Solution
Expand the memory of the target instance and then initiate a new DTS migration task. For detailed directions, see Changing Instance Specification.
Issue 11
Description
The following error message is displayed during migration with DTS:
Causes
If an error is reported when you start a DTS migration task with a proxy, it is usually because the bandwidth of the proxy is insufficient.
Solution
Expand the bandwidth of the proxy or perform migration tasks serially in sequence. For detailed directions, see Bandwidth Adjustment.
Issue 12
Description
The following error message is displayed during migration with DTS:
[launch]SrcInstance nodes has changed.
Causes
The source node experienced an HA master/replica switch, causing the DTS task sync to fail.
Solution
Create a new DTS task and configure the new node after the HA switch as the target node for data migration.
Yes
No
Was this page helpful?