tencent cloud

All product documents
Tencent Cloud TCHouse-P
DocumentationTencent Cloud TCHouse-PData IngestionSyncing Incremental Data from MySQL with DataX
Syncing Incremental Data from MySQL with DataX
Download PDF
Last updated: 2024-11-27 15:36:05
Syncing Incremental Data from MySQL with DataX
Last updated: 2024-11-27 15:36:05
Download PDF
This document describes how to use DataX modified by HashData to incrementally sync data from MySQL to Tencent Cloud TCHouse-P.
Follow the steps below to use DataX to incrementally sync data from MySQL to Tencent Cloud TCHouse-P:
1. Read the MaxTime since the last successful sync from the local file (for the initial sync, you can specify an initial time value as required by your business).
2. Use MaxTime as the LastTime (lower limit of the incremental sync) and CurTime as the upper limit.
3. Modify the datax.json configuration to specify the time interval (WHERE clause in SQL) of the synced table as [LastTime, CurTime).
4. Execute DataX sync. After successful sync, write CurTime to the local file for the next sync.
5. Repeat steps 1–4 for multiple incremental syncs.
A sample datax.json configuration file is as shown below:
{
    "job": {
        "setting": {
            "speed": {
                "channel": 3, 
                "byte": 1048576, 
                "record": 1000
            }, 
            "errorLimit": {
                "record": 2, 
                "percentage": 0.02
            }
        }, 
        "content": [
            {
                "reader": {
                    "name": "mysqlreader", 
                    "parameter": {
                        "username": "******", 
                        "password": "******", 
                        "connection": [
                            {
                                "jdbcUrl": [
                                    "jdbc:mysql://***:***/test?serverTimezone=Asia/Shanghai"
                                ],
                                "querySql": [
                                    "select * from cdw_test_table where updateTime >= '${lastTime}' and updateTime < '${currentTime}'"
                                ]
                            }
                        ]
                    }
                }, 
                "writer": {
                    "name": "gpdbwriter", 
                    "parameter": {
                        "username": "******", 
                        "password": "******", 
                        "column": [
                                 "*"
                        ],
                        "segment_reject_limit": 0, 
                        "copy_queue_size": 2000, 
                        "num_copy_processor": 1, 
                        "num_copy_writer": 1, 
                        "connection": [
                            {
                                "jdbcUrl": "jdbc:postgresql://***:***/***", 
                                "table": [
                                    "ods_cdw_test_table"
                                ]
                            }
                        ]
                    }
                }
            }
        ]
    }
}

Was this page helpful?
You can also Contact Sales or Submit a Ticket for help.
Yes
No

Feedback

Contact Us

Contact our sales team or business advisors to help your business.

Contact Us
Technical Support

Open a ticket if you're looking for further assistance. Our Ticket is 7x24 avaliable.

Submit a Ticket
7x24 Phone Support
Hong Kong, China
+852 800 906 020 (Toll Free)
United States
+1 844 606 0804 (Toll Free)
United Kingdom
+44 808 196 4551 (Toll Free)
Canada
+1 888 605 7930 (Toll Free)
Australia
+61 1300 986 386 (Toll Free)
EdgeOne hotline
+852 300 80699
More local hotlines coming soon