tencent cloud

All product documents
Tencent Cloud TCHouse-P
DocumentationTencent Cloud TCHouse-PData IngestionImporting TencentDB Data Offline with DataX
Importing TencentDB Data Offline with DataX
Download PDF
Last updated: 2024-11-27 15:36:05
Importing TencentDB Data Offline with DataX
Last updated: 2024-11-27 15:36:05
Download PDF
DataX is an open-source CLI that supports importing full or incremental data from TencentDB to Tencent Cloud TCHouse-P. The tool is developed in Java and uses JDBC to connect the source database to the target database. It can run on Windows and Linux. Install the Java environment before use.
DataX installation:
1. Download the source code here and compile it.
2. Directly use datax-v1.0.4-hashdata.tar.gz, an already compiled version.
The following section introduces DataX modified by HashData, which is more efficient to import data to Tencent Cloud TCHouse-P. Tests show that it can import more than 100,000 entries per second. The following is the configuration file to import data from MySQL to Tencent Cloud TCHouse-P:
{
    "job": {
        "setting": {
            "speed": {
                "channel": 3, 
                "byte": 1048576, 
                "record": 1000
            }, 
            "errorLimit": {
                "record": 2, 
                "percentage": 0.02
            }
        }, 
        "content": [
            {
                "reader": {
                    "name": "mysqlreader", 
                    "parameter": {
                        "username": "****", 
                        "password": "****", 
                        "column": [
                            "*"
                        ], 
                        "splitPk": "id", 
                        "connection": [
                            {
                                "table": [
                                    "test1"
                                ], 
                                "jdbcUrl": [
                                    "jdbc:mysql://***:***/db1?serverTimezone=Asia/Shanghai"
                                ]
                            }
                        ]
                    }
                }, 
                "writer": {
                    "name": "gpdbwriter", 
                    "parameter": {
                        "username": "******", 
                        "password": "******", 
                        "column": [
                            "*"
                        ], 
                        "preSql": [
                            "truncate table test1"
                        ], 
                        "postSql": [
                            "select count(*) from test2"
                        ], 
                        "segment_reject_limit": 0, 
                        "copy_queue_size": 2000, 
                        "num_copy_processor": 1, 
                        "num_copy_writer": 1, 
                        "connection": [
                            {
                                "jdbcUrl": "jdbc:postgresql://****:**/db1", 
                                "table": [
                                    "test1"
                                ]
                            }
                        ]
                    }
                }
            }
        ]
    }
}
Parameter description:
1. The writer should be gpdbwriter. postgresqlwriter can also be used to write data to Tencent Cloud TCHouse-P, with a poor insertion efficiency though.
2. For specific meanings and parameter tuning, see DataX.
3. We recommend you add the serverTimezone=Asia/Shanghai parameter to the JDBC URL of mysqlreader to avoid data inconsistency caused by time zone issues.
Was this page helpful?
You can also Contact Sales or Submit a Ticket for help.
Yes
No

Feedback

Contact Us

Contact our sales team or business advisors to help your business.

Contact Us
Technical Support

Open a ticket if you're looking for further assistance. Our Ticket is 7x24 avaliable.

Submit a Ticket
7x24 Phone Support
Hong Kong, China
+852 800 906 020 (Toll Free)
United States
+1 844 606 0804 (Toll Free)
United Kingdom
+44 808 196 4551 (Toll Free)
Canada
+1 888 605 7930 (Toll Free)
Australia
+61 1300 986 386 (Toll Free)
EdgeOne hotline
+852 300 80699
More local hotlines coming soon