tencent cloud

All product documents
Tencent Cloud WeData
DocumentationTencent Cloud WeDataOperation GuideData IntegrationOffline Synchronization Task Configuration and Operation and MaintenanceData Sources Supported by Offline SynchronizationData Source ListDLC Data Source
DLC Data Source
Download PDF
Last updated: 2024-11-01 17:50:37
DLC Data Source
Last updated: 2024-11-01 17:50:37
Download PDF

DLC Offline Single Table Read Node Configuration




Parameters
Description
Data Source
Available DLC Data Source.
Database
Supports selection or manual input of the library name to read from.
By default, the database bound to the data source is used as the default database. Other databases need to be manually entered.
If the data source network is not connected and the database information cannot be fetched directly, you can manually enter the database name. Data synchronization can still be performed when the DataInLong network is connected.
Table
Supports selecting or manually entering the table name to be read.
Split Key
Specify the field for data sharding. After specifying, concurrent tasks will be launched for data synchronization. You can use a column in the source data table as the partition key. It is recommended to use the primary key or indexed column as the partition key.
Filter Conditions (Optional)
In actual business scenarios, it is common to synchronize data of the current day, with the where condition specified as gmt_create>$bizdate.
The where condition can effectively perform incremental business synchronization.
If the WHERE clause is not provided, including missing the key or value, the data synchronization is treated as full data synchronization.
Advanced Settings (Optional)
You can configure parameters according to business needs.

DLC Offline Single Table Write Node Configuration




Parameters
Description
Data Destination
DLC Data Source to be written.
Database
Support selection or manual entry of the library name to be written to.
By default, the database bound to the data source is used as the default database. Other databases need to be manually entered.
If the data source network is not connected and the database information cannot be fetched directly, you can manually enter the database name. Data synchronization can still be performed when the DataInLong network is connected.
Table
Support selection or manual entry of the table name to be written to.
If the data source network is not connected and the table information cannot be fetched directly, you can manually enter the table name. Data synchronization can still be performed when the DataInLong network is connected.
When the source table is of MySQL, ES, or Kafka type, it supports one-click creation of the target table.
Write Mode
DLC write supports three modes:
overwrite: Overwrite the original data.
append: Append write, an error occurs when there is a primary key conflict.
upsert: Update write, data is updated when there is a primary key conflict.
Advanced Settings (Optional)
You can configure parameters according to business needs.

Data type conversion support

Read

DLC Data Type
Internal Types
INTEGER
Long
FLOAT,DOUBULE,DECIMAL
Double
STRING,VARCHAR,CHAR,ARRAY
String
TIMESTAMP,DATE,TIMESTAMP WITH TIME ZONE
Date
BOOLEAN
Boolean

Write

Internal Types
DLC Data Type
Long
INTEGER
Double
FLOAT,DOUBULE,DECIMAL
String
STRING,MAP,STRUCT
Date
TIMESTAMP,DATE,TIMESTAMP WITH TIME ZONE
Boolean
BOOLEAN
Bytes
BINARY

FAQs

1. DLC actual field value is null but the read value is '' empty string, causing dirty data due to conversion to LONG type

Cause:
Since the integrated DLC read task submits a query SQL to the DLC, after the DLC engine processing is complete, the integration directly reads the result file. The file will treat null as an empty string, but integration does not support processing it into a specified value.
Solutions:
Use the DLC function IFNULL for conversion when configuring tasks, and complete the value preprocessing in the DLC engine. Other similar situations can refer to the use of DLC functions.





Was this page helpful?
You can also Contact Sales or Submit a Ticket for help.
Yes
No

Feedback

Contact Us

Contact our sales team or business advisors to help your business.

Contact Us
Technical Support

Open a ticket if you're looking for further assistance. Our Ticket is 7x24 avaliable.

Submit a Ticket
7x24 Phone Support
Hong Kong, China
+852 800 906 020 (Toll Free)
United States
+1 844 606 0804 (Toll Free)
United Kingdom
+44 808 196 4551 (Toll Free)
Canada
+1 888 605 7930 (Toll Free)
Australia
+61 1300 986 386 (Toll Free)
EdgeOne hotline
+852 300 80699
More local hotlines coming soon