Tencent Cloud

DLC Data Source

Last updated: 2024-11-01 17:50:37

DLC Offline Single Table Read Node Configuration
﻿
﻿
﻿
Parameters
Description
Data Source
Available DLC Data Source.
Database
Supports selection or manual input of the library name to read from.
By default, the database bound to the data source is used as the default database. Other databases need to be manually entered.
If the data source network is not connected and the database information cannot be fetched directly, you can manually enter the database name. Data synchronization can still be performed when the DataInLong network is connected.
Table
Supports selecting or manually entering the table name to be read.
Split Key
Specify the field for data sharding. After specifying, concurrent tasks will be launched for data synchronization. You can use a column in the source data table as the partition key. It is recommended to use the primary key or indexed column as the partition key.
Filter Conditions (Optional)
In actual business scenarios, it is common to synchronize data of the current day, with the where condition specified as gmt_create>$bizdate.
The where condition can effectively perform incremental business synchronization.
If the WHERE clause is not provided, including missing the key or value, the data synchronization is treated as full data synchronization.
Advanced Settings (Optional)
You can configure parameters according to business needs.
DLC Offline Single Table Write Node Configuration
﻿
﻿
﻿
Parameters
Description
Data Destination
DLC Data Source to be written.
Database
Support selection or manual entry of the library name to be written to.
By default, the database bound to the data source is used as the default database. Other databases need to be manually entered.
If the data source network is not connected and the database information cannot be fetched directly, you can manually enter the database name. Data synchronization can still be performed when the DataInLong network is connected.
Table
Support selection or manual entry of the table name to be written to.
If the data source network is not connected and the table information cannot be fetched directly, you can manually enter the table name. Data synchronization can still be performed when the DataInLong network is connected.
When the source table is of MySQL, ES, or Kafka type, it supports one-click creation of the target table.
Write Mode
DLC write supports three modes:
overwrite: Overwrite the original data.
append: Append write, an error occurs when there is a primary key conflict.
upsert: Update write, data is updated when there is a primary key conflict.
Advanced Settings (Optional)
You can configure parameters according to business needs.
Data type conversion support
Read
DLC Data Type
Internal Types
INTEGER
Long
FLOAT,DOUBULE,DECIMAL
Double 
STRING,VARCHAR,CHAR,ARRAY
String 
TIMESTAMP,DATE,TIMESTAMP WITH TIME ZONE
Date
BOOLEAN 
Boolean
Write
Internal Types
DLC Data Type
Long
INTEGER
Double
FLOAT,DOUBULE,DECIMAL
String
STRING,MAP,STRUCT
Date
TIMESTAMP,DATE,TIMESTAMP WITH TIME ZONE
Boolean
BOOLEAN
Bytes
BINARY
FAQs
1. DLC actual field value is null but the read value is '' empty string, causing dirty data due to conversion to LONG type
Cause:
Since the integrated DLC read task submits a query SQL to the DLC, after the DLC engine processing is complete, the integration directly reads the result file. The file will treat null as an empty string, but integration does not support processing it into a specified value.
Solutions:
Use the DLC function IFNULL for conversion when configuring tasks, and complete the value preprocessing in the DLC engine. Other similar situations can refer to the use of DLC functions.
﻿
﻿
﻿
﻿
﻿

tencent cloud

Recent Pages

DLC Data Source

DLC Offline Single Table Read Node Configuration

DLC Offline Single Table Write Node Configuration

Data type conversion support

Read

Write

FAQs

1. DLC actual field value is null but the read value is '' empty string, causing dirty data due to conversion to LONG type

Was this page helpful?

Was this page helpful?

Parameters	Description
Data Source	Available DLC Data Source.
Database	Supports selection or manual input of the library name to read from. By default, the database bound to the data source is used as the default database. Other databases need to be manually entered. If the data source network is not connected and the database information cannot be fetched directly, you can manually enter the database name. Data synchronization can still be performed when the DataInLong network is connected.
Table	Supports selecting or manually entering the table name to be read.
Split Key	Specify the field for data sharding. After specifying, concurrent tasks will be launched for data synchronization. You can use a column in the source data table as the partition key. It is recommended to use the primary key or indexed column as the partition key.
Filter Conditions (Optional)	In actual business scenarios, it is common to synchronize data of the current day, with the where condition specified as gmt_create>$bizdate. The where condition can effectively perform incremental business synchronization. If the WHERE clause is not provided, including missing the key or value, the data synchronization is treated as full data synchronization.
Advanced Settings (Optional)	You can configure parameters according to business needs.

Parameters	Description
Data Destination	DLC Data Source to be written.
Database	Support selection or manual entry of the library name to be written to. By default, the database bound to the data source is used as the default database. Other databases need to be manually entered. If the data source network is not connected and the database information cannot be fetched directly, you can manually enter the database name. Data synchronization can still be performed when the DataInLong network is connected.
Table	Support selection or manual entry of the table name to be written to. If the data source network is not connected and the table information cannot be fetched directly, you can manually enter the table name. Data synchronization can still be performed when the DataInLong network is connected. When the source table is of MySQL, ES, or Kafka type, it supports one-click creation of the target table.
Write Mode	DLC write supports three modes: overwrite: Overwrite the original data. append: Append write, an error occurs when there is a primary key conflict. upsert: Update write, data is updated when there is a primary key conflict.
Advanced Settings (Optional)	You can configure parameters according to business needs.

DLC Data Type	Internal Types
INTEGER	Long
FLOAT,DOUBULE,DECIMAL	Double
STRING,VARCHAR,CHAR,ARRAY	String
TIMESTAMP,DATE,TIMESTAMP WITH TIME ZONE	Date
BOOLEAN	Boolean

tencent cloud

Sign Up

Log in

Recent Pages

DLC Data Source

DLC Offline Single Table Read Node Configuration

DLC Offline Single Table Write Node Configuration

Data type conversion support

Read

Write

FAQs

1. DLC actual field value is null but the read value is '' empty string, causing dirty data due to conversion to LONG type

Was this page helpful?

Was this page helpful?