Dameng DM Data Source

DataInLong offers single table read and write capabilities for Dameng DM. This document introduces the pre-environment configuration for real-time data synchronization using Dameng DM, as well as the current capability support.
Supported Editions
Currently, DataInLong supports real-time single table read and write for Dameng DM. To use real-time synchronization, adhere to the following version limitations:
Type
Edition
Dameng DM
7,8
Use Limits 
Currently only supports real-time single table read and write, and does not support full database tasks.
Only At-least-once semantics are supported. Target-side deduplication capability is required to ensure data uniqueness.
DM as the target end does not support one-click creation of DM target-end table structures based on source table structures. The target table needs to be created in the DM database in advance.
Preparing the Database Environment
Enable Archive mode
1. Use the following statements to verify if the ARCH_INI and RLOG_APPEND_LOGIC parameters are correctly set to ensure that the database has Archive mode enabled:
select para_name, para_value 
from v$dm_ini 
where para_name in ('ARCH_INI','RLOG_APPEND_LOGIC');
2. If the parameters show 0, follow these steps to set Archive mode and modify the parameters.
2.1 Modify the parameters in dm.ini as follows:
ARCH_INI = 1
RLOG_APPEND_LOGIC = 1
2.2 dmarch.ini needs to configure local archiving. Here is an example:
[ARCHIVE_LOCAL1]
ARCH_TYPE = LOCAL
ARCH_DEST = d:\dmdbms\arch
ARCH_FILE_SIZE = 128 #Unit: MB
ARCH_SPACE_LIMIT = 0 #Unit: MB, 0 means unlimited, range 1024~4294967294M
ARCH_INI and RLOG_APPEND_LOGIC
﻿﻿
Enable Autocommit
Most DMC tools have autocommit enabled by default. That is, after executing a Select, Insert, Delete, or Update statement, it will automatically perform a commit operation. However, Dameng DMC tools do not; whether it's TCCLI disql or visual management tools, they do not autocommit by default and require a manual commit operation.
Method 1: Select Window > Options from the management tools menu, click Query Analyzer, and check Autocommit.
Method 2: Use SQL commands.
Temporary modifications can be configured as follows, as described in the official documentation:
1. Set whether the current session automatically commits changes to the data.
ON: Enable autocommit. The transaction for any executed SQL statement will be automatically committed.
OFF: Disable autocommit. The transaction for any executed SQL statement will need to be committed explicitly by the user, which is the default setting.
SET AUTO[COMMIT] <ON|OFF>
Permission granting
grant create session to my_user; 
grant select on all_users to  my_user ; 
grant select on all_objects to  my_user ; 
grant select on all_tab_cols to  my_user ; 
grant select on all_indexes to  my_user ; 
grant select on dba_objects to  my_user ; 
grant select on V$LOGMNR_CONTENTS to  my_user ; 
grant select on V$ARCHIVED_LOGS to  my_user ; 
Single Table Read Node Configuration
﻿
﻿
﻿
Parameters
Description
Data Source
Select Dameng DM Data Source.
Database
Support selecting or manually entering the name of the required database.
By default, the database bound to the data source is used as the default database. Other databases need to be manually entered.
If the data source network is not connected and the database information cannot be fetched directly, you can manually enter the database name. Data synchronization can still be performed when the DataInLong network is connected.
Schema
Support selecting or manually entering the available schema under the data source.
Table
Support selecting or manually entering the table name to be read.
In the case of sharding, multiple table names can be selected or entered in the Dameng DM source, and multiple tables must have the same structure.
In the case of partitioned tables, you can configure the table index range. For example, 'table_[0-99]' indicates reading 'table_0','table_1','table_2' up to 'table_99'. If your table number suffix length is consistent, such as 'table_000','table_001','table_002' up to 'table_999', you can configure it as '"table": ["table_00[0-9]", "table_0[10-99]", "table_[100-999]"]'.
If the data source network is not connected and the table information cannot be fetched directly, you can manually enter the table name. Data synchronization can still be performed when the DataInLong network is connected.
Add Sharded Database/Table
Applicable to sharding scenarios. Click to configure multiple data sources, databases, and table information. In sharding scenarios, ensure all table structures are consistent. The task configuration will default to displaying and using the structure of the first table to obtain data.
Table Primary Key
Select a field as the primary key for the target table.
Read Mode
Full + Increment: Data synchronization is divided into full and increment phases. After the full phase is completed, the task enters the increment phase. The full phase will synchronize historical data in the database, and the incremental phase starts synchronizing from the binlog cdc location after the task starts.
Increment Only: Only synchronize data from the binlog cdc position after the task starts.
Parameter (optional)
You can configure parameters according to business needs.
Real-time Single Table Write Node Configuration
﻿
﻿
﻿
Parameters
Description
Data Destination
Dameng DM Data Source to be written to.
Database
Supports selection or manual input of the database name to write to
By default, the database bound to the data source is used as the default database. Other databases need to be manually entered.
If the data source network is not connected and the database information cannot be fetched directly, you can manually enter the database name. Data synchronization can still be performed when the DataInLong network is connected.
Schema
Support selection or manual entry of the Dameng DM Data Mode to be written to.
If the data source network is not connected and the Schema information cannot be fetched directly, you can manually enter the Schema name. Data synchronization can still be performed when the DataInLong network is connected.
Table
Support selection or manual entry of the table name to be written to.
If the data source network is not connected and the table information cannot be fetched directly, you can manually enter the table name. Data synchronization can still be performed when the DataInLong network is connected.
Advanced Settings (Optional)
You can configure parameters according to business needs.
Supported Data Types
Read 
The supported data types for Dameng DM reading and their conversion relationships are as follows (when processing Dameng DM, the data types of the Dameng DM data source and the data processing engine will be mapped first):
DM Types
Internal Types
Remarks
INT
INT
﻿
TINYINT
TINYINT
TINYINT(1) maps to BOOLEAN 
Add an option to support mapping TINYINT(1) to either bool or tinyint
SMALLINT
SMALLINT
﻿
BIGINT
Long
﻿
REAL
DOUBLE
﻿
FLOAT
FLOAT
﻿
DOUBLE
DOUBLE
﻿
DECIMAL
DECIMAL
Using the actual precision of the user database p<=38 maps to DECIMAL, 38 < p <= 65 maps to String
CHAR
STRING
﻿
VARCHAR
STRING
﻿
TEXT
STRING
﻿
DATE
DATE
﻿
TIMESTAMP
TIMESTAMP
﻿
Write 
The supported data types for Dameng DM writing and their conversion relationships are as follows:
Internal Types
DM Types
INT
INT,NUMBER,INTEGER
TINYINT
TINYINT(3)
SMALLINT
SMALLINT
BIGINT
BIGINT
FLOAT
FLOAT
DOUBLE
REAL,DOUBLE,DECIMAL,DOUBLE PRECISION
BOOLEAN 
BIT(1)
CHAR
CHARACTER(1)
VARCHAR
VARBINARY,BYTE,VARCHAR(8188),VARCHAR2(8188),LONGVARCHAR(2147483647)
TEXT
CLOB,BINARY,TIME WITH TIME ZONE
TINYTEXT
TIME
DATE
DATE,DATETIME,DATETIME WITH TIME ZONE
TIMESTAMP
TIMESTAMP,TIMESTAMP WITH LOCAL TIME ZONE
BLOB
BLOB,IMAGE,LONGVARBINARY

Was this page helpful?

You can also Contact Sales or Submit a Ticket for help.

Yes

Parameters	Description
Data Source	Select Dameng DM Data Source.
Database	Support selecting or manually entering the name of the required database. By default, the database bound to the data source is used as the default database. Other databases need to be manually entered. If the data source network is not connected and the database information cannot be fetched directly, you can manually enter the database name. Data synchronization can still be performed when the DataInLong network is connected.
Schema	Support selecting or manually entering the available schema under the data source.
Table	Support selecting or manually entering the table name to be read. In the case of sharding, multiple table names can be selected or entered in the Dameng DM source, and multiple tables must have the same structure. In the case of partitioned tables, you can configure the table index range. For example, 'table_[0-99]' indicates reading 'table_0','table_1','table_2' up to 'table_99'. If your table number suffix length is consistent, such as 'table_000','table_001','table_002' up to 'table_999', you can configure it as '"table": ["table_00[0-9]", "table_0[10-99]", "table_[100-999]"]'. If the data source network is not connected and the table information cannot be fetched directly, you can manually enter the table name. Data synchronization can still be performed when the DataInLong network is connected.
Add Sharded Database/Table	Applicable to sharding scenarios. Click to configure multiple data sources, databases, and table information. In sharding scenarios, ensure all table structures are consistent. The task configuration will default to displaying and using the structure of the first table to obtain data.
Table Primary Key	Select a field as the primary key for the target table.
Read Mode	Full + Increment: Data synchronization is divided into full and increment phases. After the full phase is completed, the task enters the increment phase. The full phase will synchronize historical data in the database, and the incremental phase starts synchronizing from the binlog cdc location after the task starts. Increment Only: Only synchronize data from the binlog cdc position after the task starts.
Parameter (optional)	You can configure parameters according to business needs.

Parameters	Description
Data Destination	Dameng DM Data Source to be written to.
Database	Supports selection or manual input of the database name to write to By default, the database bound to the data source is used as the default database. Other databases need to be manually entered. If the data source network is not connected and the database information cannot be fetched directly, you can manually enter the database name. Data synchronization can still be performed when the DataInLong network is connected.
Schema	Support selection or manual entry of the Dameng DM Data Mode to be written to. If the data source network is not connected and the Schema information cannot be fetched directly, you can manually enter the Schema name. Data synchronization can still be performed when the DataInLong network is connected.
Table	Support selection or manual entry of the table name to be written to. If the data source network is not connected and the table information cannot be fetched directly, you can manually enter the table name. Data synchronization can still be performed when the DataInLong network is connected.
Advanced Settings (Optional)	You can configure parameters according to business needs.

DM Types	Internal Types	Remarks
INT	INT
TINYINT	TINYINT	TINYINT(1) maps to BOOLEAN Add an option to support mapping TINYINT(1) to either bool or tinyint
SMALLINT	SMALLINT
BIGINT	Long
REAL	DOUBLE
FLOAT	FLOAT
DOUBLE	DOUBLE
DECIMAL	DECIMAL	Using the actual precision of the user database p<=38 maps to DECIMAL, 38 < p <= 65 maps to String
CHAR	STRING
VARCHAR	STRING
TEXT	STRING
DATE	DATE
TIMESTAMP	TIMESTAMP

Internal Types	DM Types
INT	INT,NUMBER,INTEGER
TINYINT	TINYINT(3)
SMALLINT	SMALLINT
BIGINT	BIGINT
FLOAT	FLOAT
DOUBLE	REAL,DOUBLE,DECIMAL,DOUBLE PRECISION
BOOLEAN	BIT(1)
CHAR	CHARACTER(1)
VARCHAR	VARBINARY,BYTE,VARCHAR(8188),VARCHAR2(8188),LONGVARCHAR(2147483647)
TEXT	CLOB,BINARY,TIME WITH TIME ZONE
TINYTEXT	TIME
DATE	DATE,DATETIME,DATETIME WITH TIME ZONE
TIMESTAMP	TIMESTAMP,TIMESTAMP WITH LOCAL TIME ZONE
BLOB	BLOB,IMAGE,LONGVARBINARY

tencent cloud

New User Offers

Next-Generation CDN：EdgeOne

Elasticsearch Service free trial

Free Tier

Tencent Cloud Startup Program

Special Offers

Lighthouse Special Offers

Cloud Object Storage Special Offers

Featured Products

New Products

Education

Tencent Cloud Online Education Solutions

Gaming

Gaming Solution

Game Media Solutions

E-commerce

E-commerce retail solutions

Audio & Video

Audio/Video Solution

LVB Recording Solution

Interactive Classroom Solution

Interactive Live Streaming Solution

Audio Chat Social Networking Solution

Financial Services

Financial Services Solution

Compute

Cloud Virtual Machine

Auto Scaling

Batch Compute

CVM Dedicated Host

Database

TencentDB for MySQL

TencentDB for Redis®

TencentDB for CTSDB

TDSQL for MySQL

Data Transfer Service

TencentDB for MongoDB

TencentDB for PostgreSQL

TencentDB for SQL Server

Video Service

Cloud Streaming Services

Video on Demand

Media Processing Service

Cloud Application Rendering

Cloud Contact Center

Game Multimedia Engine

Chat

Real-time Communication

Tencent Effect SDK

AI and Machine Learning

Image Creation Large Model

Face Fusion

eKYC

Optical Character Recognition

Video Creation Large Model

Industry Applications

Tencent HealthCare Omics Platform

Container and Middleware

TDMQ for CKafka

Serverless Cloud Function

Tencent Kubernetes Engine

Tencent Kubernetes Engine for Serverless

Networking

Cloud Load Balancer

Virtual Private Cloud

Direct Connect

Cloud Connect Network

NAT Gateway

VPN Connection

Bandwidth Package

Anycast Internet Acceleration

Elastic Network Interface

Flow Logs

Global Application Acceleration Platform

Security

Captcha

Cloud Workload Protection Platform

Data Security Governance Center

Key Management Service