ElasticSearch Data Source

DataInLong provides real-time writing capability for Elasticsearch. This document introduces the current support status for Elasticsearch real-time data synchronization.
Supported Editions
Currently, DataInLong supports database-level real-time writing for Elasticsearch. To use the real-time writing capability, the following version restrictions must be followed:
Type
Edition
Elasticsearch
5.x, 6.x, 7.x
Use Limits 
Currently, only one real-time full database link is supported from TDSQL-C MySQL to Elasticsearch.
Elasticsearch primary key values default to the _id value, which cannot be modified.
Currently, only the _id of the document is supported as the partition column.
Elasticsearch currently uses dynamic field mapping for table creation. The DDL change strategy does not support automatic column type modification, column deletion, or renaming a column (it will be recognized as adding a new column).
Kerberos authentication is not supported for Elasticsearch.
Real-time full database writing configuration
Supported Data Sources
Currently, the following source types are supported for real-time full database synchronization to the Elasticsearch target end:
TDSQL-C MySQL Whole Database Source Configuration Details﻿
Data Target Configuration
There are some differences in the configuration between Elasticsearch versions 6.x and above (inclusive) and below:
Configuration for versions 6.x and above (inclusive) is as follows:
﻿
﻿
﻿
Configuration for versions below 6.x is as follows:
Parameters
Description
Data Destination
Select the target data source to be synchronized.
ES Version
The corresponding version is automatically displayed based on the selected data source
Write Mode
Upsert: Update Write, update all fields of each record (currently only this mode is supported).
Index/Type Matching Strategy
ES index and type name matching rules:
Index versions 6.x and above (inclusive) default to the same name as the source table; type defaults to _doc and cannot be modified
Index versions below 6.x default to the same name as the source database; type defaults to the same name as the source table
Custom Definition: Supports generating target index/type names using a combination of built-in parameters and strings.
Note:
Example: If the source table name is table1, and the mapping rule is ${table_name_di_src}_inlong, the data from table1 will be finally mapped to table1_inlong.
Gradual Value Method
Currently only supports default generation of _id value
Log collection write node configuration
Parameters
Description
Data Source
Select an available Elasticsearch data source from the current project.
Index
Index name in Elasticsearch data source.
type
Automatically identify based on index; Elasticsearch version 7.X defaults type to _doc.
Write Mode
Elasticsearch only supports row-by-row updating, updating all fields of each record.
Primary key value method
Supports three value methods:
Source table primary key: document id uses the primary key of the source table.
Composite Primary Key: document id uses multiple columns of the source table to determine.
No Primary Key: defaults to generating _id value.
Enabling a Route
Does Elasticsearch enable routing partition index data? After enabling the routing feature, you can control which partition to use for storing documents in Elasticsearch.
Advanced Settings (optional)
You can configure parameters according to business needs.
Write Data Type Conversion Supported
Internal Types
JSON Types
CHAR / VARCHAR / STRING
string
BOOLEAN
boolean
BINARY / VARBINARY
string with encoding: base64
DECIMAL
number
TINYINT
number
SMALLINT
number
INT
number
BIGINT
number
FLOAT
number
DOUBLE
number
DATE
string with format: date
TIME
string with format: time
TIMESTAMP
string with format: date-time
TIMESTAMP_WITH_LOCAL_TIME_ZONE
string with format: date-time (with UTC time zone)
INTERVAL
number
ARRAY
array
MAP / MULTISET
object
ROW
object
﻿

Was this page helpful?

You can also Contact Sales or Submit a Ticket for help.

Yes

Parameters	Description
Data Destination	Select the target data source to be synchronized.
ES Version	The corresponding version is automatically displayed based on the selected data source
Write Mode	Upsert: Update Write, update all fields of each record (currently only this mode is supported).
Index/Type Matching Strategy	ES index and type name matching rules: Index versions 6.x and above (inclusive) default to the same name as the source table; type defaults to _doc and cannot be modified Index versions below 6.x default to the same name as the source database; type defaults to the same name as the source table Custom Definition: Supports generating target index/type names using a combination of built-in parameters and strings. Note: Example: If the source table name is table1, and the mapping rule is ${table_name_di_src}_inlong, the data from table1 will be finally mapped to table1_inlong.
Gradual Value Method	Currently only supports default generation of _id value

Parameters	Description
Data Source	Select an available Elasticsearch data source from the current project.
Index	Index name in Elasticsearch data source.
type	Automatically identify based on index; Elasticsearch version 7.X defaults type to _doc.
Write Mode	Elasticsearch only supports row-by-row updating, updating all fields of each record.
Primary key value method	Supports three value methods: Source table primary key: document id uses the primary key of the source table. Composite Primary Key: document id uses multiple columns of the source table to determine. No Primary Key: defaults to generating _id value.
Enabling a Route	Does Elasticsearch enable routing partition index data? After enabling the routing feature, you can control which partition to use for storing documents in Elasticsearch.
Advanced Settings (optional)	You can configure parameters according to business needs.

Internal Types	JSON Types
CHAR / VARCHAR / STRING	string
BOOLEAN	boolean
BINARY / VARBINARY	string with encoding: base64
DECIMAL	number
TINYINT	number
SMALLINT	number
INT	number
BIGINT	number
FLOAT	number
DOUBLE	number
DATE	string with format: date
TIME	string with format: time
TIMESTAMP	string with format: date-time
TIMESTAMP_WITH_LOCAL_TIME_ZONE	string with format: date-time (with UTC time zone)
INTERVAL	number
ARRAY	array
MAP / MULTISET	object
ROW	object

tencent cloud

New User Offers

Next-Generation CDN：EdgeOne

Elasticsearch Service free trial

Free Tier

Tencent Cloud Startup Program

Special Offers

Lighthouse Special Offers

Cloud Object Storage Special Offers

Featured Products

New Products

Education

Tencent Cloud Online Education Solutions

Gaming

Gaming Solution

Game Media Solutions

E-commerce

E-commerce retail solutions

Audio & Video

Audio/Video Solution

LVB Recording Solution

Interactive Classroom Solution

Interactive Live Streaming Solution

Audio Chat Social Networking Solution

Financial Services

Financial Services Solution

Compute

Cloud Virtual Machine

Auto Scaling

Batch Compute

CVM Dedicated Host

Database

TencentDB for MySQL

TencentDB for Redis®

TencentDB for CTSDB

TDSQL for MySQL

Data Transfer Service

TencentDB for MongoDB

TencentDB for PostgreSQL

TencentDB for SQL Server

Video Service

Cloud Streaming Services

Video on Demand

Media Processing Service

Cloud Application Rendering

Cloud Contact Center

Game Multimedia Engine

Chat

Real-time Communication

Tencent Effect SDK

AI and Machine Learning

Image Creation Large Model

Face Fusion

eKYC

Optical Character Recognition

Video Creation Large Model

Industry Applications

Tencent HealthCare Omics Platform

Container and Middleware

TDMQ for CKafka

Serverless Cloud Function

Tencent Kubernetes Engine

Tencent Kubernetes Engine for Serverless

Networking

Cloud Load Balancer

Virtual Private Cloud

Direct Connect

Cloud Connect Network

NAT Gateway

VPN Connection

Bandwidth Package

Anycast Internet Acceleration

Elastic Network Interface

Flow Logs

Global Application Acceleration Platform

Security

Captcha

Cloud Workload Protection Platform

Data Security Governance Center

Key Management Service