Elasticsearch/Tencent Cloud Elasticsearch Data Source

Tencent Cloud Elasticsearch data source configuration is the same as Elasticsearch data source configuration. Here, we will explain using the Elasticsearch data source as an example.
Supported Editions
Supports Elasticsearch 6.x, 7.x versions.
Use Limits
1. Elasticsearch Reader will obtain shard information from the server side for data synchronization, ensuring that server-side shards are alive during task synchronization. Otherwise, there is a risk of data inconsistency.
2. Synchronization of fields with type scaled_float is not supported.
3. Synchronization of indexes with fields containing the keyword $ref is not supported.
Elasticsearch Offline Single Table Read Node Configuration
﻿
﻿
﻿
Parameters
Description
Data Source
Select an available Elasticsearch data source from the current project.
Index
Supports multiple index names or regular expressions. Use wildcards (*) for index name regular expressions, e.g., index_*.
ES Version
Determine the ES version based on the data source and index.
Split Key
Specify the field for data sharding. After specifying, concurrent tasks will be launched for data synchronization. You can use a column in the source data table as the partition key. It is recommended to use the primary key or indexed column as the partition key.
Search Condition (Optional)
Use JSON format for search.
Elasticsearch Offline Single Table Write Node Configuration
﻿
﻿
﻿
Parameters
Description
Data Destination
Select an available Elasticsearch data source from the current project.
Index
Index Name in Elasticsearch.
Dynamic Mapping
Definition: When an unknown field is found in the document, decide whether the synchronization task should use Elasticsearch's dynamic mapping mechanism to add a mapping for the field.
Enable: Retain Elasticsearch's automatic mappings.
Disable: Default is off. Generate and update Elasticsearch mappings according to the columns configured in the synchronization task.
In Elasticsearch 7.x, the default type is _doc. When using Elasticsearch auto mappings, please configure _doc and set esVersion to 7.
Clear original index data
Manually choose whether to clear the original index data:
No: Retain the existing data in the index before importing new data.
Yes: Delete the original index and rebuild a new index with the same name before importing new data. This action will remove the data under that index.
Write mode
Support two write modes: insert and update:
Insert: All data is directly inserted.
Update: Update data if the same primary key exists; otherwise, insert.
Primary Key Value Acquisition Method
Supports three value methods:
Source table primary key: Use the primary key of the source table as the document's id.
Composite primary key: Use multiple columns from the source table to determine the document's id.
No primary key: Generate a default _id value.
Batch Submission Size
Batch submission record size for one-time submissions: This value can significantly reduce the network interactions between the data synchronization system and Elasticsearch and improve overall throughput. If set too high, it may cause OOM exceptions in the data synchronization process.
Advanced Settings (Optional)
You can configure parameters according to business needs.
Data type conversion support
Read
ElasticSearch data types
Internal Types
byte,short,integer,long,unsigned long
Long
float,double,half_float
Double
string,text,keyword,integer_range,long_range,float_range,double_range,date_range,array,object,nested,flattened,geo_point,geo_shape
String
date
Date
binary
Bytes
boolean
Boolean
Write
Internal Types
ElasticSearch data types
Long
byte,short,integer,long
Double
float,double
String
string,text,keyword,object,nested,geo_point,geo_shape,ip,binary,completion
Date
date
Boolean
boolean
FAQs
After ES writer clears the index, the automatically rebuilt index's field types do not meet expectations
Problem information:
When configuring the ES writer node to "Clean original index data" and choosing "Yes" for rerunning tasks, the ES writer will not actively create the index. When writing data directly, ES will auto-create the index based on data inference. The new index field types may differ from the old version.
﻿﻿
Cause:
The ES writer clears the original index data by directly deleting the index. It does not have a syntax similar to MySQL's truncate table. After the index is deleted, ES server infers the index field types based on the new data, which may differ from the field types of the original index.
Solution:
Define index field types in advance through ES template.
Creation example: (merlion_suggest_words_* applies to all indexes starting with merlion_suggest_words_).
PUT _template/merlion_suggest_words
{
  "template": "merlion_suggest_words_*",
  "order": 1,
  "mappings": {
    "properties": {
      "biz_owner": {
        "type": "keyword"
      },
      "create_time": {
        "type": "date",
        "format": "yyyy-MM-dd HH:mm:ss||date_hour_minute_second||strict_date_optional_time||epoch_millis"
      },
      "is_deleted": {
        "type": "short"
      },
      "sug_words": {
        "type": "keyword"
      },
      "language": {
        "type": "keyword"
      }
    }
  },
  "settings": {
    "refresh_interval": "5s",
    "number_of_replicas": 1,
    "number_of_shards": 1
  }
}
﻿

Was this page helpful?

You can also Contact Sales or Submit a Ticket for help.

Yes

Parameters	Description
Data Source	Select an available Elasticsearch data source from the current project.
Index	Supports multiple index names or regular expressions. Use wildcards () for index name regular expressions, e.g., index_.
ES Version	Determine the ES version based on the data source and index.
Split Key	Specify the field for data sharding. After specifying, concurrent tasks will be launched for data synchronization. You can use a column in the source data table as the partition key. It is recommended to use the primary key or indexed column as the partition key.
Search Condition (Optional)	Use JSON format for search.

Parameters	Description
Data Destination	Select an available Elasticsearch data source from the current project.
Index	Index Name in Elasticsearch.
Dynamic Mapping	Definition: When an unknown field is found in the document, decide whether the synchronization task should use Elasticsearch's dynamic mapping mechanism to add a mapping for the field. Enable: Retain Elasticsearch's automatic mappings. Disable: Default is off. Generate and update Elasticsearch mappings according to the columns configured in the synchronization task. In Elasticsearch 7.x, the default type is _doc. When using Elasticsearch auto mappings, please configure _doc and set esVersion to 7.
Clear original index data	Manually choose whether to clear the original index data: No: Retain the existing data in the index before importing new data. Yes: Delete the original index and rebuild a new index with the same name before importing new data. This action will remove the data under that index.
Write mode	Support two write modes: insert and update: Insert: All data is directly inserted. Update: Update data if the same primary key exists; otherwise, insert.
Primary Key Value Acquisition Method	Supports three value methods: Source table primary key: Use the primary key of the source table as the document's id. Composite primary key: Use multiple columns from the source table to determine the document's id. No primary key: Generate a default _id value.
Batch Submission Size	Batch submission record size for one-time submissions: This value can significantly reduce the network interactions between the data synchronization system and Elasticsearch and improve overall throughput. If set too high, it may cause OOM exceptions in the data synchronization process.
Advanced Settings (Optional)	You can configure parameters according to business needs.

ElasticSearch data types	Internal Types
byte,short,integer,long,unsigned long	Long
float,double,half_float	Double
string,text,keyword,integer_range,long_range,float_range,double_range,date_range,array,object,nested,flattened,geo_point,geo_shape	String
date	Date
binary	Bytes
boolean	Boolean

tencent cloud

New User Offers

Next-Generation CDN：EdgeOne

Elasticsearch Service free trial

Free Tier

Tencent Cloud Startup Program

Special Offers

Lighthouse Special Offers

Cloud Object Storage Special Offers

Featured Products

New Products

Education

Tencent Cloud Online Education Solutions

Gaming

Gaming Solution

Game Media Solutions

E-commerce

E-commerce retail solutions

Audio & Video

Audio/Video Solution

LVB Recording Solution

Interactive Classroom Solution

Interactive Live Streaming Solution

Audio Chat Social Networking Solution

Financial Services

Financial Services Solution

Compute

Cloud Virtual Machine

Auto Scaling

Batch Compute

CVM Dedicated Host

Database

TencentDB for MySQL

TencentDB for Redis®

TencentDB for CTSDB

TDSQL for MySQL

Data Transfer Service

TencentDB for MongoDB

TencentDB for PostgreSQL

TencentDB for SQL Server

Video Service

Cloud Streaming Services

Video on Demand

Media Processing Service

Cloud Application Rendering

Cloud Contact Center

Game Multimedia Engine

Chat

Real-time Communication

Tencent Effect SDK

AI and Machine Learning

Image Creation Large Model

Face Fusion

eKYC

Optical Character Recognition

Video Creation Large Model

Industry Applications

Tencent HealthCare Omics Platform

Container and Middleware

TDMQ for CKafka

Serverless Cloud Function

Tencent Kubernetes Engine

Tencent Kubernetes Engine for Serverless

Networking

Cloud Load Balancer

Virtual Private Cloud

Direct Connect

Cloud Connect Network

NAT Gateway

VPN Connection

Bandwidth Package

Anycast Internet Acceleration

Elastic Network Interface

Flow Logs

Global Application Acceleration Platform

Security

Captcha

Cloud Workload Protection Platform

Data Security Governance Center

Key Management Service