tencent cloud

PrevNext

Feedback

search by keyword

Recent Pages

Documentation
Tencent Cloud WeData
    Documentation
    Download PDF

    ElasticSearch Data Source

    Last updated: 2024-11-01 17:00:28
    Download PDF
      DataInLong provides real-time writing capability for Elasticsearch. This document introduces the current support status for Elasticsearch real-time data synchronization.

      Supported Editions

      Currently, DataInLong supports database-level real-time writing for Elasticsearch. To use the real-time writing capability, the following version restrictions must be followed:
      Type
      Edition
      Elasticsearch
      5.x, 6.x, 7.x

      Use Limits

      Currently, only one real-time full database link is supported from TDSQL-C MySQL to Elasticsearch.
      Elasticsearch primary key values default to the _id value, which cannot be modified.
      Currently, only the _id of the document is supported as the partition column.
      Elasticsearch currently uses dynamic field mapping for table creation. The DDL change strategy does not support automatic column type modification, column deletion, or renaming a column (it will be recognized as adding a new column).
      Kerberos authentication is not supported for Elasticsearch.

      Real-time full database writing configuration

      Supported Data Sources

      Currently, the following source types are supported for real-time full database synchronization to the Elasticsearch target end:

      Data Target Configuration

      There are some differences in the configuration between Elasticsearch versions 6.x and above (inclusive) and below:
      Configuration for versions 6.x and above (inclusive) is as follows:
      
      
      
      Configuration for versions below 6.x is as follows:
      Parameters
      Description
      Data Destination
      Select the target data source to be synchronized.
      ES Version
      The corresponding version is automatically displayed based on the selected data source
      Write Mode
      Upsert: Update Write, update all fields of each record (currently only this mode is supported).
      Index/Type Matching Strategy
      ES index and type name matching rules:
      Index versions 6.x and above (inclusive) default to the same name as the source table; type defaults to _doc and cannot be modified
      Index versions below 6.x default to the same name as the source database; type defaults to the same name as the source table
      Custom Definition: Supports generating target index/type names using a combination of built-in parameters and strings.
      Note:
      Example: If the source table name is table1, and the mapping rule is ${table_name_di_src}_inlong, the data from table1 will be finally mapped to table1_inlong.
      Gradual Value Method
      Currently only supports default generation of _id value

      Log collection write node configuration

      Parameters
      Description
      Data Source
      Select an available Elasticsearch data source from the current project.
      Index
      Index name in Elasticsearch data source.
      type
      Automatically identify based on index; Elasticsearch version 7.X defaults type to _doc.
      Write Mode
      Elasticsearch only supports row-by-row updating, updating all fields of each record.
      Primary key value method
      Supports three value methods:
      Source table primary key: document id uses the primary key of the source table.
      Composite Primary Key: document id uses multiple columns of the source table to determine.
      No Primary Key: defaults to generating _id value.
      Enabling a Route
      Does Elasticsearch enable routing partition index data? After enabling the routing feature, you can control which partition to use for storing documents in Elasticsearch.
      Advanced Settings (optional)
      You can configure parameters according to business needs.

      Write Data Type Conversion Supported

      Internal Types
      JSON Types
      CHAR / VARCHAR / STRING
      string
      BOOLEAN
      boolean
      BINARY / VARBINARY
      string with encoding: base64
      DECIMAL
      number
      TINYINT
      number
      SMALLINT
      number
      INT
      number
      BIGINT
      number
      FLOAT
      number
      DOUBLE
      number
      DATE
      string with format: date
      TIME
      string with format: time
      TIMESTAMP
      string with format: date-time
      TIMESTAMP_WITH_LOCAL_TIME_ZONE
      string with format: date-time (with UTC time zone)
      INTERVAL
      number
      ARRAY
      array
      MAP / MULTISET
      object
      ROW
      object
      
      Contact Us

      Contact our sales team or business advisors to help your business.

      Contact Us
      Technical Support

      Open a ticket if you're looking for further assistance. Our Ticket is 7x24 avaliable.

      Submit a Ticket
      7x24 Phone Support
      Copyright © 2013-2024 Tencent Cloud. All Rights Reserved.
      Privacy PolicyLegalCookie Policy