tencent cloud

PrevNext

Feedback

search by keyword

Recent Pages

Documentation
Elasticsearch Service
    Documentation
    Download PDF

    TKE Log access

    Last updated: 2024-12-04 16:06:59
    Download PDF

      Prerequisites

      A Tencent Cloud account has been created. For account creation, see signing up.
      If you log in with a sub-user account, ensure it has read and write permissions on ES.

      Operation Steps

      Logging in to the Console

      1. Log in to the ES console.
      2. In the top menu, select Region. Currently supported regions include Beijing, Shanghai, Guangzhou, Nanjing, and Hong Kong (China).
      3. In the left sidebar, choose Log Analysis under the Serverless mode.

      Creating a Project Space

      1. Click Create a project.
      2. Enter the Project Name, which can include 1 - 20 characters, including Chinese characters, letters, digits.
      3. Click Confirm. Once validated, the project space will be created.
      
      Note
      In Elasticsearch Serverless Log Analysis, you can create an index and subsequently write data via API or access data sources such as CVM and TKE via the Data Access tab of the corresponding index. You can also set up data access during index creation, enabling one-stop access for CVM and TKE log access. Below is the instruction for one-step TKE log access setup.

      TKE Log Access

      On the ES Serverless Log Analysis homepage, select TKE to enter the TKE log access page.
      

      Data Source Settings

      Region: The region where the TKE cluster is located.
      VPC: Required. The VPC where the TKE cluster is located.
      TKE Cluster ID to Be Collected: Required. The ID of the TKE cluster to collect logs from, which should be in a running status and a standard cluster. If you need log collection for Serverless clusters (EKS), contact us via submiting a ticket.
      Based on Namespace/Host Path: Required. For Namespace, select Include/Exclude from the first dropdown, and select one or more namespaces from the second dropdown (multi-select supported, but excluding all namespaces is not allowed). For host path-based collection, enter the Absolute Path on the host, for example, /var/log/*.log.
      Pod Tag: Optional. You can create multiple Pod labels, which are logically connected using AND.
      Container Name: Optional. The specified container name should be within the target cluster and namespace. If it is left empty, Filebeat will collect all containers within the namespace that match the specified Pod tags.
      

      Collection Settings

      Basic Settings
      Collection Policy: Supports full collection and incremental collection. Once created, the collection policy cannot be modified. Full collection will collect historical logs as well as logs generated after the Filebeat configuration takes effect. Incremental collection will only collect logs generated after the Filebeat configuration becomes active.
      Collection Parsing
      Collection Template: If you need a quick setup or are testing, you can select a collection template based on your log output format. After confirmation, return to the interface to modify the log sample with actual log data, enabling a fast completion of the collection parsing settings.
      
      Collection Mode: Supports both single-line and multi-line modes; once set, the mode cannot be modified.
      Single-line text log: Each line in the log file represents a single log entry, separated by a newline character.
      Multi-line text log: Each log entry spans multiple lines, such as Java stack traces. In this mode, you need to configure a log sample and a regex pattern for line beginnings. Filebeat uses the regex to identify the start of each log entry, treating unmatched lines as part of the current log entry until the next matched line beginning appears. Once you enter a log sample, the system automatically generates a default regex pattern for line beginnings. You can also customize this pattern, with highlighted text in the input box indicating the matched line beginnings.
      Note:
      Ensure that actual scenario logs are used to facilitate automatic extraction of the leading line regular expression.
      
      
      Extraction Settings: Extraction mode can be set to full-text log, JSON format, or delimiter-based. Once set, the extraction mode cannot be modified. Details are as follows:
      Full-text log
      JSON format
      Delimiter
      No key-value extraction is performed. The log content is stored in a field named message, which can be retrieved and analyzed using automatic tokenization.
      For example, a single-line raw log might look like:
      Tue Jan 01 00:00:00 CST 2023 Running: Content of processing something
      When collected in the index, this data would appear as:
      massage:Tue Tue Jan 01 00:00:00 CST 2023 Running: Content of processing something
      For logs in standard JSON format, we can extract fields based on the Key: Value pairs within the log.
      
      For example, suppose a JSON log entry is as follows:
      {"pid":321,"name":"App01","status":"WebServer is up and running"}
      After structuring, the log entry will appear as follows:
      {
    "pid":321,
    "name":"App01",
    "status":"WebServer is up and running"
}
      For logs with content separated by a fixed delimiter, we can extract key-value pairs based on the specified delimiter. The delimiter can be a single character or a string, which can be selected or entered in the console.
      
      For example, if a log entry is as follows:
      321 - App01 - WebServer is up and running
      With the delimiter set to -, this log entry will be split into three fields. Unique keys can then be assigned to these fields in the extraction results, as shown below:
      pid: pid
      name: App01
      status: WebServer is up and running
      Extraction Results: When the extraction mode is set to JSON format or Delimiter, a sample log can be provided for automatic extraction:
      If the extraction mode is JSON format, the system will automatically populate the extracted Keys and Values. If it is deselected, the respective fields will not be written to the index.
      If the extraction mode is Delimiter, the system will automatically populate the extracted Values, allowing you to assign unique Keys to each Value. If it is deselected, the corresponding fields will not be written to the index.
      Built-in Fields: When you configure TKE log collection in the console, Filebeat will add information such as the log source and timestamp as Key-Value pairs in the logs. These fields are considered built-in fields. If a Key in your business log matches a built-in field name, the business log field content takes precedence, and the corresponding built-in field will not be added to the index. The meanings of the built-in fields are as follows:
      Built-in Field Name
      Meaning
      log.file.path
      Path where the log is stored
      kubernetes.pod.ip
      IP address of the Pod containing the log
      kubernetes.pod.name
      Name of the Pod containing the log
      kubernetes.node.hostname
      Name of the host containing the log
      @timestamp
      Timestamp of when the log was collected
      
      Preserve Original Logs: When it is selected, the original log content, prior to parsing and extraction, will be preserved in this field.
      Record Parsing Errors: If the extraction mode is set to Delimiter, you can choose whether to log parsing errors. When it is selected, any errors encountered during parsing will be uploaded to this field as values.

      Index Settings

      Project Space: You can assign the index to a specific project space for easier management of related business indexes.
      Index Name: Supports a length of 1 to 100 characters, including lowercase letters, numbers, -, _, ;, @, &, =, !, ', %, $, ., +, (, ).
      Field Mapping
      Dynamic generation: Enabled by default. When it is enabled, the system automatically parses the incoming data to generate the field mappings for the index.
      Input sample auto-configuration: If Dynamic Generation is disabled, you can use Input Sample Auto-Configuration to generate the field mappings. Input a sample in JSON format, and the system will automatically validate it. Once validated, the relevant fields will be mapped in the field mapping table.
      
      The field mapping divides the original data into distinct terms by fields (key:value) to construct the index, enabling retrieval based on this mapping. Specific details are as follows:
      Parameter
      Feature Description
      Field name
      The name of the field within the data being written.
      Field type
      The data type of the field. The interface supports 9 types: text, date, boolean, keyword, long, double, integer, ip, and geo_point. Additional field types can be supported in JSON Editing Pattern. For more details, see Official Documentation.
      Includes Chinese
      Enable this option if the field includes Chinese characters that need to be retrieved. When it is enabled, the text field will use the ik_max_word tokenizer by default.
      Enable index
      When it is enabled, an index is built for this field to facilitate retrieval.
      Enable statistics
      When it is enabled, this field's values can be analyzed statistically, which will increase index storage.
      Time Field The time field refers to a field of type date in the actual data. Once the index is created, this field cannot be modified.
      Note:
      The time field has indexing and statistics enabled by default, and these settings cannot be disabled.
      Data Storage Duration:
      1.1 You can set the storage duration of the data. By default, it is set to retain data for 30 days, though you also have the option to set it to permanent storage.
      
      1.2 Once the information is entered correctly, click Create to complete the TKE log collection.
      
      Contact Us

      Contact our sales team or business advisors to help your business.

      Contact Us
      Technical Support

      Open a ticket if you're looking for further assistance. Our Ticket is 7x24 avaliable.

      Submit a Ticket
      7x24 Phone Support
      Copyright © 2013-2024 Tencent Cloud. All Rights Reserved.
      Privacy PolicyLegalCookie Policy