tencent cloud

PrevNext

Feedback

search by keyword

Recent Pages

Documentation
Tencent Kubernetes Engine
    Documentation
    Download PDF

    Using CRD to Configure Log Collection

    Last updated: 2023-05-05 10:38:21
    Download PDF

      Overview

      Besides configuration log collection in the TKE console, you can also configure it by using the Custom Resource Definitions (CRD). CRD supports the collection of container standard outputs, container files, and host files. It also supports multiple log collection formats, and supports shipping logs to different consumers such as CLS and CKafka.

      Prerequisites

      Activate Log Collection in TKE console.

      CRD Overview

      Structure overview

      apiVersion: cls.cloud.tencent.com/v1
      kind: LogConfig                    ## Default value
      metadata:
        name: test       ## CRD resource name, unique in the cluster
      spec:
        clsDetail:					## The configuration for shipping to CLS
          ...
        inputDetail:                  ## Data source configuration for collection
          ...
        kafkaDetail:					## The configuration for shipping to a CKafka or self-built Kafka cluster
          ...
      status:							## CRD resource status
        status: ""
        code: ""						## The error code returned by the called API
        reason: ""					## Error cause

      clsDetail description

      Note
      The topic cannot be modified once it’s specified.
        clsDetail:
          ## If the log topic is created automatically, the names of logset and topic need to be specified at the same time.
          logsetName: test                       ## CLS logset name. Logset for the name will be created automatically if there is not any. If there is the logset, log topic will be created under it.
          topicName: test                        ## CLS log topic name. Log topic for the name will be created automatically if there is not any.
      
          # Select an existing logset and log topic. If the logset is specified but the log topic is not, a log topic will be created automatically.
          logsetId: xxxxxx-xx-xx-xx-xxxxxxxx  ## The ID of the CLS logset. The logset needs to be created in advance in CLS.
          topicId: xxxxxx-xx-xx-xx-xxxxxxxx      ## CLS log topic ID. The log topic needs to be created in CLS in advance and should not be occupied by other collection configurations.
      
          logType: json_log                 ## Log collection format. json_log: json format. delimiter_log: separator-based format. minimalist_log: full text in a single line. multiline_log: full text in multi lines. fullregex_log: full regex format. It defaults to minimalist_log.
          logFormat: xxx                      ## Log formatting method
          period: 30					        ## Lifecycle in days. Value range: 1–3600. `3640` indicates permanent storage.
          partitionCount:                     ## The number (an integer) of log topic partitions. Default value: `1`. Maximum value: `10`.
          tags:                             ## Tag description list. This parameter is used to bind a tag to a log topic. Up to nine tag key-value pairs are supported, and a resource can be bound to only one tag key.
           - key: xxx							  ## Tag key
           value: xxx                      ## Tag value
          autoSplit: false## Whether to enable automatic split (Boolean type). Default value: `true`. 
          maxSplitPartitions:
          storageType: hot.                ## Log topic storage class. Valid values: `hot` (STANDARD); `cold` (STANDARD_IA). Default value: `hot`.
          excludePaths:                      ## Collection path blocklist
            - type: File						  ## Type. Valid values: `File`, `Path`. 
                value: /xx/xx/xx/xx.log         ## The value of `type`
          indexs: 							 ## You can customize the indexing method and field when creating a topic.
            - indexName: ## When a key value or metafield index needs to be configured for a field, the metafield `Key` does not need to be prefixed with `__TAG__.` and is consistent with the one when logs are uploaded. `__TAG__.` will be prefixed automatically for display in the console.
                indexType: ## Valid values: `long`, `text`, `double`
                tokenizer:  ## Field delimiter. Each character represents a delimiter. Only English symbols and \\n\\t\\r are supported. For `long` and `double` fields, leave it empty. For `text` fields, we recommend you use @&?|#()='",;:<>[]{}/ \\n\\t\\r\\ as the delimiter.
                sqlFlag: ## Whether the analysis feature is enabled for the field (Boolean)
                containZH:  ## Whether Chinese characters are contained (Boolean)
          region: ap-xxx                     ## Topic region for cross-region shipping
          userDefineRule: xxxxxx             ## Custom collection rule, which is a serialized JSON string
          extractRule: {}                    ## Extraction and filter rule. If `ExtractRule` is set, `LogType` must be set.

      inputDetail description

        inputDetail:
          type: container_stdout                ## Log collection type, including container_stdout (container standard output), container_file (container file), and host_file (host file)
      
          containerStdout:                      ## Container standard output
            namespace: default    ## The Kubernetes namespace of the container to be collected. Separate multiple namespaces by comma, for example, `default,namespace`. If this field is not specified, it indicates all namespaces. Note that this field cannot be specified if `excludeNamespace` is specified.
            excludeNamespace: nm1,nm2   ## The Kubernetes namespace of the container to be excluded. Separate multiple namespaces by comma, for example, `nm1,nm2`. If this field is not specified, it indicates all namespaces. Note that this field cannot be specified if `namespace` is specified.
            nsLabelSelector: environment in (production),tier in (frontend) ## Filter namespaces by namespace label
            allContainers: false       ## Whether to collect the standard output of all containers in the specified namespace. Note that if `allContainers=true`, you cannot specify `workload`, `includeLabels`, and `excludeLabels` at the same time.
            container: xxx               ## Name of the container of which the logs will be collected. If the name is empty, it indicates the log names of all matching containers will be collected. 
            excludeLabels:  ## Pods with the specified labels will be excluded. This field cannot be specified if `workload`, `namespace`, and `excludeNamespace` are specified.
              key2: value2  ## Pods with multiple values of the same key can be matched. For example, if you enter `environment = production,qa`, Pods with the `production` or `qa` value of the `environment` key will be excluded. Separate multiple values by comma. If you also specify `includeLabels`, Pods in the intersection will be matched.
      
            includeLabels:  ## Pods with the specified labels will be collected. This field cannot be specified if `workload`, `namespace`, and `excludeNamespace` are specified.
              key: value1   ## The `metadata` will be carried in the log collected based on the collection rule and reported to the consumer. Pods with multiple values of the same key can be matched. For example, if you enter `environment = production,qa`, Pods with the `production` or `qa` value of the `environment` key will be matched. Separate multiple values by comma. If you also specify `excludeLabels`, Pods in the intersection will be matched.
      
            metadataLabels:            ## Specify the Pod labels to be collected as the metadata. If this field is not specified, all Pod labels will be collected as the metadata.
            - label1
            customLabels:              ## Custom metadata
              label: l1
      
            workloads:
            container: xxx  ## Name of the container to collect. If this parameter is not specified, it indicates all containers in the workload Pod will be collected.
              kind: deployment  ## Workload type. Supported values include deployment, daemonset, statefulset, job, and cronjob.
              name: sample-app  ## Workload name
              namespace: prod  ## Workload namespace
      
          containerFile:  ## File in the container
            namespace: default  ## The Kubernetes namespace of the container to be collected. A namespace must be specified.	  
            excludeNamespace: nm1,nm2   ## The Kubernetes namespace of the container to be excluded. Separate multiple namespaces by comma, for example, `nm1,nm2`. If this field is not specified, it indicates all namespaces. Note that this field cannot be specified if `namespace` is specified.
            nsLabelSelector: environment in (production),tier in (frontend) ## Filter namespaces by namespace label
            container: xxx          ## The name of container of which the logs will be collected. The * indicates the log names of all matching containers will be collected.
            logPath: /var/logs     ## Log folder. Wildcards are not supported.
            filePattern: app_*.log     ## Log file name. It supports the wildcards [*?]. ***** matches multiple random characters, and **?** matches a single random character.
            customLabels:   ## Custom metadata
              key: value
            excludeLabels:  ## Pods with the specified labels will be excluded. This field cannot be specified if `workload` is specified.
              key2: value2  ## Pods with multiple values of the same key can be matched. For example, if you enter `environment = production,qa`, Pods with the `production` or `qa` value of the `environment` key will be excluded. Separate multiple values by comma. If you also specify `includeLabels`, Pods in the intersection will be matched.
      
            includeLabels:  ## Pods with the specified labels will be collected. This field cannot be specified if `workload` is specified.
              key: value1   ## The `metadata` will be carried in the log collected based on the collection rule and reported to the consumer. Pods with multiple values of the same key can be matched. For example, if you enter `environment = production,qa`, Pods with the `production` or `qa` value of the `environment` key will be matched. Separate multiple values by comma. If you also specify `excludeLabels`, Pods in the intersection will be matched.
            metadataLabels:            ## Specify the Pod labels to be collected as the metadata. If this field is not specified, all Pod labels will be collected as the metadata.
            - label1               ## pod label
            workload:
              container: xxx           ## Name of the container to collect. If this parameter is not specified, it indicates all containers in the workload Pod will be collected.
              name: sample-app             ## Workload name
      
          hostFile:                ## Node file path
            filePattern: '*.log'             ## Log file name. It supports the wildcards "*" and "?". "*" matches multiple random characters, and "?" matches a single random character.
            logPath: /tmp/logs     ## Log file folder. Wildcards are not supported.
            customLabels:   ## Custom metadata
              label1: v1
      extractRule description
      Name
      Type
      Required
      Description
      timeKey
      String
      No
      Time field key name. time_key and time_format must appear in pairs.
      timeFormat
      String
      No
      Time field format. For more information, see the output parameters of the time format description of the strftime function in C language.
      delimiter
      String
      No
      Delimiter for delimited log, which is valid only if log_type is delimiter_log.
      logRegex
      String
      No
      Full log matching rule, which is valid only if log_type is fullregex_log.
      beginningRegex
      String
      No
      First-Line matching rule, which is valid only if log_type is multiline_log or fullregex_log.
      unMatchUpload
      String
      No
      Whether to upload the logs that failed to be parsed. Valid values: true: yes; false: no.
      unMatchedKey
      String
      No
      Key of the failure log
      backtracking
      String
      No
      Size of the data to be rewound in incremental collection mode. Value: -1 (collect all) and 0 (collect increment). It default to -1.
      keys
      Array of String
      No
      Key name of each extracted field. An empty key indicates to discard the field. This parameter is valid only if log_type is delimiter_log. json_log logs use the key of JSON itself.
      filterKeys
      Array of String
      No
      Key of the log to filter. It corresponds to FilterRegex by the index.
      filterRegex
      Array of String
      No
      Regex corresponding to the key of the log to filter. It corresponds to FilterKeys by the index.
      isGBK
      String
      No
      Whether it’s GBK-encoded. Values: 0 (No), 1 (Yes)
      Note: This field may return null, indicating that no valid value was found.
      jsonStandard
      String
      No
      Whether it’s standard JSON. Values: 0 (No), 1 (Yes).
      Note: This field may return null, indicating that no valid value was found.

      kafkaDetail description

        kafkaDetail:
          brokers: x.x.x.x:p     ## (Required) The broker address. Generally, it is domain name:port. If there are more than one address, separate them with ",".
          topic: test      ## 
          kafkaType: CKafka     ## Kafka type. Valid values: `CKafka` (CKafka); `SelfBuildKafka` (self-built Kafka).
          instanceId: xxxx      ## The ID of the CKafka instance when `kafkaType` is `CKafka`
             logType: minimalist_log ## The type of the parsed Kafka log. Valid values: `minimalist_log` or `""` (full text in a single line); `multiline_log` (full text in multiple lines); `json` (JSON).
          timestampFormat: xxx      ## The format of timestamp. It defaults to `double`.
          timestampKey: xxx           ## The key of timestamp. It defaults to `@timestamp`.
          metadata:
            formatType: default   ## Metadata format. Valid values: `default` (default, the same as the EKS Kafka collector); `filebeat` (Filebeat); `fluent-bit` (Fluent Bit).
          messageKey:            ## Specify a key to ship logs to the specified partition. This parameter is not enabled by default, and logs are shipped randomly. When it is enabled, logs with the same key will be shipped to the same partition. You can select the Pod field as the key. For example, if you enter a Pod name, select Field>metadata.name.
            value: Field        ## Topic ID, which is required
            valueFrom:
              fieldRef: 
                fieldPath: metadata.name ## If the key is `Field`, you can select `metadata.name`, `metadata.namespace`, `spec.nodeName`, and `spec.serviceAccountName`.

      status description

      status
      Description
      The status is empty.
      Initial status
      Synced
      Configured successfully
      Stale
      Configuration failed

      Sample CRD

      Sample CRD for the configuration of the container standard output

      All containers
      Specifying a workload
      Specifying Pod labels
      Specify a namespace
      apiVersion: cls.cloud.tencent.com/v1
      kind: LogConfig
      metadata:
        name: "test"
      spec:
        clsDetail:
          .......
          topicId: xxxxxx-xx-xx-xx-xxxxxxxx
        inputDetail:
          containerStdout:
            allContainers: true
            namespace: default,kube-public
          type: container_stdout
      
      Exclude a namespace
      apiVersion: cls.cloud.tencent.com/v1
      kind: LogConfig
      metadata:
        name: "test"
      spec:
        clsDetail:
          ........
          topicId: xxxxxx-xx-xx-xx-xxxxxxxx
        inputDetail:
          containerStdout:
            allContainers: true
            excludeNamespace: kube-system,kube-node-lease
          type: container_stdout
      
      apiVersion: cls.cloud.tencent.com/v1
      kind: LogConfig
      metadata:
        name: "test"
      spec:
        clsDetail:
          ......
          topicId: xxxxxx-xx-xx-xx-xxxxxxxx
        inputDetail:
          containerStdout:
            allContainers: false
            workloads:
            - container: prod
              kind: deployment
              name: sample-app
              namespace: kube-system
          type: container_stdout
      apiVersion: cls.cloud.tencent.com/v1
      kind: LogConfig
      metadata:
        name: test
      spec:
        clsDetail:
          ......
          topicId: xxxxxx-xx-xx-xx-xxxxxxxx
        inputDetail:
          containerStdout:
            container: prod
            excludeLabels:
              key2: v2
            includeLabels:
              key1: v1
            namespace: default,kube-system
          type: container_stdout

      Sample CRD for the configuration of the container file path

      Specifying a workload
      Specifying Pod labels
      apiVersion: cls.cloud.tencent.com/v1
      kind: LogConfig
      metadata:
        name: test
      spec:
        clsDetail:
          .......
          topicId: xxxx-xx-xx-xx-xxxx
        inputDetail:
          containerFile:
            container: prod
            filePattern: '*.log'
            logPath: /tmp/logs
            namespace: kube-system
            workload:
              kind: deployment
              name: sample-app
          type: container_file
      apiVersion: cls.cloud.tencent.com/v1
      kind: LogConfig
      metadata:
        name: test
      spec:
        clsDetail:
          .......
          topicId: xxxx-xx-xx-xx-xxxx
        inputDetail:
          containerFile:
            container: prod
            filePattern: '*.log'
            includeLabels:
              key1: v1
            excludeLabels:
              key2: v2
            logPath: /tmp/logs
            namespace: default,kube-public
          type: container_file

      Sample CRD for the configuration of the node file path

      apiVersion: cls.cloud.tencent.com/v1
      kind: LogConfig
      metadata:
        creationTimestamp: "2022-03-13T12:48:49Z"
        generation: 4
        name: test
        resourceVersion: "11729531"
        selfLink: /apis/cls.cloud.tencent.com/v1/logconfigs/test
        uid: 233f4b72-cfef-4a43-abb8-e4d033185097
      spec:
        clsDetail:
          .......
          topicId: xxxx-xx-xx-xx-xxxx
        inputDetail:
          hostFile:
            customLabels:
              testmetadata: v1
            filePattern: '*.log'
            logPath: /var/logs
          type: host_file
      
      Contact Us

      Contact our sales team or business advisors to help your business.

      Contact Us
      Technical Support

      Open a ticket if you're looking for further assistance. Our Ticket is 7x24 avaliable.

      Submit a Ticket
      7x24 Phone Support
      Copyright © 2013-2024 Tencent Cloud. All Rights Reserved.
      Privacy PolicyLegalCookie Policy