tencent cloud

Feedback

Using CRD to Configure Log Collection

Last updated: 2023-05-05 10:38:21

Overview

Besides configuration log collection in the TKE console, you can also configure it by using the Custom Resource Definitions (CRD). CRD supports the collection of container standard outputs, container files, and host files. It also supports multiple log collection formats, and supports shipping logs to different consumers such as CLS and CKafka.

Prerequisites

Activate Log Collection in TKE console.

CRD Overview

Structure overview

apiVersion: cls.cloud.tencent.com/v1
kind: LogConfig ## Default value
metadata:
name: test ## CRD resource name, unique in the cluster
spec:
clsDetail: ## The configuration for shipping to CLS
...
inputDetail: ## Data source configuration for collection
...
kafkaDetail: ## The configuration for shipping to a CKafka or self-built Kafka cluster
...
status: ## CRD resource status
status: ""
code: "" ## The error code returned by the called API
reason: "" ## Error cause

clsDetail description

Note
The topic cannot be modified once it’s specified.
clsDetail:
## If the log topic is created automatically, the names of logset and topic need to be specified at the same time.
logsetName: test ## CLS logset name. Logset for the name will be created automatically if there is not any. If there is the logset, log topic will be created under it.
topicName: test ## CLS log topic name. Log topic for the name will be created automatically if there is not any.

# Select an existing logset and log topic. If the logset is specified but the log topic is not, a log topic will be created automatically.
logsetId: xxxxxx-xx-xx-xx-xxxxxxxx ## The ID of the CLS logset. The logset needs to be created in advance in CLS.
topicId: xxxxxx-xx-xx-xx-xxxxxxxx ## CLS log topic ID. The log topic needs to be created in CLS in advance and should not be occupied by other collection configurations.

logType: json_log ## Log collection format. json_log: json format. delimiter_log: separator-based format. minimalist_log: full text in a single line. multiline_log: full text in multi lines. fullregex_log: full regex format. It defaults to minimalist_log.
logFormat: xxx ## Log formatting method
period: 30 ## Lifecycle in days. Value range: 1–3600. `3640` indicates permanent storage.
partitionCount: ## The number (an integer) of log topic partitions. Default value: `1`. Maximum value: `10`.
tags: ## Tag description list. This parameter is used to bind a tag to a log topic. Up to nine tag key-value pairs are supported, and a resource can be bound to only one tag key.
- key: xxx ## Tag key
value: xxx ## Tag value
autoSplit: false## Whether to enable automatic split (Boolean type). Default value: `true`.
maxSplitPartitions:
storageType: hot. ## Log topic storage class. Valid values: `hot` (STANDARD); `cold` (STANDARD_IA). Default value: `hot`.
excludePaths: ## Collection path blocklist
- type: File ## Type. Valid values: `File`, `Path`.
value: /xx/xx/xx/xx.log ## The value of `type`
indexs: ## You can customize the indexing method and field when creating a topic.
- indexName: ## When a key value or metafield index needs to be configured for a field, the metafield `Key` does not need to be prefixed with `__TAG__.` and is consistent with the one when logs are uploaded. `__TAG__.` will be prefixed automatically for display in the console.
indexType: ## Valid values: `long`, `text`, `double`
tokenizer: ## Field delimiter. Each character represents a delimiter. Only English symbols and \\n\\t\\r are supported. For `long` and `double` fields, leave it empty. For `text` fields, we recommend you use @&?|#()='",;:<>[]{}/ \\n\\t\\r\\ as the delimiter.
sqlFlag: ## Whether the analysis feature is enabled for the field (Boolean)
containZH: ## Whether Chinese characters are contained (Boolean)
region: ap-xxx ## Topic region for cross-region shipping
userDefineRule: xxxxxx ## Custom collection rule, which is a serialized JSON string
extractRule: {} ## Extraction and filter rule. If `ExtractRule` is set, `LogType` must be set.

inputDetail description

inputDetail:
type: container_stdout ## Log collection type, including container_stdout (container standard output), container_file (container file), and host_file (host file)

containerStdout: ## Container standard output
namespace: default ## The Kubernetes namespace of the container to be collected. Separate multiple namespaces by comma, for example, `default,namespace`. If this field is not specified, it indicates all namespaces. Note that this field cannot be specified if `excludeNamespace` is specified.
excludeNamespace: nm1,nm2 ## The Kubernetes namespace of the container to be excluded. Separate multiple namespaces by comma, for example, `nm1,nm2`. If this field is not specified, it indicates all namespaces. Note that this field cannot be specified if `namespace` is specified.
nsLabelSelector: environment in (production),tier in (frontend) ## Filter namespaces by namespace label
allContainers: false ## Whether to collect the standard output of all containers in the specified namespace. Note that if `allContainers=true`, you cannot specify `workload`, `includeLabels`, and `excludeLabels` at the same time.
container: xxx ## Name of the container of which the logs will be collected. If the name is empty, it indicates the log names of all matching containers will be collected.
excludeLabels: ## Pods with the specified labels will be excluded. This field cannot be specified if `workload`, `namespace`, and `excludeNamespace` are specified.
key2: value2 ## Pods with multiple values of the same key can be matched. For example, if you enter `environment = production,qa`, Pods with the `production` or `qa` value of the `environment` key will be excluded. Separate multiple values by comma. If you also specify `includeLabels`, Pods in the intersection will be matched.

includeLabels: ## Pods with the specified labels will be collected. This field cannot be specified if `workload`, `namespace`, and `excludeNamespace` are specified.
key: value1 ## The `metadata` will be carried in the log collected based on the collection rule and reported to the consumer. Pods with multiple values of the same key can be matched. For example, if you enter `environment = production,qa`, Pods with the `production` or `qa` value of the `environment` key will be matched. Separate multiple values by comma. If you also specify `excludeLabels`, Pods in the intersection will be matched.

metadataLabels: ## Specify the Pod labels to be collected as the metadata. If this field is not specified, all Pod labels will be collected as the metadata.
- label1
customLabels: ## Custom metadata
label: l1

workloads:
container: xxx ## Name of the container to collect. If this parameter is not specified, it indicates all containers in the workload Pod will be collected.
kind: deployment ## Workload type. Supported values include deployment, daemonset, statefulset, job, and cronjob.
name: sample-app ## Workload name
namespace: prod ## Workload namespace

containerFile: ## File in the container
namespace: default ## The Kubernetes namespace of the container to be collected. A namespace must be specified.
excludeNamespace: nm1,nm2 ## The Kubernetes namespace of the container to be excluded. Separate multiple namespaces by comma, for example, `nm1,nm2`. If this field is not specified, it indicates all namespaces. Note that this field cannot be specified if `namespace` is specified.
nsLabelSelector: environment in (production),tier in (frontend) ## Filter namespaces by namespace label
container: xxx ## The name of container of which the logs will be collected. The * indicates the log names of all matching containers will be collected.
logPath: /var/logs ## Log folder. Wildcards are not supported.
filePattern: app_*.log ## Log file name. It supports the wildcards [*?]. ***** matches multiple random characters, and **?** matches a single random character.
customLabels: ## Custom metadata
key: value
excludeLabels: ## Pods with the specified labels will be excluded. This field cannot be specified if `workload` is specified.
key2: value2 ## Pods with multiple values of the same key can be matched. For example, if you enter `environment = production,qa`, Pods with the `production` or `qa` value of the `environment` key will be excluded. Separate multiple values by comma. If you also specify `includeLabels`, Pods in the intersection will be matched.

includeLabels: ## Pods with the specified labels will be collected. This field cannot be specified if `workload` is specified.
key: value1 ## The `metadata` will be carried in the log collected based on the collection rule and reported to the consumer. Pods with multiple values of the same key can be matched. For example, if you enter `environment = production,qa`, Pods with the `production` or `qa` value of the `environment` key will be matched. Separate multiple values by comma. If you also specify `excludeLabels`, Pods in the intersection will be matched.
metadataLabels: ## Specify the Pod labels to be collected as the metadata. If this field is not specified, all Pod labels will be collected as the metadata.
- label1 ## pod label
workload:
container: xxx ## Name of the container to collect. If this parameter is not specified, it indicates all containers in the workload Pod will be collected.
name: sample-app ## Workload name

hostFile: ## Node file path
filePattern: '*.log' ## Log file name. It supports the wildcards "*" and "?". "*" matches multiple random characters, and "?" matches a single random character.
logPath: /tmp/logs ## Log file folder. Wildcards are not supported.
customLabels: ## Custom metadata
label1: v1
extractRule description
Name
Type
Required
Description
timeKey
String
No
Time field key name. time_key and time_format must appear in pairs.
timeFormat
String
No
Time field format. For more information, see the output parameters of the time format description of the strftime function in C language.
delimiter
String
No
Delimiter for delimited log, which is valid only if log_type is delimiter_log.
logRegex
String
No
Full log matching rule, which is valid only if log_type is fullregex_log.
beginningRegex
String
No
First-Line matching rule, which is valid only if log_type is multiline_log or fullregex_log.
unMatchUpload
String
No
Whether to upload the logs that failed to be parsed. Valid values: true: yes; false: no.
unMatchedKey
String
No
Key of the failure log
backtracking
String
No
Size of the data to be rewound in incremental collection mode. Value: -1 (collect all) and 0 (collect increment). It default to -1.
keys
Array of String
No
Key name of each extracted field. An empty key indicates to discard the field. This parameter is valid only if log_type is delimiter_log. json_log logs use the key of JSON itself.
filterKeys
Array of String
No
Key of the log to filter. It corresponds to FilterRegex by the index.
filterRegex
Array of String
No
Regex corresponding to the key of the log to filter. It corresponds to FilterKeys by the index.
isGBK
String
No
Whether it’s GBK-encoded. Values: 0 (No), 1 (Yes)
Note: This field may return null, indicating that no valid value was found.
jsonStandard
String
No
Whether it’s standard JSON. Values: 0 (No), 1 (Yes).
Note: This field may return null, indicating that no valid value was found.

kafkaDetail description

kafkaDetail:
brokers: x.x.x.x:p ## (Required) The broker address. Generally, it is domain name:port. If there are more than one address, separate them with ",".
topic: test ##
kafkaType: CKafka ## Kafka type. Valid values: `CKafka` (CKafka); `SelfBuildKafka` (self-built Kafka).
instanceId: xxxx ## The ID of the CKafka instance when `kafkaType` is `CKafka`
logType: minimalist_log ## The type of the parsed Kafka log. Valid values: `minimalist_log` or `""` (full text in a single line); `multiline_log` (full text in multiple lines); `json` (JSON).
timestampFormat: xxx ## The format of timestamp. It defaults to `double`.
timestampKey: xxx ## The key of timestamp. It defaults to `@timestamp`.
metadata:
formatType: default ## Metadata format. Valid values: `default` (default, the same as the EKS Kafka collector); `filebeat` (Filebeat); `fluent-bit` (Fluent Bit).
messageKey: ## Specify a key to ship logs to the specified partition. This parameter is not enabled by default, and logs are shipped randomly. When it is enabled, logs with the same key will be shipped to the same partition. You can select the Pod field as the key. For example, if you enter a Pod name, select Field>metadata.name.
value: Field ## Topic ID, which is required
valueFrom:
fieldRef:
fieldPath: metadata.name ## If the key is `Field`, you can select `metadata.name`, `metadata.namespace`, `spec.nodeName`, and `spec.serviceAccountName`.

status description

status
Description
The status is empty.
Initial status
Synced
Configured successfully
Stale
Configuration failed

Sample CRD

Sample CRD for the configuration of the container standard output

All containers
Specifying a workload
Specifying Pod labels
Specify a namespace
apiVersion: cls.cloud.tencent.com/v1
kind: LogConfig
metadata:
name: "test"
spec:
clsDetail:
.......
topicId: xxxxxx-xx-xx-xx-xxxxxxxx
inputDetail:
containerStdout:
allContainers: true
namespace: default,kube-public
type: container_stdout

Exclude a namespace
apiVersion: cls.cloud.tencent.com/v1
kind: LogConfig
metadata:
name: "test"
spec:
clsDetail:
........
topicId: xxxxxx-xx-xx-xx-xxxxxxxx
inputDetail:
containerStdout:
allContainers: true
excludeNamespace: kube-system,kube-node-lease
type: container_stdout

apiVersion: cls.cloud.tencent.com/v1
kind: LogConfig
metadata:
name: "test"
spec:
clsDetail:
......
topicId: xxxxxx-xx-xx-xx-xxxxxxxx
inputDetail:
containerStdout:
allContainers: false
workloads:
- container: prod
kind: deployment
name: sample-app
namespace: kube-system
type: container_stdout
apiVersion: cls.cloud.tencent.com/v1
kind: LogConfig
metadata:
name: test
spec:
clsDetail:
......
topicId: xxxxxx-xx-xx-xx-xxxxxxxx
inputDetail:
containerStdout:
container: prod
excludeLabels:
key2: v2
includeLabels:
key1: v1
namespace: default,kube-system
type: container_stdout

Sample CRD for the configuration of the container file path

Specifying a workload
Specifying Pod labels
apiVersion: cls.cloud.tencent.com/v1
kind: LogConfig
metadata:
name: test
spec:
clsDetail:
.......
topicId: xxxx-xx-xx-xx-xxxx
inputDetail:
containerFile:
container: prod
filePattern: '*.log'
logPath: /tmp/logs
namespace: kube-system
workload:
kind: deployment
name: sample-app
type: container_file
apiVersion: cls.cloud.tencent.com/v1
kind: LogConfig
metadata:
name: test
spec:
clsDetail:
.......
topicId: xxxx-xx-xx-xx-xxxx
inputDetail:
containerFile:
container: prod
filePattern: '*.log'
includeLabels:
key1: v1
excludeLabels:
key2: v2
logPath: /tmp/logs
namespace: default,kube-public
type: container_file

Sample CRD for the configuration of the node file path

apiVersion: cls.cloud.tencent.com/v1
kind: LogConfig
metadata:
creationTimestamp: "2022-03-13T12:48:49Z"
generation: 4
name: test
resourceVersion: "11729531"
selfLink: /apis/cls.cloud.tencent.com/v1/logconfigs/test
uid: 233f4b72-cfef-4a43-abb8-e4d033185097
spec:
clsDetail:
.......
topicId: xxxx-xx-xx-xx-xxxx
inputDetail:
hostFile:
customLabels:
testmetadata: v1
filePattern: '*.log'
logPath: /var/logs
type: host_file

Contact Us

Contact our sales team or business advisors to help your business.

Technical Support

Open a ticket if you're looking for further assistance. Our Ticket is 7x24 avaliable.

7x24 Phone Support