This document describes how to use the environment variables to configure the log collection feature for a TKE Serverless cluster, and to send the logs of services within the cluster to Cloud Log Service (CLS) or external Kafka. This feature is suitable for users who need to store and analyze service logs in TKE Serverless clusters.
To use the log collection feature in a TKE Serverless cluster, you need to manually enable it for the cluster when creating a workload. You can enable this feature by performing the following operations:
After the log collection feature is enabled for the TKE Serverless cluster, the log collection agent will send the collected logs in JSON format to the specified consumer end based on your configuration. The details of the collection path and consumer end are as follows:
Consumer end: The log collection service allows you to set Kafka or CLS as the log consumer end.
Collection path: The path of the logs to collect. The collection path supports collection standard output (stdout) and absolute path. It also supports wildcard (*). If multiple paths are specified, separate them with “,”.
Prerequisite
Confirm that the Kubernetes cluster can access the log consumer end.
The length of a log is limited to 2 M. The log is truncated if this limit is exceeded.
When the log collection is enabled for the TKE Serverless cluster, the log information is collected and output to the specified consumer end in JSON format with Kubernetes metadata attached, including the label of the Pod to which the container belongs, annotation, etc. The specific directions are as follows:
1. Log in to the TKE console and click Cluster in the left sidebar.
2. On the cluster management page, click the ID of the target Serverless cluster to enter the cluster details page.
3. Select a workload type in Workload on the left to go to the corresponding page, and then click Create.
4. In Containers in the Pod section, select Advanced settings, and check Activate to enable log collection.
5. Refer to the following information to configure the log consumer end. You can choose CLS or Kafka as the log consume end.
Configuring CLS as the log consumer end
Configuring Kafka as the log consumer end
1. Select CLS as the Consumer end, and select the Logset and Log topic.
CLS currently only supports log collection and reporting for intra-region container clusters.
2. Enable the log index for the selected log topic. Index configuration is required for CLS log search and analysis. If it is not enabled, you cannot view the logs. For how to configure the index, see Configuring Index.
You can go to the CLS console > Log topic page, select a log topic name, and enable the index in the Index configuration page.
If you select Kafka as the consumer end, it is recommended that you use CKafka. The experience of its consumption and production modes are the same as the native version, and it supports alarm configurations.
Specify the Broker address and Topic of Kafka in the container configuration, and ensure that all resources in the cluster can access the Kafka Topic specified by the user.
Note:
You need to select “delete” for cleanup.policy in Kafka Topic configuration. If you select “compact”, CLS will fail to report to Kafka, resulting in data loss.
6. Select Role or Key to authorize.
Note:
You can only select the same authorization method for the containers in the same Pod. The last modification shall prevail. For example, if you select key authorization for the first container and role authorization for the second container, finally both containers will adopt role authorization.
You can only select the same role to authorize for the containers in the same Pod.
Role authorization
Key authorization
Select a role that can access CLS, as shown in the figure below:
If there is no suitable role, you can create one as follows:
Creating a policy
You need to create a policy before creating a role. This policy determines the permissions your role can have.
1. Log in to the CAM console and select Policies in the left sidebar.
2. On the “Polices” page, click Create custom policy.
3. Select Create by policy generator in “Select policy creation method” pop-up window.
4. In the “Visual Policy Generator”, select Cloud Log Service (cls) for Service, and select Write: pushLog for Action.
5. Click Next to go to the “Associate users/user groups” page.
6. Confirm the policy name and click Done.
Creating a role
After creating a policy, you need to bind this policy to a role, so that the role has permissions corresponding to the policy.
1. Log in to the CAM console, and select Roles in the left sidebar.
2. On the “Roles” page, click Create role.
3. In the “Select role entity” window that appears, select Tencent Cloud Product Service to go to the “Create custom role” page.
4. In the “Enter role entity info” step, select Cloud Virtual Machine (cvm) and click Next.
Note:
You must select **Cloud Virtual Machine (cvm)** as the role entity, otherwise, you cannot complete the authorization process.
5. In the “Configure role policy” step, select Created policy, and click Next.
6. In the “Review” step, enter the role name to review the role information, and then click Done. For more information, see Creating a Role.
Select the “SecretId” and “SecretKey” of your account API key as the variable values to create the cluster Secret.
If there is no suitable Secret, you need to create one. For more information, see Secret Management. You can view the SecretId and SecretKey in API Keys.
Note:
The user corresponding to the API key must have the permission to access the CLS. If there is no API key, you need to create one. For more information, see Access Key.
7. Configure the collection path.
At this point, you have configured the log collection feature. You can set other configurations of the workload as needed.
Configuring log collection via yaml
This document provides three collection methods for your choice: Collecting logs to Kafka, collecting logs to CLS via a secret and collecting logs to CLS via a role.
Note:
If both key and role authorization are configured in yaml, the Pod actually uses role authorization.
Collecting logs to Kafka
Collecting logs to CLS via a secret
Collecting logs to CLS via a role
Enable log collection by adding environment variables.
The consumer end can be kafka or CLS. The key indicates whether log collection is enabled.
EKS_LOGS_LOG_PATHS
Log path. It supports stdout (collection standard output) and absolute path. It also supports wildcard (*). If multiple paths are specified, separate them with “,”.
EKS_LOGS_METADATA_ON
Valid values: `true` or `false`. Default value: `true`.
EKS_LOGS_KAFKA_TOPIC
Log topic
EKS_LOGS_KAFKA_BROKERS
kafka brokers: ip1:port1, ip1:port2, and ip2:port2 formats, separated by “,”. Use this environmental variable for external application. EKS_LOGS_KAFKA_HOST will no longer be visible to external users.
EKS_LOGS_KAFKA_MESSAGE_KEY
Optional. A key can be specified to deliver the log to the specified partition.
If the delivery by key is not enabled, logs will be randomly delivered to different partitions.
If the delivery by key is enabled, logs with the same key will be delivered to the same partition.
Note
: Here the key gets the variable value from the field of the Pod. The above examples all take the Pod name as an example, and it also supports namespace, PodIP, etc. For more information, see
Verify whether the delivery by key is enabled, as shown below:
If it is not enabled, the key is not displayed in the message details, as shown below:
If it is enabled, the key is displayed in the message details, as shown below:
Creating a secret
Note:
The following sample is to manually create a secret through yaml. If you create a secret through the console, you do not need to perform 64 encoding. For more information, see Secret Management.
Run the following command via kubectl to obtain the secretid and secretkey for base64 encoding. Replace secretid and secretkey with the actual secretid and secretkey that you use. For more information, see API Keys.
$ echo -n 'secretid'| base64
c2VjcmV0aWQ=
$ echo -n 'secretkey'| base64
c2VjcmV0a2V5
Manually create a secret via yaml. Specify secretid and secretkey based on the values obtained in the step of Creating a secret.
apiVersion: v1
kind: Secret
metadata:
name: secretidkey
data:
secretid:
secretkey:
Creating a deployment
Enable log collection by adding environment variables.
The consumer end can be kafka or CLS. The key indicates whether log collection is enabled.
EKS_LOGS_LOG_PATHS
Log path. It supports stdout (collection standard output) and absolute path. It also supports wildcard (*). If multiple paths are specified, separate them with “,”.
EKS_LOGS_METADATA_ON
Valid values: `true` or `false`. Default value: `true`.
EKS_LOGS_LOGSET_NAME
CLS logset name
EKS_LOGS_TOPIC_ID
CLS logset topic ID
EKS_LOGS_SECRET_ID
SecretId
EKS_LOGS_SECRET_KEY
SecretKey
Step 1: Creating a role
Creating a policy
You need to create a policy before creating a role. This policy determines the permissions your role can have.
1. Log in to the CAM console and select Policies in the left sidebar.
2. On the “Polices” page, click Create custom policy.
3. Select Create by policy generator in “Select policy creation method” pop-up window.
4. In the “Visual Policy Generator”, select Cloud Log Service (cls) for Service, and select Write: pushLog for Action.
5. Click Next to go to the “Associate users/user groups” page.
6. Confirm the policy name and click Done.
Creating a role
After creating a policy, you need to bind this policy to a role, so that the role has permissions corresponding to the policy.
1. Log in to the CAM console, and select Roles in the left sidebar.
2. On the “Roles” page, click Create role.
3. In the “Select role entity” window that appears, select Tencent Cloud Product Service to go to the “Create custom role” page.
4. In the “Enter role entity info” step, select Cloud Virtual Machine (cvm) and click Next.
Note:
You must select Cloud Virtual Machine (cvm) as the role entity, otherwise, you cannot complete the authorization process.
5. In the “Configure role policy” step, select Created policy, and click Next.
6. In the “Review” step, enter the role name to review the role information, and then click Done. For more information, see Creating a Role.
After creating a role, you need to add annotation, specify the role name, and obtain the permission policy contained in the role in the Pod template.
The consumer end can be kafka or CLS. The key indicates whether log collection is enabled.
EKS_LOGS_LOG_PATHS
Log path. It supports stdout (collection standard output) and absolute path. It also supports wildcard (*). If multiple paths are specified, separate them with “,”.
EKS_LOGS_METADATA_ON
Valid values: true or false. Default value: true.
EKS_LOGS_LOGSET_NAME
CLS logset name
EKS_LOGS_TOPIC_ID
CLS logset topic ID
Updating the log collection
You can update log collection in the console or via yaml. Please refer to the following directions:
Updating log collection in the console
Updating log collection via yaml
1. Log in to the TKE console and click Cluster in the left sidebar.
2. On the cluster management page, click the ID of the target Serverless cluster to enter the cluster details page.
3. Choose Workload on the left, find the row of the desired workload for which you want to update log collection, and click Update Pod configuration > Advanced settings on the right to modify configurations.
4. Click Done to complete the process.
Find the yaml corresponding to the workload for which you want to update log collection. Then modify the corresponding variable values based on the relevant variable name changes of the configuration. You can view the meanings of variable names in Configuring log collection.
FAQs
If you have any questions, please refer to Log Collection FAQs. If your question remains unresolved, please Contact Us.
Yes
No
Was this page helpful?