tencent cloud

This document describes how to ship audit logs of a registered cluster to CLS.

Overview

Cluster audit is a feature based on Kubernetes Audit that can store and search the records of JSON logs with configurable policies generated by kube-apiserver. This feature records the access events of kube-apiserver and records the activities of each user, admin, or system component that has an impact on the cluster in sequence.

Notes

• You have created a registered cluster, and it is in Running status.
• Currently, audit logs of a registered cluster can be shipped to only CLS but not other log consumers.
• To enable the auditing feature of a registered cluster, you need to log in to all master nodes of the cluster to configure relevant audit policies and API server parameters.
• If the cluster auditing feature is enabled, cluster log collection will also be enabled automatically at the same time by default.
• To use the cluster auditing feature, check whether nodes in the Kubernetes cluster can access the log consumer. Here, logs can be shipped over the public and private networks. You can select one option based on your business needs.
  • Shipping over public network: Cluster auditing logs will be shipped to CLS over the public network. This requires that the cluster nodes can access the public network.
  • Shipping over private network: Cluster auditing logs will be shipped to CLS over the private network. This requires that the cluster nodes are interconnected with CLS over the private network. Before choosing this option, submit a ticket for confirmation.

Directions

Configuring audit policies on master nodes in the cluster

Log in to all master nodes in the cluster one by one and configure the audit policy file /etc/kubernetes/audit-policy.yaml based on your actual business conditions.

apiVersion: audit.k8s.io/v1beta1
kind: Policy
omitStages:
 - "RequestReceived"
rules:
 - level: None
   users: ["system:kube-proxy"]
   verbs: ["watch"]
   resources:
     - group: ""
       resources: ["endpoints", "services"]
 - level: None
   users: ["system:unsecured"]
   namespaces: ["kube-system"]
   verbs: ["get"]
   resources:
     - group: ""
       resources: ["configmaps"]
 - level: None
   users: ["kubelet"]
   verbs: ["get"]
   resources:
     - group: ""
       resources: ["nodes"]
 - level: None
   userGroups: ["system:nodes"]
   verbs: ["get"]
   resources:
     - group: ""
       resources: ["nodes"]
 - level: None
   users:
     - system:kube-controller-manager
     - system:kube-scheduler
     - system:serviceaccount:kube-system:endpoint-controller
   verbs: ["get", "update"]
   namespaces: ["kube-system"]
   resources:
     - group: ""
       resources: ["endpoints"]
 - level: None
   users: ["system:apiserver"]
   verbs: ["get"]
   resources:
     - group: ""
       resources: ["namespaces"]
 - level: None
   nonResourceURLs:
     - /healthz*
     - /version
     - /swagger*
 - level: None
   resources:
     - group: ""
       resources: ["events"]
 - level: Metadata
   resources:
     - group: "" # core
       resources: ["secrets", "configmaps"]
     - group: authentication.k8s.io
       resources: ["tokenreviews"]
 - level: Request
   verbs: ["get", "list", "watch"]
   resources:
     - group: ""
     - group: "admissionregistration.k8s.io"
     - group: "apps"
     - group: "authentication.k8s.io"
     - group: "authorization.k8s.io"
     - group: "autoscaling"
     - group: "batch"
     - group: "certificates.k8s.io"
     - group: "extensions"
     - group: "networking.k8s.io"
     - group: "policy"
     - group: "rbac.authorization.k8s.io"
     - group: "settings.k8s.io"
     - group: "storage.k8s.io"
 - level: RequestResponse
   resources:
     - group: ""
     - group: "admissionregistration.k8s.io"
     - group: "apps"
     - group: "authentication.k8s.io"
     - group: "authorization.k8s.io"
     - group: "autoscaling"
     - group: "batch"
     - group: "certificates.k8s.io"
     - group: "extensions"
     - group: "networking.k8s.io"
     - group: "policy"
     - group: "rbac.authorization.k8s.io"
     - group: "settings.k8s.io"
     - group: "storage.k8s.io"
 - level: Metadata

Configuring API server parameters on master nodes

Log in to all master nodes in the cluster one by one and modify the /etc/kubernetes/manifests/kube-apiserver.yaml file.

1. Add the following command parameters:

spec:
containers:
- command:
  - kube-apiserver
  - --audit-log-maxbackup=10
  - --audit-log-maxsize=100
  - --audit-log-path=/var/log/kubernetes/kubernetes.audit
  - --audit-log-maxage=30
  - --audit-policy-file=/etc/kubernetes/audit-policy.yaml

2. Add volume parameters to mount /etc/kubernetes/audit-policy.yaml to the API server Pod.

spec:
containers:
- command:
  - kube-apiserver
  - --audit-log-maxbackup=10
  - --audit-log-maxsize=100
  - --audit-log-path=/var/log/kubernetes/kubernetes.audit
  - --audit-log-maxage=30
  - --audit-policy-file=/etc/kubernetes/audit-policy.yaml
  ...
  ...    
  volumeMounts:
  - mountPath: /var/log/kubernetes
    name: k8s-audit
  - mountPath: /etc/kubernetes/audit-policy.yaml
    name: audit-policy
    readOnly: true
  ...
  ...
volumes:
- hostPath:
    path: /var/log/kubernetes
    type: DirectoryOrCreate
  name: k8s-audit
- hostPath:
    path: /etc/kubernetes/audit-policy.yaml
    type: FileOrCreate
  name: audit-policy
...
 

Enabling cluster auditing

1. Log in to the TKE console and select Ops Feature Management on the left sidebar.
2. At the top of the Feature Management page, select the Region and Registered Cluster. Then, click Set on the right of the target cluster.
   
3. In the Configure features pop-up window, click Edit on the right of the Cluster Auditing feature.
   
4. Select Enable Cluster Auditing and select the shipping method and the logset and log topic for audit log storage. We recommend you select Auto-create Log Topic.
   
5. Click Confirm.

Audit Dashboard

TKE provides out-of-the-box audit dashboards and can automatically configure dashboards of audit overview, node operation overview, K8s object operation overview, and aggregated search for the clusters with cluster auditing enabled. With user-defined filters and built-in CLS global search, TKE makes it convenient for you to observe and search for cluster operations, so as to promptly find and locate problems. For more information, see Auditing Dashboard.