tencent cloud

Feedback

how to use CAM to authenticate databases for workloads running in TKE

Last updated: 2024-02-05 17:22:47

    Background

    Running containerized workloads in a Tencent Cloud managed cluster involves accessing SQL or NoSQL databases outside the cluster. To use SQL databases together with Kubernetes, we need to consider the issues of secret rotation and sensitive data transfer. With Secrets Manager (SSM) ‍and Cloud Access Management (CAM), we can eliminate risks arising from verifying databases with username and key. In addition, SSM features scheduled secret rotation to reduce labor efforts. This document describes how to use CAM to authenticate databases for workloads running in TKE. In the example below, we create a TencentDB instance and create a secret for it in SSM. Then, enable the resource access control feature of OIDC, make the created CAM OIDC provider as the carrier for role creation, and associate with the policies ‍of accessing TencentDB and SSM. Finally, we securely connect to the TencentDB database through the Kubernetes service account with CAM and SSM. The entire architecture is as follows:
    
    
    

    ‍Limits

    ‍Only TKE managed clusters are supported.
    Supports cluster version ≥ v1.20.6-tke.27/v1.22.5-tke.1

    Directions

    Step 1. Create a managed cluster

    1. Log in to the TKE console to create a cluster.
    Notes
    You can create a managed cluster as instructed in Creating a cluster.
    To use an existing managed cluster, check the cluster version on the details page, and upgrade the version if necessary. See Upgrading a Cluster.
    2. Run the following command to access the managed cluster through the kubectl client.
    kubectl get node
    The following message indicates that the cluster can be accessed.
    kubectl get node
    NAME STATUS ROLES AGE VERSION
    10.0.4.144 Ready <none> 24h v1.22.5-tke.1
    Notes
    You can connect to a TKE cluster from a local client using kubectl, the Kubernetes command line tool. For details, see Connecting to a Cluster.

    Step 2. Enable resource access control of OIDC

    1. On the cluster details page, click
    
    on the right of ServiceAccountIssuerDiscovery.
    
    
    
    2. On the Modify ServiceAccountIssuerDiscovery parameters page, if you are prompted that you do not have permission to modify the parameters, please obtain permission first.
    
    
    View the authorization policy QcloudAccessForTKERoleInOIDCConfig on the role management page, and click Grant.
    
    
    
    3. Select Create CAM OIDC provider and Create WEBHOOK component, and enter the client ID. Then click OK.
    Notes
    Client ID is optional. The default value "sts.cloud.tencent.com" is entered when it is not specified. In this example, we use the default value.
    
    
    
    4. Go back to the cluster details page. When ServiceAccountIssuerDiscovery is available for modification again, the resource access control is enabled successfully.
    Notes
    The values of "service-account-issuer" and "service-account-jwks-uri" are default and cannot be modified.

    Step 3. Check if the CAM OIDC provider and WEBHOOK component are created successfully

    1. Click
    
    on the right of ServiceAccountIssuerDiscovery on the cluster details page.
    2. On the Modify ServiceAccountIssuerDiscovery parameters page, you can see the prompt: "You have created the identity provider. Check details". Click Check details.
    
    
    
    3. Check details of the CAM OIDC provider you created.
    
    
    
    4. Go to Cluster information > Add-on management. If the status of the pod-identity-webhook component is "Succeeded", the component is installed successfully.
    
    
    You can also run the command to check the installation status. If the status of the Pod with the prefix of "pod-identity-webhook" is "Running", the component is installed successfully.
    kubectl get pod -n kube-system
    NAMESPACE NAME READY STATUS RESTARTS AGE
    kube-system pod-identity-webhook-78c76****-9qrpj 1/1 Running 0 43h

    Step 4. Confirm the TencentDB instance

    You need to confirm whether a TencentDB instance exists. If not exists, please create an instance first, and create a database in the instance. Skip the database creation if a TencentDB instance exists. In this example, a TencentDB for MySQL instance is used, and the public network access is enabled for the instance. For details on instance creation, see Creating MySQL Instance.
    
    
    
    Notes
    The value of Public network address is identified as $db_address.
    The value of Port is identified as $db_port.

    Step 5. Update security group of the database

    To allow the Pods in a managed cluster to access the TencentDB for MySQL database, you need to configure the security group rules for the database. Modify the security group rules on the security group management page.
    
    
    
    To create inbound rules for Kubernetes Pods, please click Security group ID to go to the security group instance page. On the security group instance details page, click Security group rules > Inbound rules > Add rule. In the Add inbound rules window, create inbound rules. In this example, the Source is 0.0.0.0/0, and the Protocol port is TCP:3306.
    
    
    

    Step 6. Test connectivity of the database

    In the instance where a MySQL client is installed, ‍connect to the database with the username "root" and the database password you set at the time of creation. If the connection is failed, please go back to check if the public network is enabled and the security group is correctly configured.
    mysql -h $db_address -P $db_port -uroot -p
    Enter password:
    Welcome to the MariaDB monitor. Commands end with ; or \\g.
    Your MySQL connection id is 4238098
    Server version: 5.7.36-txsql-log 20211230
    
    Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
    
    Type 'help;' or '\\h' for help. Type '\\c' to clear the current input statement.
    
    MySQL [(none)]>

    Step 7. Create a database

    To verify the connectivity and operation permissions for the database, please create a database first.
    MySQL [(none)]> CREATE DATABASE mydb;
    Query OK, 1 row affected (0.00 sec)
    
    MySQL [(none)]> CREATE TABLE mydb.user (Id VARCHAR(120), Name VARCHAR(120));
    Query OK, 0 rows affected (0.00 sec)
    
    MySQL [(none)]> INSERT INTO mydb.user (Id,Name) VALUES ('123','tke-oidc');
    Query OK, 1 row affected (0.01 sec)
    
    MySQL [(none)]> SELECT * FROM mydb.user;
    +------+----------+
    | Id | Name |
    +------+----------+
    | 123 | tke-oidc |
    +------+----------+
    1 row in set (0.01 sec)
    After the database is created, you can view it in the console.
    
    
    
    Notes
    The value of the Database name is identified as $db_name.

    Step 8. Create a database secret instance in SSM

    Check whether you have a database secret. If not, please create a database secret, enable secret rotation and select encryption in the SSM console to reduce disclosure risks and security threats to your account. In this example, we will create two database secrets, ‍one of which has the "select" permission to the database. The two secrets are distinguished by the value of Description.
    1. Log in to the SSM console.
    2. On the Create secret page, configure the database account as instructed below. For details, see Creating Database Secret.
    
    
    
    
    Bound instance: You can select an existing database instance or create a new one.
    Server: Client IP. Enter % if you don't want to specify
    Permission configuration: Grant permissions as needed.
    Create the first database secret
    Create the second database secret
    Click Authorization. Select the following permissions on the Permission configuration page.
    
    
    
    Click Authorization. Select All on the Permission configuration page.
    
    
    
    Notes
    The value of Secret name is identified as $ssm_name.
    The value of Secret region is identified as $ssm_region_name.
    3. Click Create. View the created secrets on the secret list page.
    
    
    

    Step 9. Create a CAM role and associate with the access policies

    1. Log in to the CAM console.
    2. On the Role page, click Create role > Identity provider.
    3. On the Create custom role page, complete the configuration with the following information.
    
    
    
    Notes
    The value of odic:aud must be consistent with the value of Client ID of the CAM OIDC provider.
    The value of odic:aud is identified as $my_pod_audience. When multiple values are available to odic:aud, select any one of them.
    
    
    
    Notes
    You can select an existing custom policy or create a new one. In this example, we use QcloudSSMReadOnlyAccess and QcloudCDBReadOnlyAccess
    
    
    
    Notes
    The value of RoleArn is identified as $my_pod_role_arn.

    Step 10. Deploy the sample application

    1. Create a Kubernetes namespace to deploy resources.
    kubectl create namespace my-namespace
    2. Save the following contents to my-serviceaccount.yaml. Replace $my_pod_role_arn with the value of RoleArn, and replace $my_pod_audience with the value of odic:aud.
    apiVersion: v1
    kind: ServiceAccount
    metadata:
    name: my-serviceaccount
    namespace: my-namespace
    annotations:
    tke.cloud.tencent.com/role-arn: $my_pod_role_arn
    tke.cloud.tencent.com/audience: $my_pod_audience
    tke.cloud.tencent.com/token-expiration: "86400"
    3. Save the following contents to sample-application.yaml.
    apiVersion: apps/v1
    kind: Deployment
    metadata:
    name: nginx-deployment
    namespace: my-namespace
    spec:
    selector:
    matchLabels:
    app: my-app
    replicas: 1
    template:
    metadata:
    labels:
    app: my-app
    spec:
    serviceAccountName: my-serviceaccount
    containers:
    - name: nginx
    image: $image
    ports:
    - containerPort: 80
    Note that in this example, ccr.ccs.tencentyun.com/tkeimages/sample-application:latest is selected for $image, which integrates with the compiled demo file. You can enter it as needed.
    4. Deploy the sample application.
    kubectl apply -f my-serviceaccount.yaml
    kubectl apply -f sample-application.yaml
    5. Check the Pod deploying with the sample application.
    kubectl get pods -n my-namespace
    Below is the sample output:
    NAME READY STATUS RESTARTS AGE
    nginx-deployment-6bfd845f47-9zxld 1/1 Running 0 67s
    6. Check workload environment variable information.
    kubectl describe pod nginx-deployment-6bfd845f47-9zxld -n my-namespace
    Below is the sample output:
    

    Step 11. Access the pseudo-code implementation of the database demo

    1. Confirm that the sub-account has permission to access the AssumeRoleWithWebIdentity API. If not, contact the admin to add the permission.
    2. If the sub-account has permission to access the AssumeRoleWithWebIdentity API, obtain the temporary key to access DB + SSM with reference to step 5 in Secret Management.
    3. Clone ssm-rotation-sdk-golang code.
    shell git clone https://github.com/TencentCloud/ssm-rotation-sdk-golang.git
    4. Replace the pseudo-code implementation in the demo:
    package main
    
    import (
    "flag"
    "fmt"
    _ "github.com/go-sql-driver/mysql"
    "github.com/tencentcloud/ssm-rotation-sdk-golang/lib/db"
    "github.com/tencentcloud/ssm-rotation-sdk-golang/lib/ssm"
    "github.com/tencentcloud/tencentcloud-sdk-go/tencentcloud/common"
    "log"
    "time"
    )
    
    var (
    roleArn, tokenPath, providerId, regionName, saToken string
    secretName, dbAddress, dbName, ssmRegionName string
    dbPort uint64
    dbConn *db.DynamicSecretRotationDb
    Header = map[string]string{
    "Authorization": "SKIP",
    "X-TC-Action": "AssumeRoleWithWebIdentity",
    "Host": "sts.internal.tencentcloudapi.com",
    "X-TC-RequestClient": "PHP_SDK",
    "X-TC-Version": "2018-08-13",
    "X-TC-Region": regionName,
    "X-TC-Timestamp": "1659944952",
    "Content-type": "application/json",
    }
    )
    
    type Credentials struct {
    TmpSecretId string
    TmpSecretKey string
    Token string
    ExpiredTime uint64
    }
    
    func main() {
    flag.StringVar(&secretName, "ssmName", "", "ssm name")
    flag.StringVar(&ssmRegionName, "ssmRegionName", "", "ssm region")
    flag.StringVar(&dbAddress, "dbAddress", "", "database address")
    flag.StringVar(&dbName, "dbName", "", "database name")
    flag.Uint64Var(&dbPort, "dbPort", 0, "database port")
    flag.Parse()
    
    provider, err := common.DefaultTkeOIDCRoleArnProvider()
    if err != nil {
    log.Fatal("failed to assume role with web identity, err:", err)
    }
    assumeResp, err := provider.GetCredential()
    if err != nil {
    log.Fatal("failed to assume role with web identity, err:", err)
    }
    
    var credential Credentials
    if assumeResp != nil {
    credential = Credentials{
    TmpSecretId: assumeResp.GetSecretId(),
    TmpSecretKey: assumeResp.GetSecretKey(),
    Token: assumeResp.GetToken(),
    }
    }
    log.Printf("secretId:%v,secretey%v,token%v\\n", credential.TmpSecretId, credential.TmpSecretKey, credential.Token)
    DB(credential)
    }
    
    func DB(credential Credentials) {
    // Initialize the database connection
    dbConn = &db.DynamicSecretRotationDb{}
    err := dbConn.Init(&db.Config{
    DbConfig: &db.DbConfig{
    MaxOpenConns: 100,
    MaxIdleConns: 50,
    IdleTimeoutSeconds: 100,
    ReadTimeoutSeconds: 5,
    WriteTimeoutSeconds: 5,
    SecretName: secretName, // Secret name
    IpAddress: dbAddress, // Database address
    Port: dbPort, // Database port
    DbName: dbName, // Leave it empty or specify a database name
    ParamStr: "charset=utf8&loc=Local",
    },
    SsmServiceConfig: &ssm.SsmAccount{
    ‌SecretId: credential.TmpSecretId, // Fill in the actual available SecretId
    SecretKey: credential.TmpSecretKey, // Fill in the actual available SecretKey
    Token: credential.Token,
    Region: ssmRegionName, // Select the region where the secret is stored
    },
    WatchChangeInterval: time.Second * 10, // Interval to check the secret rotation
    })
    if err != nil {
    fmt.Errorf("failed to init dbConn, err:%v\\n", err)
    return
    }
    // In ‍the simulation process, you need to get a db connection to operate the database at regular intervals (usually in milliseconds)
    t := time.Tick(time.Second)
    for {
    select {
    case <-t:
    accessDb()
    queryDb()
    }
    }
    }
    
    func accessDb() {
    fmt.Println("--- accessDb start")
    c := dbConn.GetConn()
    if err := c.Ping(); err != nil {
    log.Fatal("failed to access db with err:", err)
    }
    log.Println("--- succeed to access db")
    }
    
    func queryDb() {
    var (
    id int
    name string
    )
    log.Println("--- queryDb start")
    c := dbConn.GetConn()
    rows, err := c.Query("select id, name from user where id = ?", 1)
    if err != nil {
    log.Printf("failed to query db with err: ", err)
    log.Fatal(err)
    }
    defer rows.Close()
    for rows.Next() {
    err := rows.Scan(&id, &name)
    if err != nil {
    log.Fatal(err)
    }
    log.Println(id, name)
    }
    err = rows.Err()
    if err != nil {
    log.Fatal(err)
    }
    log.Println("--- succeed to query db")
    }

    Step 12. Test the demo sample

    Go to the nginx container based on the result in the step of Deploy the sample.
    kubectl exec -ti nginx-deployment-6bfd845f47-9zxld -n my-namespace -- /bin/bash
    cd /root/
    Replace the values of $ssm_name and $ssm_region_name according to SSM Instance. Replace the values of $db_address, $db_name and $db_port according to Database Instance.
    ./demo --ssmName=$ssm_name --ssmRegionName=$ssm_region_name --dbAddress=$db_address --dbName=$db_name --dbPort=$db_port
    In this example, when $ssm_name=tke-oidc-1, the "select" permission to database is not available.
    
    In this example, when $ssm_name=tke-oidc-2, the "select" permission to database is available.
    

    Test conclusion

    The test shows that the expected effect is achieved. Validating the authentication tokens for workloads ‍in a managed cluster through CAM ensures the security of authentication. With the rotation and encryption of database usernames and passwords by SSM, you don't need to worry about the storage and lifecycle of database secrets, and it is no need to use usernames and passwords when connecting managed clusters to the database.

    pod-identity-webhook Permission Description

    Permission Description

    The permission of this component is the minimal dependency required for the current feature to operate.

    Permission Scenarios

    Feature
    Involved Object
    Involved Operation Permission
    It is required to inquire about the resource status of the specified serviceaccounts on the created pod.
    serviceaccount
    list/watch/get
    When creating components, it is required to inject the client's certificate in the resource of mutatingwebhookconfigurations.
    mutatingwebhookconfigurations
    get/update

    Permission Definition

    rules:
    - apiGroups:
    - ""
    resources:
    - serviceaccounts
    verbs:
    - get
    - watch
    - list
    - apiGroups:
    - ""
    resources:
    - events
    verbs:
    - patch
    - update
    - apiGroups:
    - "admissionregistration.k8s.io"
    resources:
    - "mutatingwebhookconfigurations"
    verbs:
    - get
    
    
    Contact Us

    Contact our sales team or business advisors to help your business.

    Technical Support

    Open a ticket if you're looking for further assistance. Our Ticket is 7x24 avaliable.

    7x24 Phone Support