This document describes how to host a Spring Cloud application to TKE.
Hosting Spring Cloud applications to TKE has the following advantages:
Improve the resource utilization.
Kubernetes is a natural fit for microservice architectures.
Improve the Ops efficiency and facilitate DevOps implementation.
Highly scalable Kubernetes makes it easy to dynamically scale applications.
TKE provides Kubernetes master management to ease Kubernetes cluster Ops and management.
TKE is integrated with other cloud-native products of Tencent Cloud to help you better use Tencent Cloud products.
Best Practices
PiggyMetrics overview
This document describes how to host a Spring Cloud application to TKE by forking the open-source PiggyMetrics on GitHub and adapting it to Tencent Cloud products.
Note:
The modified PiggyMetrics deployment project is hosted on GitHub. After creating the basic service cluster, you can download the deployment project and deploy it in TKE.
The PiggyMetrics homepage is as shown below:
PiggyMetrics is a microservice-architecture application for personal finances developed by using the Spring Cloud framework.
PiggyMetrics consists of the following microservices:
Microservice
Description
API gateway
It's a Spring Cloud Zuul-based gateway and the aggregated portal for calling backend APIs, providing reverse routing and load balancing (Eureka + Ribbon) as well as rate limiting (Hystrix). Client single-page applications and the Zuul gateway are deployed together to simplify deployment.
Service registration and discovery
A Spring Cloud Eureka registry. Business services are registered through Eureka when they are enabled, and service discovery is performed through Eureka when services are called.
Authorization and authentication service
An authorization and authentication center based on Spring Security OAuth2. The client gets the access token through the Auth Service during logins, and so does service call. Each resource server verifies the token through the Auth Service.
Configuration service
A configuration center based on Spring Cloud Config to centrally manage configuration files for all Spring services.
Soft loading and rate limiting
Ribbon and Hystrix based on Spring Cloud. Zuul calls backend services through Ribbon for soft loading and Hystrix for rate limiting.
Metrics and dashboard
Hystrix Dashboard based on Spring Cloud Turbine, aggregating all the PiggyMetrics streams generated by Hystrix and displaying them on the Hystrix Dashboard.
PiggyMetrics deployment architecture and add-ons
In the following best practice, applications deployed in CVM are containerized and hosted to TKE. In this use case, one VPC is used and divided into two subnets:
Subnet-Basic is deployed with stateful basic services, including Dubbo's service registry Nacos, MySQL, and Redis.
Subnet-K8S is deployed with PiggyMetrics application services, all of which are containerized and run in TKE.
The VPC is divided as shown below:
The network planning for the PiggyMetrics instance is as shown below:
Network Planning
Description
Region/AZ
Nanjing/Nanjing Zone 1
VPC
CIDR: 10.0.0.0/16
Subnet-Basic
Nanjing Zone 1, CIDR block: 10.0.1.0/24
Subnet-K8S
Nanjing Zone 1, CIDR block: 10.0.2.0/24
Nacos cluster
Nacos cluster built with three 1-core 2 GB MEM Standard SA2 CVM instances with IP addresses of 10.0.1.9, 10.0.1.14, and 10.0.1.15
The add-ons used in the PiggyMetrics instance are as shown below:
Add-on
Version
Source
Remarks
K8S
1.8.4
Tencent Cloud
TKE management mode
MongoDB
4.0
Tencent Cloud
TencentDB for MongoDB WiredTiger engine
CLS
N/A
Tencent Cloud
Log service
TSW
N/A
Tencent Cloud
Accessed with SkyWalking 8.4.0 Agent, which can be downloaded here
Tencent Cloud Tencent Container Registry (TCR) are available in Personal Edition and Enterprise Edition as differentiated below:
TCR Personal Edition is only deployed in Guangzhou, while TCR Enterprise Edition is deployed in every region.
TCR Personal Edition doesn't offer SLA guarantee.
PiggyMetrics is a Dubbo containerized demo project, so TCR Personal Edition perfectly meets its needs. However, for enterprise users, TCR Enterprise Edition is recommended. To use an image repository, see Basic Image Repository Operations.
TSW
Tencent Service Watcher (TSW) provides cloud-native service observability solutions that can trace upstream and downstream dependencies in distributed architectures, draw topologies, and provide multidimensional call observation by service, API, instance, and middleware.
TSW is architecturally divided into four modules:
Data collection (client)
You can use an open-source probe or SDK to collect data. If you are migrating to the cloud, you can change the reporting address and authentication information only and keep most of the configurations on the client.
Data processing (server)
Data is reported to the server via the Pulsar message queue, converted by the adapter into an OpenTracing-compatible format, and assigned to real-time and offline computing as needed.
- Real-time computing provides real-time monitoring, statistical data display, and fast response to the connected alarming platform. - Offline computing aggregates the statistical data in large amounts over long periods of time and leverages big data analytics to provide business value.
Storage
The storage layer can adapt to use cases with different data types, writing at the server layer, and query and reading requests at the data usage layer.
Data usage
The data usage layer provides underlying support for console operations, data display, and alarming.
A guest user of the piggymetrics library is created in the MongoDB initialization script mongo-init.js by default, which can be modified as needed.
In the CLB console, create a private network CLB instance for Subnet-K8S (the ID of this CLB instance will be used later).
TSW is currently in beta test and supports both Java and Go.
Building Docker image
Writing Dockerfile
The following uses account-service as an example to describe how to write a Dockerfile. The project directory structure of account-service is displayed, Dockerfile is in the root directory of the project, and account-service.jar is the packaged file that needs to be added to the image.
➜ account-service tree
├── Dockerfile
├── skywalking
│ ├── account.config
│ └── skywalking-agent.zip
├── pom.xml
├── src
│ ....
├── target
│ .....
│ └── account-service.jar
└── account-service.iml
Note:
Here, SkyWalking Agent is used as the TSW access client that reports call chain information to the TSW backend. For more information on how to download SkyWalking Agent, see PiggyMetrics deployment architecture and add-ons.
The Dockerfile of account-service is as shown below:
FROM java:8-jre
# Working directory in the container
/appWORKDIR /app
# Add the locally packaged application to the image.
ADD ./target/account-service.jar
# Copy SkyWalking Agent to the image.
COPY ./skywalking/skywalking-agent.zip
# Decompress SkyWalking Agent and delete the original compressed file.
RUN unzip skywalking-agent.zip &&rm -f skywalking-agent.zip
As each Run command in the Dockerfile will generate an image layer, we recommend you combine these commands into one.
Image build
TCR provides both automatic and manual methods to build an image. To demonstrate the build process, the manual method is used.
The image name needs to be in line with the convention of ccr.ccs.tencentyun.com/[namespace]/[ImageName]:[image tag]:
Here, namespace can be the project name to facilitate image management and use. In this document, piggymetrics represents all the images under the PiggyMetrics project.
ImageName can contain the subpath, generally used for multi-project use cases of enterprise users. In addition, if a local image is already built, you can run the docker tag command to rename the image in line with the naming convention.
1. Run the following command to build an image as shown below:
# Recommended build method, which eliminates the need for secondary tagging operations
2. Click TCR > Personal > Namespace to enter the Namespace page.
3. Click Create and create the piggymetrics namespace in the pop-up window. All the images of the PiggyMetrics project are stored under this namespace as shown below:
You can get your Tencent Cloud account ID on the Account Info page.
If you forget your TCR login password, you can reset it in My Images of TCR Personal Edition.
If you are prompted that you have no permission to run the command, add sudo before the command and run it as shown below. In this case, you need to enter two passwords, the server admin password required for sudo and the TCR login password.
When a cluster is created, we recommend you enable Placement Group on the Select Model page. It helps distribute CVM instances across different hosts to increase the system reliability.
2. After the cluster is created, you can view its information on the Cluster Management page in the TKE console. Here, the new cluster is named piggyMetrics.
3. Click the PiggyMetrics-k8s-demo cluster to enter the Basic Info page to view the cluster configuration information.
4. (Optional) If you want to use K8s management tools such as kubectl and Lens, you need to follow two steps:
4.1 Enable public network access.
4.2 Store the API authentication token in the local config file under user home/.kube (choose another if the config file has content) to ensure that the default cluster can be accessed each time. If you choose not to store the token in the config file under .kube, see the Instructions on Connecting to Kubernetes Cluster via kubectl under Cluster API Server Info in the console as shown below:
Creating namespace
A namespace is a logical environment in a Kubernetes cluster, allowing you to divide teams or projects. You can create a namespace in the following three methods, and method 1 is recommended.
Method 1. Use the command line
Method 2. Use the console
Method 3. Use YAML
Run the following command to create a namespace:
kubectl create namespace piggymetrics
1. Log in to the TKE console and click the Cluster ID/Name to enter the cluster details page.
2. Click Namespace > Create to create a namespace named PiggyMetrics.
Run the following command to create a namespace with YAML:
kubctl create –f namespace.yaml
Here, namespace.yaml is as shown below:
# Create the `piggymetrics` namespace.
apiVersion: v1
kind: Namespace
metadata:
name: piggymetrics
spec:
finalizers:
- kubernetes
Using ConfigMap to store configuration information
ConfigMap allows you to decouple the configuration from the running image, making the application more portable. The PiggyMetrics backend service needs to get the MongoDB host and port information from the environment variables and store them by using the ConfigMap.You can use ConfigMap to store configuration information in the following two methods:
Method 1. Use YAML
Method 2. Use the console
The following is the ConfigMap YAML for PiggyMetrics, where values of pure digits require double quotation marks.
1. Log in to the TKE console and click the Cluster ID/Name to enter the cluster details page.
2. Click Configuration Management > ConfigMap > Create to create a ConfigMap named piggymetrics-env for storing the configuration. The piggymetrics namespace is as shown below:
Using Secret to store sensitive information
A Secret can be used to store sensitive information such as passwords, tokens, and keys to reduce exposure risks. PiggyMetrics uses it to store account and password information.You can use a Secret to store sensitive information in the following two methods:
Method 1. Use YAML
Method 2. Use the console
The following is the YAML for creating a Secret in PiggyMetrics, where the value of the Secret needs to be a Base64-encoded string.
# Create a Secret.
apiVersion: v1
kind: Secret
metadata:
name: piggymetrics-keys
namespace: piggymetrics
labels:
qcloud-app: piggymetrics-keys
data:
# Replace `XXX` below with the actual value.
MONGODB_USER: XXX
MONGODB_PASSWORD: XXX
SW_AGENT_AUTHENTICATION: XXX
type: Opaque
1. Log in to the TKE console and click the Cluster ID/Name to enter the cluster details page.
2. Click Configuration Management > Secret > Create to create a Secret named piggymetrics-keys as shown below:
Deploying stateful service with StatefulSet
A StatefulSet is used to manage stateful applications. A Pod created accordingly has a persistent identifier in line with the specifications, which will be retained after the Pod is migrated, terminated, or restarted. When using persistent storage, you can map storage volumes to identifiers.The basic add-ons and services under the PiggyMetrics project such as configuration services, registry, and RabbitMQ have their own data stored and are therefore suitable for deployment through StatefulSet.
Below is a sample deployment YAML for config-server:
A Deployment declares the Pod template and controls the Pod running policy, which is suitable for deploying stateless applications. PiggyMetrics backend services such as Account are stateless and can use the Deployment.
YAML parameters for the account-service Deployment are as follows:
Parameter
Description
replicas
Indicates the number of Pods to be created.
image
Image address
imagePullSecrets
The key to pull an image, which can be obtained from Cluster > Configuration Management > Secret. It is not required for public images.
env
Defines Pod environment variables and values.
The key-value defined in the ConfigMap can be referenced by using configMapKeyRef.
The key-value defined in the Secret can be referenced by using secretKeyRef.
ports
Specifies the port number of the container. It is 6000 for account-service.
Below is a complete sample YAML file for the account-service Deployment:
You can specify the Service type with Kubernetes ServiceType, which defaults to ClusterIP. Valid values of ServiceType include the following:
LoadBalancer: Provides public network, VPC, and private network access.
NodePort: : Accesses services through the CVM IP and host port.
ClusterIP: Accesses services through the service name and port.
The frontend pages and the gateway of PiggyMetrics are packaged together and need to provide services, so ServiceType is set to LoadBalancer. TKE enriches the LoadBalancer mode by configuring the Service through annotations.
If you use the service.kubernetes.io/qcloud-loadbalancer-internal-subnetid annotations, a private network CLB instance will be created when the Service is deployed. In general, we recommend you create the CLB instance in advance and use the service.kubernetes.io/loadbalance-id annotations in the deployment YAML to improve the efficiency.
Below is the deployment YAML for gateway service:
# Deploy `gateway service`.
apiVersion: v1
kind: Service
metadata:
name: gateway
namespace: piggymetrics
annotations:
# Replace it with the ID of the CLB instance of `Subnet-K8S`.
service.kubernetes.io/loadbalance-id: lb-hfyt76co
spec:
externalTrafficPolicy: Cluster
ports:
- name: http
port: 80
targetPort: 4000
protocol: TCP
selector: # Map the backend `gateway` to the Service.
app: gateway
version: v1
type: LoadBalancer
Viewing deployment result
At this point, you have completed the deployment of PiggyMetrics in TKE and can view the deployment result in the following steps:
1. Log in to the TKE console and click the Cluster ID/Name to enter the cluster details page.
2. Click Services and Routes > Service to enter the Service page, where you can see the created Service. You can access the PiggyMetrics page through the gateway service VIP.
Integrating CLS
Enabling container log collection
The container log collection feature is disabled by default and needs to be enabled as instructed below:
1. Log in to the TKE console and click Cluster Ops > Feature Management on the left sidebar.
2. At the top of the Feature Management page, select the region. On the right of the target cluster, click Set.
3. On the Configure Features page, click Edit for log collection, enable log collection, and confirm this operation as shown below:
Creating log topic and logset
CLS is region-specific. To reduce the network latency, we recommend you select a region closest to your business when creating log resources, which are mainly logsets and log topics. A logset represents a project, a log topic represents a class of services, and a single logset can contain multiple log topics.
PiggyMetrics is deployed in Nanjing region, so you need to select Nanjing region on the Log Topic page when creating logsets:
1. Log in to the CLS console and select Nanjing region on the Log Topic page.
2. Click Create Log Topic and enter the relevant information in the pop-up window as prompted as shown below:
Log Topic Name: Enter piggymetrics.
Logset Operation: Select Create Logset.
Logset Name: Enter piggymetrics-logs.
3. Click OK.
Note:
As PiggyMetrics has multiple backend microservices, you can create a log topic for each microservice to facilitate log categorization.
A log topic is created for each PiggyMetrics service.
You need the log topic ID when creating log rules for containers.
Configuring log collection rule
You can configure container log collection rules in the console or with CRD.
Method 1. Use the console
Method 2. Use CRD
Log rules specify the location of a log in a container:
1. Log in to the TKE console and click Cluster Ops > Log Rules on the left sidebar.
2. On the Log Rules page, click Create to create a rule.
Log Source: Specify the location of a log in a container. PiggyMetrics uses the default Spring Cloud configuration where all logs are printed to the standard output. Therefore, you can use the standard container output and specify a Pod Label.
Consumer: Select the previously created logset and topic.
3. Click Next to enter the Log Parsing Method. Here, single-line text is used for PiggyMetrics. For more information on the log formats supported by CLS, see Full Text in a Single Line.
You can also configure log collection rules via Custom Resource Definition (CRD). PiggyMetrics uses a container file path for collection and single-line text. The following is a configuration YAML for account-service log collection. For more information on CRD collection configuration, see Using CRD to Configure Log Collection via YAML.
apiVersion: cls.cloud.tencent.com/v1
kind: LogConfig
metadata:
name: account-log-rule
spec:
clsDetail:
extractRule: {}
# Single-line text
logType: minimalist_log
# Log topic ID
topicId: 8438cc9b-888f-469f-9cff-9891270a0a13
inputDetail:
# Standard container output
containerStdout:
container: account-service
includeLabels:
app: account-service
version: v1
namespace: piggymetrics
type: container_stdout
Viewing log
1. Log in to the CLS console and enter the Search and Analysis page.
2. On the Search and Analysis page, Create Index for the logs first and then click Search and Analysis to view the logs.
Note:
You can't find logs if no indexes are created.
Integrating TSW
TSW is currently in beta test and can be deployed in Guangzhou and Shanghai. Here, deployment in Shanghai is used as an example (PiggyMetrics is deployed in Nanjing).
Accessing TSW - getting access point information
1. Log in to the TSW console and click Service Observation > Service List on the left sidebar.
2. Click Access Service and select Java and the SkyWalking data collection method. The access method provides the Access Point and Token information.
Accessing TSW - application and container configuration
Enter the Access Point and Token of the TSW obtained in the previous step in collector.backend_service and agent.authentication respectively in the agent.config of SkyWalking. agent.service_name is the service name, and agent.namespace can be used to group microservices under the same domain. account-service configuration is as shown below:
You can also configure SkyWalking Agent by using environment variables. PiggyMetrics uses the ConfigMap and Secret to configure environment variables:
Use the ConfigMap to configure SW_AGENT_COLLECTOR_BACKEND_SERVICES.
Use the Secret to configure SW_AGENT_AUTHENTICATION.
As shown below:
At this point, you have completed TSW access. After starting the container service, you can view the call chain, service topology, and SQL analysis in the TSW console.
Using TSW
Viewing call exception through service API or call chain
1. Log in to the TSW console and click Service Observation > API Observation on the left sidebar.
2. On the API Observation page, you can view the call status of all APIs under a service, including the number of requests, success rate, error rate, response time, and other metrics.
The figure shows that the gateway and account-service responded too slowly and all statistic-service requests failed in the past hour.
3. Click the service name statistics-service to enter the information page. Click API Observation, and you can see that the API {PUT}/{accountName} throws a NestedServletException exception, which makes the API unavailable.
4. Click the Trace ID to view the call chain details.
Viewing service topology
1. Log in to the TSW console and click Chain Tracing > Distributed Dependency Topology on the left sidebar.
2. On the Distributed Dependency Topology page, you can view the completed service dependencies as well as information such as the number of calls and average latency.
Yes
No
Was this page helpful?