tencent cloud

All product documents
Tencent Kubernetes Engine
Optimization for High Concurrency Scenarios
Last updated: 2024-08-12 17:48:23
Optimization for High Concurrency Scenarios
Last updated: 2024-08-12 17:48:23

Operation Scenarios

This document introduces how to configure and optimize Nginx Ingress for high concurrency scenarios.

Operation Guide

Increasing CLB Specifications and Bandwidth

High concurrency scenarios require high traffic throughput and forwarding performance of CLB. You can manually create a CLB in the CLB Console, select LCU-supported instance specifications, choose the model as needed, and increase the bandwidth limit (ensure the VPC is consistent with that of the TKE cluster).
After the CLB is created, configure Nginx Ingress to reuse this CLB as the traffic entry. For details, refer to Custom Definition CLB.

Tuning Kernel Parameters and Nginx Configuration

Optimize kernel parameters and Nginx configuration for high concurrency scenarios. values.yaml configuration method:
controller:
extraInitContainers:
- name: sysctl
image: busybox
imagePullPolicy: IfNotPresent
securityContext:
privileged: true
command:
- sh
- -c
- |
sysctl -w net.core.somaxconn=65535 # Increase connection queue to prevent queue overflow.
sysctl -w net.ipv4.ip_local_port_range="1024 65535" # Expand the source port range to prevent port exhaustion.
sysctl -w net.ipv4.tcp_tw_reuse=1 # Enable TIME_WAIT reuse to allow new connections after port exhaustion.
sysctl -w fs.file-max=1048576 # Increase the file handle count to prevent connection overflow from exhausting file handles.
config:
# The number of requests that can be processed by a persistent connection between Nginx and the client is 100 by default. We recommend increasing this number in high-concurrency scenarios, but setting it too high may cause uneven load after Nginx Ingress scale-out.
# Reference: https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#keep-alive-requests
keep-alive-requests: "1000"
# The maximum number of idle persistent connections (not the maximum number of connections) between Nginx and the upstream is 320 by default. We recommend increasing this number in high-concurrency scenarios to prevent the frequent establishment of connections from significantly increasing the number of TIME_WAIT connections.
# Reference: https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#upstream-keepalive-connections
upstream-keepalive-connections: "2000"
# The maximum number of connections that each worker process can open is 16384 by default.
# Reference: https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#max-worker-connections
max-worker-connections: "65536"
Note:

Log Rotation

Nginx Ingress will print logs to the container's standard output by default, which will be managed automatically by the container during running. In high-concurrency scenarios, this may lead to high CPU occupancy.
The solution is to output Nginx Ingress logs to log files and use a sidecar to automatically rotate the log files, preventing the disk space from being filled up with logs.
values.yaml configuration method:
controller:
config:
# Nginx logs are written to log files to avoid high CPU utilization under high concurrency.
access-log-path: /var/log/nginx/nginx_access.log
error-log-path: /var/log/nginx/nginx_error.log
extraVolumes:
- name: log # Log mounting directory of the controller
emptyDir: {}
extraVolumeMounts:
- name: log # Log directory shared by the logrotate and controller
mountPath: /var/log/nginx
extraContainers: # Logrotate sidecar container for log rotation
- name: logrotate
image: imroc/logrotate:latest # https://github.com/imroc/docker-logrotate
imagePullPolicy: IfNotPresent
env:
- name: LOGROTATE_FILE_PATTERN # Pattern of rotated log files, matching the log file path configured in Nginx
value: "/var/log/nginx/nginx_*.log"
- name: LOGROTATE_FILESIZE # Threshold of log file size for rotation
value: "100M"
- name: LOGROTATE_FILENUM # Number of rotations per log file
value: "3"
- name: CRON_EXPR # Crontab expression for periodic logrotate running, which is once every minute
value: "*/1 * * * *"
- name: CROND_LOGLEVEL # Crond log level, ranging from 0 to 8, the smaller the value, the more detailed
value: "8"
volumeMounts:
- name: log
mountPath: /var/log/nginx


Was this page helpful?
You can also Contact Sales or Submit a Ticket for help.
Yes
No

Feedback

Contact Us

Contact our sales team or business advisors to help your business.

Technical Support

Open a ticket if you're looking for further assistance. Our Ticket is 7x24 avaliable.

7x24 Phone Support