tencent cloud

Feedback

Optimization for High Concurrency Scenarios

Last updated: 2024-08-12 17:48:23

    Operation Scenarios

    This document introduces how to configure and optimize Nginx Ingress for high concurrency scenarios.

    Operation Guide

    Increasing CLB Specifications and Bandwidth

    High concurrency scenarios require high traffic throughput and forwarding performance of CLB. You can manually create a CLB in the CLB Console, select LCU-supported instance specifications, choose the model as needed, and increase the bandwidth limit (ensure the VPC is consistent with that of the TKE cluster).
    After the CLB is created, configure Nginx Ingress to reuse this CLB as the traffic entry. For details, refer to Custom Definition CLB.

    Tuning Kernel Parameters and Nginx Configuration

    Optimize kernel parameters and Nginx configuration for high concurrency scenarios. values.yaml configuration method:
    controller:
    extraInitContainers:
    - name: sysctl
    image: busybox
    imagePullPolicy: IfNotPresent
    securityContext:
    privileged: true
    command:
    - sh
    - -c
    - |
    sysctl -w net.core.somaxconn=65535 # Increase connection queue to prevent queue overflow.
    sysctl -w net.ipv4.ip_local_port_range="1024 65535" # Expand the source port range to prevent port exhaustion.
    sysctl -w net.ipv4.tcp_tw_reuse=1 # Enable TIME_WAIT reuse to allow new connections after port exhaustion.
    sysctl -w fs.file-max=1048576 # Increase the file handle count to prevent connection overflow from exhausting file handles.
    config:
    # The number of requests that can be processed by a persistent connection between Nginx and the client is 100 by default. We recommend increasing this number in high-concurrency scenarios, but setting it too high may cause uneven load after Nginx Ingress scale-out.
    # Reference: https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#keep-alive-requests
    keep-alive-requests: "1000"
    # The maximum number of idle persistent connections (not the maximum number of connections) between Nginx and the upstream is 320 by default. We recommend increasing this number in high-concurrency scenarios to prevent the frequent establishment of connections from significantly increasing the number of TIME_WAIT connections.
    # Reference: https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#upstream-keepalive-connections
    upstream-keepalive-connections: "2000"
    # The maximum number of connections that each worker process can open is 16384 by default.
    # Reference: https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#max-worker-connections
    max-worker-connections: "65536"
    Note:

    Log Rotation

    Nginx Ingress will print logs to the container's standard output by default, which will be managed automatically by the container during running. In high-concurrency scenarios, this may lead to high CPU occupancy.
    The solution is to output Nginx Ingress logs to log files and use a sidecar to automatically rotate the log files, preventing the disk space from being filled up with logs.
    values.yaml configuration method:
    controller:
    config:
    # Nginx logs are written to log files to avoid high CPU utilization under high concurrency.
    access-log-path: /var/log/nginx/nginx_access.log
    error-log-path: /var/log/nginx/nginx_error.log
    extraVolumes:
    - name: log # Log mounting directory of the controller
    emptyDir: {}
    extraVolumeMounts:
    - name: log # Log directory shared by the logrotate and controller
    mountPath: /var/log/nginx
    extraContainers: # Logrotate sidecar container for log rotation
    - name: logrotate
    image: imroc/logrotate:latest # https://github.com/imroc/docker-logrotate
    imagePullPolicy: IfNotPresent
    env:
    - name: LOGROTATE_FILE_PATTERN # Pattern of rotated log files, matching the log file path configured in Nginx
    value: "/var/log/nginx/nginx_*.log"
    - name: LOGROTATE_FILESIZE # Threshold of log file size for rotation
    value: "100M"
    - name: LOGROTATE_FILENUM # Number of rotations per log file
    value: "3"
    - name: CRON_EXPR # Crontab expression for periodic logrotate running, which is once every minute
    value: "*/1 * * * *"
    - name: CROND_LOGLEVEL # Crond log level, ranging from 0 to 8, the smaller the value, the more detailed
    value: "8"
    volumeMounts:
    - name: log
    mountPath: /var/log/nginx
    
    
    Contact Us

    Contact our sales team or business advisors to help your business.

    Technical Support

    Open a ticket if you're looking for further assistance. Our Ticket is 7x24 avaliable.

    7x24 Phone Support