Tencent Collective Communication Library (TCCL) is a high-performance customized acceleration communication library designed for Tencent Cloud's StarPulse network architecture. The main feature leverages the StarPulse network hardware architecture to provide more efficient network communication performance for large-scale AI model training, along with intelligent operations and maintenance capabilities for rapid perception and self-healing of network failures. TCCL has been expanded and optimized based on open source NCCL code, ensuring full compatibility with NCCL's features and usage methods. TCCL currently supports main features including:
Dynamic aggregation optimization of dual network interfaces maximizes the performance of bonding devices.
Global Hash Routing enables load balancing to avoid congestion.
This document describes how to configure the TCCL acceleration communication library in the Tencent Cloud environment to improve multi-machine multi-card communication performance in the Tencent Cloud RDMA environment. In large-scale model training scenarios, TCCL is expected to increase the bandwidth utilization rate by approximately 50% compared with the open source NCCL scheme.
Directions
Environment Preparations
1. Create GPU Hyper Computing ClusterPNV4sne or GPU Hyper Computing ClusterPNV4sn Hyper Computing Cluster instances, which support 1.6 Tbps and 800 Gbps RDMA networks respectively.
TCCL operating software environment requires glibc version 2.17 or later and CUDA version 10.0 or later.
Selecting Installation Methods
TCCL currently supports three installation methods. You can select the installation method suitable for business scenarios as required.
TCCL communication library + compile and install pytorch
TCCL communication library + pytorch communication plugins
NCCL plugins + sorted IP list
Note:
Since most large-scale model training is based on the Pytorch framework, we will take Pytorch as an example.
The comparison of three access schemes for TCCL is shown in the following table:
Installation Methods
Method 1: compile and install Pytorch.
Method 2: install Pytorch communication plugins.
(Recommended) Method 3: install NCCL communication plugins.
Usage Steps
Install TCCL.
Recompile and install Pytorch.
Install Pytorch communication plugins.
Modify the distributed communication backend.
Install NCCL plugins.
Modify the startup script.
Advantage
No intrusion into business code.
Easy to install.
Easy to install.
Disadvantage
Require to recompile and install Pytorch.
Have requirements for the software environment.
Require to modify the business code.
Have requirements for the software environment.
Require to update the sorted list after scaling out cluster nodes.
Software Dependency on the Environment
Corresponding to NCCL version 2.12.
Require glibc version 2.17 or later.
Require CUDA version 10.0 or later.
Current installation package only supports Pytorch 1.12
Require glibc version 2.17 or later.
Require CUDA version 10.0 or later.
Install NCCL.
If your machine resources and model training scenarios are relatively fixed, it is recommended to use Method 3, which is compatible with different NCCL and CUDA versions, and easy to install and use without modifying business code or recompiling pytorch.
If your resources need to be provided to different business teams or frequently require scale-out, it is recommended to use the first two methods, which do not require algorithm personnel or scheduling frameworks to deliberately perceive the network topology information of machines.
If you do not want to adapt the business code, you can use Method 1, which only requires recompiling the pytorch framework.
Configuring the TCCL Environment and Verifying
Method 1: compile and install Pytorch.
Method 2: install Pytorch communication plugins.
(Recommended) Method 3: install NCCL plugins
As the community pytorch connects to the NCCL communication library statically by default, TCCL cannot be used by replacing the shared libraries.
1. Install TCCL.
Take Ubuntu 20.04 as an example, you can use the following commands to install. After installation, TCCL will be located in the /opt/tencent/tccl directory.
# Uninstall the existing tccl versions and nccl plugins.
You need to enable network topology awareness through TCCL_TOPO_AFFINITY=4.
4. Verify Pytorch.
During the process of running single-machine multi-GPU or multi-machine multi-GPU training, the following log will be printed (export NCCL_DEBUG=INFO):
5. Verify nccl-tests.
Before running nccl-tests, you need to export the corresponding TCCL path:
At present, TCCL corresponds to NCCL version 2.12, requiring glibc version 2.17 or later and CUDA version 10.0 or later. For other supported CUDA version, please contact your pre-sales manager for support.
Pytorch supports integrating third-party communication backends through plugins, so users can use TCCL communication backends without recompiling Pytorch. The API is fully compatible with NCCL. See Introduction to Existing Communication Backends in Pytorch for details.
You need to enable the network topology awareness feature through TCCL_TOPO_AFFINITY=4.
4. Verify Pytorch.
When executing distributed training, the following prompts indicate that the communication backend has been loaded correctly.
5. Software version limit
The current installation package only supports Pytorch 1.12. For other supported Pytorch and CUDA versions, please contact your pre-sales manager for support.
Note:
If running nccl-tests or other scenarios that require dynamically linked communication libraries, use Method 1 to install TCCL.
If you have installed NCCL, you can also use the TCCL acceleration capability through the NCCL plugins.
1. Install NCCL plugins.
Take Ubuntu 20.04 as an example, you can use the following commands to install plugins.
# Please ensure that the version of nccl plugins used within the cluster is consistent. The following are the download and installation commands for nccl 1.0 version. It is recommended to use the more stable nccl 1.2 version.
# Ensure that the version of nccl plugins used within the cluster is consistent. The following is the download and installation commands for nccl 1.0 version. It is recommended to use the more stable nccl 1.2 version.
The NCCL plugins do not require dependency files to provide two optimizations: dynamic aggregation on bonding interfaces and global hash routing. If affinity awareness of the network topology is needed, users can achieve it through the sorted IP list.
IP sorting can be completed as follows:
Prepare the IP list file.
The VPC IP address can be obtained through ifconfig eth0, and each row has one node IP. The format is as follows:
# After the machine IP is manually sorted, there is no need to add the following variables.
# export TCCL_TOPO_AFFINITY=4
4. Modify the startup script.
You need to modify the startup script when starting distributed training. For example, if you use the deepspeed launcher to start the training process, you need to obtain the sorted IP list, write the corresponding IP list into the hostfile, and then start the training process.
Yes
No
Was this page helpful?