CN105550085A - RDMA (remote direct memory Access) testing method based on GPUDerict - Google Patents

Abstract

The invention provides a GPUDerict-based? RDMA test methods, including HCA card, GPU card, Nvidia? Driver, Nvidia? CUDA? toolkit, MLNX _ offset driver, and nv _ peer _ mem packets for GPU to IB card communication. Directly accessing GPU memory, avoiding access to fixed (pined)? Unnecessary system memory copy and CPU overhead when CUDA host memory speed up the communication with the network and storage device to enable one GPU in the same system to directly access another GPU to use direct high-speed DMA transfer. The test method is simple to operate, high in automation degree and high in practicability, can save manpower, effectively ensures the stability of the performance of the server, and is a very effective method for verifying the product quality of the GPU server.

Description

A testing method based on GPUDerict RDMA

technical field

The invention relates to the field of GPU server testing, in particular to a testing method based on GPUDerictRDMA.

Background technique

With the continuous development of technology in the IT field, traditional information services and increasingly powerful cloud computing services have higher and higher requirements for servers, and the replacement time of technology is getting faster and faster. , multi-core and even process and instruction set optimization have encountered unprecedented difficulties. However, GPU servers have opened a door in the field of heterogeneous computing. More and more supercomputing centers, enterprises and research institutions are building computing resource pools with coprocessors as the core, and developing and optimizing computing resources on heterogeneous platforms. For the adapted application layer, customers have more and more demands on GPU servers to meet the growing demand for computing power.

Contents of the invention

The technical task of the present invention is to provide a testing method based on GPUDerictRDMA aiming at the deficiencies of the prior art. This method not only effectively tests the performance of the GPU server, but also provides important performance data for the customer's performance requirements on the GPU server.

The technical solution adopted by the present invention to solve its technical problems is:

A test method based on GPUDerictRDMA, which directly accesses GPU memory, avoids unnecessary system memory copy and CPU overhead when accessing pinned CUDA host memory, and accelerates communication with network and storage devices in the same system One GPU directly accesses another GPU using direct high-speed DMA transmission, which increases P2P memory access, truly releases the host CPU resources, eliminates unnecessary frequent data transmission in the host CPU, and does not participate in input RDMA operations at all; including HCA card, GPU card, NvidiaDriver, NvidiaCUDAtoolkit necessary for GPU, and MLNX_OFED driver necessary for infiniband, and an nv_peer_mem package for communication between GPU and IB card.

HCA cards are MellanoxConnectX and later products, and GPU cards are K20 and later products.

The GPUDirectRDMA test method is as follows:

1. Test tools

a. cuda_6.5.14_linux_64.run

b. nvidia_peer_memory-1.0-0.tar.gz

c. mvapich2-gdr-cuda6.5-gnu-2.1-0.1.a.el6.x86_64.rpm

d. MLNX_OFED_LINUX-2.4-1.0.0-rhel6.2-x86_64.iso

2. Test method

a. HCA driver installation

mount-oro,loopMLNX_OFED_LINUX-2.4-1.0.0-rhel6.2-x86_64.iso/mnt

cd/mnt

./mlnxofedinstall

b. Graphics card driver installation

chmod 777cuda_6.5.14_linux_64.run

./cuda_6.5.14_linux_64.run --extract=/root/rdma

./NVIDIA-Linux-x86_64-340.29.run

c. CUDA installation

./cuda-linux64-rel-6.5.14-18749181.run

d. Environment variable settings

vi ~/.bashrc

Add at the end: exportPATH=/usr/local/cuda-6.5/bin:$PATHexportLD_LIBRARY_PATH=/usr/local/cuda-6.5/lib64:$LD_LIBRARY_PATH

source ~/.bashrc

vi /etc/ld.so.conf

Add at the end: /usr/local/cuda-6.5/lib64

Ldconfig

e. nv_peer_mem installation

tar-zxf ../nvidia_peer_memory-1.0-0.tar.gz

rpmbuild --rebuild nvidia_peer_memory-1.0-0.src.rpm

rpm-ivh/root/rpmbuild/RPMS/x86_64/nvidia_peer_memory-1.0-0.x86_64.rpm

/etc/init.d/nv_peer_memstart starts nv_peer_mem service

f. mvapich2 installation

rpm-Uvh--nodepsmvapich2-gdr-cuda6.5-gnu-2.1-0.1.a.el6.x86_64.rpm

g. GPUDirectRDMA bandwidth test

/opt/mvapich2/gdr/2.1/cuda6.5/gnu/bin/mpirun_rsh-np2c1c2MV2_USE_CUDA=1MV2_USE_GPUDIRECT=1 /opt/mvapich2/gdr/2.1/cuda6.5/gnu/libexec/mvapich2/osu_bw-dcudaDD

h. GPUDirectRDMA latency test

/opt/mvapich2/gdr/2.1/cuda6.5/gnu/bin/mpirun_rsh-np2c1c2MV2_USE_CUDA=1MV2_USE_GPUDIRECT=1/opt/mvapich2/gdr/2.1/cuda6.5/gnu/libexec/mvapich2/osu_latency-dcudaDD

A kind of test method based on GPUDerictRDMA of the present invention compares with prior art, the beneficial effect produced is that the present invention directly accesses GPU memory, avoids unnecessary system memory copy and CPU overhead when accessing fixed (pinned) CUDA host memory , speeding up the communication with the network and storage devices. One GPU in the same system can directly access another GPU using direct high-speed DMA transfer, which increases P2P memory access, truly releases the host CPU resources, and eliminates the host CPU. Unnecessarily frequent data transfers in the middle, do not participate in the incoming RDMA operation at all.

The GPUDirectRDMA test method not only effectively tests the performance of GPU servers, but also provides important performance data for customers' performance requirements for GPU servers. The test method is simple to operate, has a high degree of automation, strong practicability, can save manpower, effectively ensures the stability of server performance, and is a very effective method for verifying the quality of GPU server products.

Description of drawings

Figure 1 is a schematic diagram of the test method based on GPUDerictRDMA.

detailed description

A test method based on GPUDerictRDMA of the present invention will be described in detail below in conjunction with the accompanying drawings.

A test method based on GPUDerictRDMA, which directly accesses GPU memory, avoids unnecessary system memory copy and CPU overhead when accessing pinned CUDA host memory, and accelerates communication with network and storage devices in the same system One GPU directly accesses another GPU using direct high-speed DMA transmission, which increases P2P memory access, truly releases the host CPU resources, eliminates unnecessary frequent data transmission in the host CPU, and does not participate in input RDMA operations at all; including HCA card, GPU card, NvidiaDriver, NvidiaCUDAtoolkit necessary for GPU, and MLNX_OFED driver necessary for infiniband, and an nv_peer_mem package for communication between GPU and IB card.

HCA cards are MellanoxConnectX and later products, and GPU cards are K20 and later products.

The GPUDirectRDMA test method is as follows:

3. Test tools

h. cuda_6.5.14_linux_64.run

i. nvidia_peer_memory-1.0-0.tar.gz

j. mvapich2-gdr-cuda6.5-gnu-2.1-0.1.a.el6.x86_64.rpm

k. MLNX_OFED_LINUX-2.4-1.0.0-rhel6.2-x86_64.iso

4. Test method

b. HCA driver installation

mount-oro,loopMLNX_OFED_LINUX-2.4-1.0.0-rhel6.2-x86_64.iso/mnt

cd/mnt

./mlnxofedinstall

b. Graphics card driver installation

chmod 777cuda_6.5.14_linux_64.run

./cuda_6.5.14_linux_64.run --extract=/root/rdma

./NVIDIA-Linux-x86_64-340.29.run

c. CUDA installation

./cuda-linux64-rel-6.5.14-18749181.run

d. Environment variable settings

vi ~/.bashrc

Add at the end: exportPATH=/usr/local/cuda-6.5/bin:$PATHexportLD_LIBRARY_PATH=/usr/local/cuda-6.5/lib64:$LD_LIBRARY_PATH

source ~/.bashrc

vi /etc/ld.so.conf

Add at the end: /usr/local/cuda-6.5/lib64

Ldconfig

l. nv_peer_mem installation

tar-zxf ../nvidia_peer_memory-1.0-0.tar.gz

rpmbuild --rebuild nvidia_peer_memory-1.0-0.src.rpm

rpm-ivh/root/rpmbuild/RPMS/x86_64/nvidia_peer_memory-1.0-0.x86_64.rpm

/etc/init.d/nv_peer_memstart starts nv_peer_mem service

m, mvapich2 installation

rpm-Uvh--nodepsmvapich2-gdr-cuda6.5-gnu-2.1-0.1.a.el6.x86_64.rpm

n. GPUDirectRDMA bandwidth test

/opt/mvapich2/gdr/2.1/cuda6.5/gnu/bin/mpirun_rsh-np2c1c2MV2_USE_CUDA=1MV2_USE_GPUDIRECT=1 /opt/mvapich2/gdr/2.1/cuda6.5/gnu/libexec/mvapich2/osu_bw-dcudaDD

h. GPUDirectRDMA latency test

/opt/mvapich2/gdr/2.1/cuda6.5/gnu/bin/mpirun_rsh-np2c1c2MV2_USE_CUDA=1MV2_USE_GPUDIRECT=1/opt/mvapich2/gdr/2.1/cuda6.5/gnu/libexec/mvapich2/osu_latency-dcudaDD

As traditional information services and increasingly powerful cloud computing services have higher and higher requirements for servers, customers' demand for GPU servers is increasing. The GPUDirectRDMA test method not only effectively tests the performance of GPU servers, but also provides customers with GPU The performance requirements provide important performance data. The test method is simple to operate, has a high degree of automation, strong practicability, can save manpower, effectively ensures the stability of server performance, and is a very effective method for verifying the quality of GPU server products.

Claims (2)

1. A test method based on GPUDerictRDMA, characterized in that it directly accesses GPU memory, avoids unnecessary system memory copy and CPU overhead when accessing fixed (pinned) CUDA host memory, and accelerates the communication between network and storage devices. One GPU in the same system directly accesses another GPU using direct high-speed DMA transfer, which increases P2P memory access, truly releases the host CPU resources, eliminates unnecessary frequent data transmission in the host CPU, and does not participate in input at all. RDMA operation; including HCA card, GPU card, NvidiaDriver, NvidiaCUDAtoolkit, MLNX_OFED driver and nv_peer_mem package for communication between GPU and IB card. 2. A kind of test method based on GPUDerictRDMA according to claim 1, characterized in that the HCA card is MellanoxConnectX.