JP7256811B2 - Method and system for accelerating AI training using advanced interconnect technology - Google Patents

Description

Embodiments of the present disclosure relate generally to machine learning. More specifically, embodiments of the present disclosure relate to neural network training.

Neural networks are becoming increasingly complex to solve complex problems. Complex deep learning algorithms and more bandwidth are required when training complex neural networks, resulting in increased training time, cost and power consumption. To accelerate training, a high-end server (eg, a fast server or server cluster with a more complex interface) is utilized to improve computation and communication and reduce the cost of expensive hardware. However, conventional solutions remain challenging in terms of performance and cost.

According to a first aspect, some embodiments of the present disclosure are a computer-implemented method for training an artificial intelligence (AI) model using a data processing (DP) accelerator, comprising: receiving a request from the CPU to train the AI model based on a training data set comprising a plurality of data blocks delivered by the CPU; and a plurality of general purpose processing units (GPUs) arranged in a logical ring. and training the AI model by performing multiple DP iterations by: each perform a first predetermined DP operation on one of said plurality of data blocks in parallel to produce a respective first DP result; and in a second DP cycle, said plurality of each of the GPUs of is forwarding, via an inter-processor link, each first DP result to a GPU downstream in the logical ring for further processing. do.

According to a second aspect, some embodiments of the present disclosure are a data processing system comprising at least one CPU and a plurality of general purpose processing units (GPUs) coupled to said CPU, Each of the plurality of GPUs is configured to perform an artificial intelligence AI data processing (DP) operation delivered by the CPU, the operation being performed on a training data set comprising a plurality of blocks of data delivered by the CPU. and performing multiple DP iterations by a plurality of general purpose processing units (GPUs) arranged in a logical ring to train the AI model based on and training, wherein the plurality of DP iterations, for each DP iteration, in a first DP cycle, the plurality of GPUs each train for one of the plurality of data blocks. , executing a first predetermined DP operation in parallel to generate a respective first DP result, and in a second DP cycle, the plurality of GPUs each, via an inter-processor link, each A data processing system is provided, including forwarding a first DP result to a GPU downstream in a logical ring for further processing.

According to a third aspect, some embodiments of the present disclosure are a non-transitory machine-readable medium having instructions stored thereon, said instructions, when executed by a processor, giving said processor artificial intelligence. causing an AI training operation to be performed, said operation receiving a request from said CPU to train said AI model based on a training data set comprising a plurality of data blocks delivered by said CPU; training the AI model by executing multiple DP iterations by multiple general purpose processing units (GPUs) located in a , in a first DP cycle, each of said plurality of GPUs performs a first predetermined DP operation in parallel on one of said plurality of data blocks, and outputs a respective first DP result to generating, and in a second DP cycle, each of the plurality of GPUs forwards, via an inter-processor link, a respective first DP result to a GPU downstream in the logical ring for further processing; To provide a non-transitory machine-readable medium comprising:

According to a fourth aspect, some embodiments of the present disclosure provide a computer program for implementing the method according to the first aspect when said computer program is executed by a processor. do.

The drawings illustrate, by way of example, embodiments of the invention and are not intended to limit embodiments of the invention. In the drawings, similar elements are numbered the same.
1 illustrates an example system for training an AI model, according to embodiments herein; FIG. 2A-2F illustrate an exemplary process of data transfer in training an AI model in accordance with embodiments herein. FIG. 2F is a flow chart showing a variation of the process of FIGS. 2A-2F; FIG. FIG. 3 illustrates an exemplary architecture for data compression, data manipulation, and interconnection buses, according to embodiments herein; FIG. 4 illustrates a zero-sum compression technique according to one embodiment; FIG. 4 is a diagram illustrating an example of operations on compressed data according to one embodiment; FIG. 4 illustrates an exemplary process of AI model training according to one embodiment;

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described with reference to the drawings. The following description and drawings are illustrative of the disclosure and should not be construed as limiting the disclosure. Many specific details are described in order to provide a thorough understanding of various embodiments of the disclosure. However, in some cases well known or conventional details have not been described in order to provide a concise description of the embodiments of the disclosure.

References herein to "one embodiment" or "an embodiment" may include a particular feature, structure, or feature described using an embodiment in at least one embodiment of the disclosure means that The appearances of the phrase "in one embodiment" in various places in the specification are not necessarily all referring to the same embodiment.

According to various embodiments, methods and systems are provided for accelerating artificial intelligence (AI) training utilizing advanced interconnect technology. Embodiments described in the present disclosure significantly reduce interconnect communication bandwidth requirements, power consumption, and reduce training time by utilizing software and hardware components, thereby reducing accuracy loss. Improve training performance in situations where there is no AI model training is performed in a distributed system using system data compression and decompression, combined with an efficient All-Reduce algorithm.

According to one embodiment, a computer-implemented method of AI model training includes performing multiple iterations in a Scatter-Reduce process on a cluster of processors, each processor running a graphics It may also be a processing unit (GPU). To train the neural network model, the processors are arranged in a logical ring, each processor having a plurality of data blocks, each data block providing a set of parameters or sets of parameters within the neural network model. Each may represent a set of gradients to update.

In each iteration, the processor receives a compressed data block from a previous processor in the logical ring, performs an operation on the received compressed data block and on the compressed data block currently generated by the processor, It operates on data blocks and transmits the operated data blocks to subsequent processors in the logical ring. After multiple iterations, each block of data on multiple processors has all been compressed and manipulated. The method further includes, in each of the plurality of processors, identifying compressed data blocks, where the compressed data blocks were computed from corresponding data blocks of the plurality of processors.

In one embodiment, the identified compressed data blocks are distributed to each of the other processors in the logical ring, decompressed thereon, and used to update parameters within the neural network model. Processors can be attached to central processing units (CPUs) in different systems of the distributed AI model training system. In one embodiment, each processor may include a hardware- or software-based compression module for compressing and decompressing data blocks using zero-value compression techniques. A compressed data block can be represented by a data structure having a bitmask portion and a compressed data portion, where the bitmask contains bits that indicate the location of non-zero values within the data block.

According to one embodiment, upon receiving a request for AI training from a central processing unit (CPU), each of the general purpose processing units (GPUs) arranged in a logical ring processes data blocks delivered by the CPU. It is configured to repeatedly execute data processing (DP) operations in a pipeline fashion. Each GPU operates as a DP accelerator with respect to the CPU. For each iteration, in the first DP cycle, the multiple GPUs each perform a first predetermined DP operation (e.g., data compression) in parallel on one of the data blocks; Generate respective DP results. In the second DP cycle, the multiple GPUs each forward their respective DP results via corresponding inter-processor links to corresponding downstream GPUs in the logical ring for further processing there. For purposes of explanation, a GPU is used as an example of a DP accelerator, but other types of processors or processing logic may be used as DP accelerators.

In one embodiment, during the second DP cycle, each GPU also receives processing results from corresponding upstream GPUs in the logical ring via corresponding inter-processor links, and the received processing results are sent to GPUs is used to perform further processing. In one embodiment, during the third DP cycle, each of the multiple GPUs receives a first data block processed by it (e.g., processing results) and a second data block received from the upstream GPU ( a second predetermined DP operation (eg, a join operation such as an addition) simultaneously. In one embodiment, during the fourth DP cycle, each of the multiple GPUs performs a further DP operation, such as a data decompression operation.

FIG. 1 illustrates an example system for training AI models, according to one embodiment. As shown in FIG. 1, the system includes a general purpose processing unit (GPU) cluster 101 distributed across multiple servers (e.g., Server A 103 and Server B 105), each server including one or more CPUs; Each CPU is associated with one or more data processing (DP) accelerators, such as GPUs.

The server may include CPU 107 and CPU 109 communicating with each other via Ethernet connection 111 . In the example system shown in FIG. 1, each CPU may have multiple GPUs connected to it via Peripheral Device Interconnection High Speed (PCIe) switches. For example, in server A103, GPU117, GPU119 and GPU121 are connected to CPU A107 via PCIe switch A113. In server B105, GPU123, GPU125 and GPU127 are connected to CPU B109 via PCIe B115.

CPU 107 and CPU 109 can communicate with each other via an inter-processor link, such as Ethernet connection 111, to coordinate tasks for training the neural network. For example, job commands can be delivered to each server via Ethernet connection 111 . Job commands can then be delivered from a CPU in the server to a GPU connected to that CPU. When the job command is delivered, data can be transferred between the GPUs in the system via the corresponding inter-chip links 122 . Inter-chip links 112 may employ a variety of inter-chip interconnect solutions such as, for example, cache coherent interconnect (CCIX) links for accelerators. As shown in FIG. 1, a unidirectional ring topology can be used, but the GPUs in the system are arranged in a bidirectional ring topology.

CCIX is an open cache coherence interconnect architecture developed by the CCIX Alliance. CCIX is designed to simplify communication between a central processor, such as a CPU, and various accelerators, such as GPUs, in a system by extending standard PCIe's cache coherency. CCIX is a high performance chip-to-chip interconnect architecture that provides a cache coherence framework for heterogeneous system architectures. Cache coherency between the central processing unit and various other accelerators in the system is always automatically maintained. Each device that supports CCIX includes at least one CCIX port, which is pin-compatible with any other CCIX-enabled device. CCIX supports various topologies such as chip-to-chip, chip-to-switch-to-chip, grid, daisy chain, and ring.

In one embodiment, the GPUs are configured to perform AI training operations in a pipeline fashion on data blocks delivered from each CPU. Each GPU also communicates with each other via an inter-processor link. GPUs may be configured in a loop to receive processing results from upstream GPUs for further data processing. Each GPU may further send processing results to its corresponding downstream GPU for performing further processing. Thus, each GPU performs the distributed DP operations in parallel and sends the DP results to downstream GPUs. Each GPU also receives processing results from its upstream GPU and performs further processing.

2A-2F illustrate an exemplary process of data transfer in training an AI model according to one embodiment. Although three GPUs are shown here, GPUs 203, 205, and 207, an exemplary process is the complexity of the neural network to be trained, the size of the data for training, and the speed of training desired by the user. As many GPUs as possible (eg, thousands of GPUs) can be used, depending on a number of factors.

Examples of neural networks trained on the exemplary system include multi-layer perceptron (MLP) neural networks comprising a set of connected neurons. Neurons in an MLP neural network may be fully connected when each neuron in one layer is connected to each neuron in subsequent layers by parameters (eg, weights and biases).

During training of a neural network model, gradient descent (i.e. back transfer) can be used to determine a set of parameters to minimize the difference between the expected and actual output of the neural network model. . Gradient descent involves computing the slope of a loss/error function and updating existing parameters in response to the slope. This cycle is repeated until a local minimum of the loss function is reached.

In one embodiment, a neural network model training data set is divided into multiple subsets, each subset being distributed on one of the GPUs so that training of the neural network is performed by multiple GPUs in parallel. used to train neural network models. Each GPU can have a complete copy of the neural network model.

Each subset of the training data set can be logically divided into multiple equally sized data blocks. In the exemplary process, the number of blocks equals the number of GPUs. Parallel training of neural network models requires multiple iterations of gradient descent. At each iteration, each GPU performs forward propagation of the neural network model on the data on the GPU, followed by back propagation of the error, to compute the gradient of the loss versus the network parameters. The GPUs then communicate with each other to compute a gradient statistic (e.g., mean, maximum, or minimum), and utilize the statistic (e.g., mean gradient) to update the parameters can be obtained. A neural network model may have a large number of parameters (eg, billions of parameters), each associated with a respective gradient value. Thus, for neural networks, the magnitude of gradients is very large and transferring gradients between GPUs takes up considerable bandwidth.

Referring again to FIGS. 2A-2F, the exemplary process shows algorithms for reducing the bandwidth required by data transfers between GPUs. In one embodiment, bandwidth as used in this disclosure is the maximum data transfer rate over a given network connection. The algorithm can include two processes. The first process is the Scatter-Reduce process and the second process is the Allgather process. During the Scatter-Reduce process, GPUs can exchange data such that each GPU ends up with a number of final result blocks. During the Allgather process, GPUs can exchange these result blocks so that all GPUs end up with a complete final result.

Each GPU may include one or more applications configured to divide a subset of the training data set on the GPU into equally sized blocks of data. In an exemplary system, the number of data blocks on each GPU is the number of GPUs. During training of the neural network model, each data block can generate its own set of gradients.

In this example, as mentioned above, there are three GPUs in the system, so the number of data blocks on each GPU is three. From the subset of training data on GPU #0 203, we can generate 3 sets of gradients a ₀ 215, b ₀ 231, c ₀ 237, and from the subset of training data on GPU #1 205, 3 more A set of gradients a ₁ 217, b ₁ 223, c ₁ 239 can be generated. Similarly, from a subset of training data on GPU#2 207, three sets of gradients a ₂ 219, b ₂ 235, c ₂ 241 are generated. In one embodiment, the different gradient sets on each GPU may be stored in an array or another data structure.

As an example, the algorithm may be designed to sum the gradients produced by each subset of the training dataset. Thus, when the algorithm is complete, each GPU will have a sum of gradients generated from the training dataset.

A GPU in the exemplary process can have N−1 iterations during the Scatter-Reduce process, where N is the total number of GPUs in the system. Thus, the GPU in the exemplary system can have two iterations. On each iteration, each GPU sends a set of gradients on the GPU to its right neighbor, receives a set of gradients from its left neighbor, and adds the two sets of gradients to produce a set of It can be a new gradient. The set of gradients sent or received by each GPU is different for each iteration. The nth GPU begins with the nth set of gradients being sent and reverses the process to receive the (n-1)th set of gradients.

2A to 2C are diagrams showing the Scatter-Reduce processing. FIG. 2A shows data transmission in the first iteration of the Scatter-Reduce process. After the first transmission and first reception are completed, each GPU has an array element with values representing the sum of two sets of gradients on two different GPUs. For example, a first element a1 in GPU 205 may include the sum of gradient sets from second GPU 205 and first GPU 203 . FIG. 2B shows the data transfer in the second iteration of the Scatter-Reduce process and also shows the intermediate sum after the completion of the first iteration of the Scatter-Reduce process. In the second iteration, the Scatter-Reduce process follows, and at the end of the Scatter-Reduce process (i.e., after the second iteration in this example), each GPU has a corresponding It has one array element containing the sum of all the gradients of the array elements. FIG. 2C shows the final state at the end of the Scatter-Reduce process.

Figures 2D-2F illustrate the Allgather process. The process is similar to Scatter-Reduce and has N−1 iterations. Compared to Scatter-Reduce, it differs in that the received gradient covers the gradient in the corresponding array element on the receiving GPU instead of accumulating on the gradient received by the GPU. FIG. 2D shows data transfer in the first iteration of the Allgather process. After the first iteration is complete, each GPU has two array elements each containing the sum of all gradients in the corresponding array element across all GPUs, as shown in FIG. 2E. FIG. 2E shows the Allgather process in the second iteration, the final iteration in the exemplary process. As shown in FIG. 2F, at the end of the Allgather process, the GPU has fully accumulated gradients from the entire training dataset. The exemplary process is bandwidth optimized because all data transfers occur synchronously in discrete iterations.

FIG. 3 is a flow chart showing a modification of the processing of FIGS. 2A-2F. In one embodiment, the exemplary process illustrated in FIG. 3 can be used to transfer gradients for updating neural network parameters during training of the neural network model. Here, tens of megabytes of data need to be transferred between distributed servers, and they also need to operate cooperatively. This requires efficient hardware and software that can improve performance and latency.

In one embodiment, the exemplary process utilizes an All-Reduce algorithm and improves performance and latency through software and hardware co-design. Co-design of software and hardware means designing hardware and software at the same time in order to realize a desired function. This exemplary process includes hardware components such as the accelerator's cache coherence interconnect (CCIX) used to connect GPUs in the cluster, and a zero-value compression module that enables hardware computations based on compressed data. and using software modules such as compression modules. This exemplary process uses system data compression in a distributed system designed to perform an efficient All-Reduce process. This allows gradients generated from different subsets of the training data set to be accumulated and distributed to each GPU faster, thus making AI model training faster.

In FIG. 3, the left row shows the exemplary All-Reduce process 302 detailed in FIGS. 2A-2F, and the right row shows the improved All-Reduce process using system compression on distributed systems. indicates FIG. 3 uses, as an example, three GPUs arranged to form a logical ring.

In the typical All-Reduce process 302 and the improved All-Reduce process, blocks of data transferred between CPUs are stored in data structures (e.g., arrays), and the blocks of data are used to train a neural network model. may be gradients generated from different blocks of a subset of the training data set used to Each GPU can have a complete copy of the neural network model being trained. Gradients are passed between GPUs to update the parameters of the neural network model.

In one embodiment, data blocks on each GPU may be compressed in the first iteration or first processing cycle of the Scatter-Reduce process by a compression module, which is implemented in hardware. may be implemented as software modules. For example, in operations 301, 315, and 329, data block _a0 on GPU#0 203, data block _b1 on GPU#1 205, and data block _c2 on GPU#2 207 are compressed, respectively.

Compressed data blocks may be sent to neighboring GPUs in the next processing cycle. For example, in operation 303 a compressed data block on GPU #0 203 may be sent to GPU #1 205, and in operation 317 a compressed data block on GPU #1 205 may be sent to GPU #2 207. Well, in operation 331 the compressed data block on GPU #2 207 may be sent to GPU #0 203 .

In one embodiment, compressed data blocks may be sent to neighboring GPUs at the same time that different data blocks on each GPU are compressed and appended to the received compressed data as described above. Although the exemplary embodiment exemplifies a summation operation, other operations (eg, multiplication, deduction, mathematical averaging, etc.) may be used.

For example, at operation 305 , data block c ₀ on GPU # 0 203 may be compressed and appended to compressed data block c ₂ received from GPU # 2 207 . At operation 319 , data block a ₁ on GPU # 1 205 may be compressed and appended to compressed data block a ₀ received from GPU # 0 203 . At operation 333 , data block b ₂ on GPU # 2 207 is compressed and added to compressed data block b ₁ received from GPU # 1 205 .

The above process can be repeated for each remaining iteration of the Scatter-Reduce process. The number of iterations may be the number of GPUs minus one. Thus, the Scatter-Reduce process 305 in the improved All-Reduce process can have two iterations. In each of the remaining iterations, each GPU may send the sum of the compressed data blocks from multiple GPUs to the next GPU instead of sending the original compressed data block on the GPU.

For example, in the second iteration, GPU # 0 203 may send the sum of compressed data block c ₀ and compressed data block c ₂ to GPU # 1 205 in operation 307 . At operation 321 , GPU # 1 205 may send the sum of compressed data block a ₀ and compressed data block a ₁ to GPU # 2 207 . At operation 335 , GPU # 2 207 may send the sum of compressed data block b ₁ and compressed data block b ₂ to GPU # 0 203 .

In one embodiment, while the sum of the compressed data blocks is being sent to neighboring GPUs, each GPU compresses the remaining data blocks on the GPU to match the compressed data received from the previous GPU in this previous logical ring. The sum of the data blocks may be added to the compressed data blocks. For example, in operation 309, data block _b0 on GPU #0 202 may be compressed and added to the sum of compressed data blocks _b1 and _b2 . In operation 323, data block _c1 on GPU #1 205 may be compressed and added to the sum of compressed data blocks c0 and _c2 . In operation 337, data block _a2 on GPU #2 207 may be compressed and added to the sum of compressed data blocks a0 and a1.

Thus, at the end of the Scatter-Reduce process, in the exemplary process each GPU has a sum of compressed data blocks from corresponding locations across all GPUs in the array.

During the Allgather process, each GPU may distribute sums of compressed data blocks from corresponding locations in the array to other GPUs. As a result, at the end of the Allgather process, each GPU will have a copy of the sum of all compressed data blocks. Each GPU may then decompress the compressed sum, as shown in operations 313 , 327 and 341 . The uncompressed sums on each GPU can be used to update the parameters of the copy of the neural network model on the GPU.

FIG. 4 shows an exemplary architecture for data compression, data manipulation, and an interconnect bus, according to one embodiment.

The graph of FIG. 4 illustrates compressing raw data blocks 405 and 407, transferring the compressed data blocks over interconnect buses 416 and 418, and performing operations 413 and 419 on the compressed data. , decompressing compressed data into RAW data.

As shown in FIG. 3, a pair of compression and decompression modules may be used on each GPU. For example, on GPU A 401 compression module 412 and decompression module 409 may be used, and on GPU B 403 compression module 417 and decompression module 415 may be used.

Any compression algorithm can be used in compression modules 412 and 417 . Examples of compression algorithms include zero-value compression algorithms/techniques, which are described in detail in the following disclosure. If the zero-value ratio is 50%, employing a zero-value compression algorithm can save nearly 50% of the data transfer bandwidth. Bandwidth benefits can exceed 50% when the interconnect bus and various operations on compressed data are combined.

FIG. 5 illustrates a zero compression technique according to one embodiment. In FIG. 5, matrix 513 is the original 4×4 data array for training the neural network model. Data structure 510 shows the compressed form of matrix 513 using zero-value compression techniques. Data structure 510 includes multiple fields such as, for example, type field 501 , length field 503 , bitmask field 505 , and compressed data field 507 . Matrix 513 and data structure 510 can be converted to each other using compression 511 and decompression 509 .

In one embodiment, type field 501 represents the data type of the values in matrix 513 . Examples of data types include floating point (FP) 32, FP 16, and integer (INT) 8. The length, in bytes, represents the total size of bitmask field 505 and compressed data field 507, or represents the size of compressed data field 507 with bitmask bytes of a fixed size. Bitmask field 505 is set to '1' to represent a non-zero value at a particular location in matrix 513 and to '0' to represent a zero value. Compressed data field 507 contains only non-zero valued data with correct alignment/offset. The bitmask field may be used by a decompression module (eg, decompression module 409 or 415 of FIG. 4) to write non-zero values back to their original locations in 4×4 data array 513 .

FIG. 6 is a diagram showing an example of operations on compressed data according to this embodiment. As shown in FIG. 6, we take the summation operation as an example to illustrate how to operate on two compressed data blocks.

In one embodiment, compressed data 617 is a data structure representing matrix A 613 in the compressed form of matrix A 613, and compressed data 619 is a data structure representing matrix B 615 in the compressed form of matrix B 615. These two structures are generated by the compression technique shown in FIG. 5 and decompressed by a decompression module (eg, decompression module 409 or 415) into matrix A 613 and matrix B 615, respectively.

In one embodiment, to sum the two compressed matrices 613 and 615 in their compressed form, a hardware compression module (eg, compression module 411 or 417 in FIG. 4) first creates two compressed data structures 617 and 619 can be pipelined to compare the bits in the bitmask field in one data structure with the bits in the bitmask field of the other data structure and output the result 621 of the comparison.

By transferring data between GPUs in compressed form, the bandwidth required for data transfer can be reduced. In addition, compressed data blocks occupy less memory than their uncompressed forms, and fewer bits are read from and written to memory during operation, thus reducing the memory required to manipulate compressed data blocks. can.

For example, a sum operation may require two reads and one write. Because the data read and written from memory is in compressed form, the memory required for sum operations can be reduced.

FIG. 7 shows an exemplary process 700 of AI model training according to one embodiment. Process 700 may be performed by processing logic including software, hardware, or a combination thereof.

Referring again to FIG. 7, at operation 701, a neural network model is trained by performing multiple iterations on multiple processors arranged as a logical ring, each processor containing multiple data blocks. At operation 702, for each of the plurality of iterations, one of the plurality of processors receives a compressed data block from a previous processor in the logical ring and combines the received compressed data block with the compressed data block generated on that processor. It performs operations on the compressed data blocks to compute data blocks and transmits the computed data blocks to subsequent processors in the logical ring. In operation 703, in each of the plurality of processors, a compressed data block computed based on corresponding data blocks from the plurality of processors is identified. The identified data blocks are distributed to each of the other processors, where they are decompressed and used for AI model training, such as updating parameters of neural network models.

Some or all of the components described above may be realized by software, hardware, or a combination thereof. For example, such components may be implemented as software installed and stored on a permanent storage device, which software is loaded into memory and executed by a processor (not shown) and described herein. The entire described process or operation can be performed. Alternatively, such components execute programmed or embedded in application-specific hardware such as an integrated circuit (e.g., an application-specific IC or ASIC), a digital signal processor (DSP), or a field-programmable gate array (FPGA). It can be implemented as executable code, which can be accessed by applications through corresponding drivers and/or operating systems. Further, such components may be implemented as specific hardware logic within a processor or processor core as part of an instruction set accessible by software components via one or more specific instructions.

Some of the above detailed descriptions are presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the ways used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. Here an algorithm is generally considered to be a self-adapting sequence of operations that produces a desired result. These operations are those requiring physical manipulations of physical quantities.

All these and similar terms are only convenient tags to be associated with and applied to appropriate physical quantities. It should be clear from the discussion above that, unless expressly indicated otherwise, descriptions using terms such as those in the following claims refer to physical (electronic) operations within the registers and memory of a computer system. a computer system that converts data represented as physical quantities into other data similarly represented as physical quantities within the memory or registers of a computer system or other such information storage, transmission or display device, or It should be understood that similar electronic computing device operations and processes are meant throughout this specification.

Embodiments of the present disclosure also relate to apparatus for performing the operations herein. Such computer programs are stored on non-transitory computer-readable media. A machine-readable medium includes any mechanism for storing information in a form readable by a machine (eg, a computer). For example, a machine-readable (eg, computer-readable) medium includes a machine-readable storage medium, such as read-only memory (“ROM”), random-access memory (“RAM”), magnetic disk storage media, optical storage media, flash memory devices. including.

The processes or methods depicted in the preceding figures may be implemented by processing logic including hardware (e.g., circuits, dedicated logic, etc.), software (e.g., contained on non-transitory computer-readable media), or a combination of both. may be performed. Although the process or method has been described above according to some sequence of operations, it should be understood that some of the operations described may be performed in different sequences. Moreover, some operations may be performed in parallel rather than serially.

Embodiments of the present disclosure are not described with reference to any particular programming language. It should be appreciated that the teachings of the embodiments of the disclosure described herein may be implemented using a variety of programming languages.

The invention has been described in detail above with reference to specific embodiments. It will be evident that various changes can be made without departing from the broader spirit and scope of this disclosure as set forth in the following claims. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense.

Claims (6)

1. A computer-implemented method for training an artificial intelligence (AI) model using a data processing (DP) accelerator, comprising:
receiving a request from the CPU to train the AI model based on a training data set containing a plurality of data blocks distributed by the CPU;
performing multiple DP iterations by multiple general purpose processing units (GPUs) arranged in a logical ring to train the AI model;
The multiple DP iterations are
For each DP iteration ,
In a first DP cycle, each of the plurality of GPUs performs a data compression operation in parallel on one of the plurality of data blocks to generate a respective first compressed data block ;
In a second DP cycle, each of the plurality of GPUs forwards, via an interprocessor link, a respective first compressed data block to a GPU downstream in the logical ring for further processing; and each receive, via a corresponding interprocessor link or CCIX connection, from an upstream GPU in the logical ring for further processing a second compressed data block generated by the upstream GPU performing a data compression operation; death,
In the third DP cycle, each of the plurality of GPUs processes the first compressed data block processed by the current GPU and the second compressed data block processed by the corresponding upstream GPU and received from that GPU. performing a summation operation on the blocks to produce a first DP result;
In a fourth DP cycle, the plurality of GPUs each perform a data decompression operation on the first DP result, and the data blocks obtained by decompression are used for the next DP iteration. including
The summing operation pipelines the first compressed data block and the second compressed data block such that the bits in the bitmask field of one compressed data block are replaced by the bits in the bitmask field of the other compressed data block. A computer-implemented method that is an operation that compares the bits of and outputs the result of the comparison. 2. The method of claim 1, wherein at least some of the data blocks represent parameters or gradients generated as part of training the AI model. The data compression operation is performed using a zero-value compression algorithm that compresses one or more data blocks into a data structure having a bitmask portion and a compressed data portion, wherein the bitmask portion is a containing bits indicating the positions of non-zero values,
The method of claim 1. A data processing system,
at least one CPU;
a plurality of general purpose processing units (GPUs) connected to the CPU;
each of the plurality of GPUs is configured to perform artificial intelligence AI data processing (DP) operations delivered from the CPU;
Said operation is
receiving a request from the CPU to train an AI model based on a training data set containing a plurality of data blocks distributed by the CPU;
performing multiple DP iterations by multiple general purpose processing units (GPUs) arranged in a logical ring to train the AI model;
The multiple DP iterations are
For each DP iteration ,
In a first DP cycle, each of the plurality of GPUs performs a data compression operation in parallel on one of the plurality of data blocks to generate a respective first compressed data block ;
In a second DP cycle, each of the plurality of GPUs forwards, via an interprocessor link, a respective first compressed data block to a GPU downstream in the logical ring for further processing; and each receive, via a corresponding interprocessor link or CCIX connection, from an upstream GPU in the logical ring for further processing a second compressed data block generated by the upstream GPU performing a data compression operation; death,
In the third DP cycle, each of the plurality of GPUs processes the first compressed data block processed by the current GPU and the second compressed data block processed by the corresponding upstream GPU and received from that GPU. performing a summation operation on the blocks to produce a first DP result;
In a fourth DP cycle, the plurality of GPUs each perform a data decompression operation on the first DP result, and the data blocks obtained by decompression are used for the next DP iteration. including
The summing operation pipelines the first compressed data block and the second compressed data block such that the bits in the bitmask field of one compressed data block are replaced by the bits in the bitmask field of the other compressed data block. A data processing system that is an operation that compares the bits in and outputs the result of the comparison. A non-transitory machine-readable medium having instructions stored thereon,
The instructions, when executed by a processor, cause the processor to perform operations of artificial intelligence AI training, the operations comprising:
receiving a request from the CPU to train an AI model based on a training data set containing a plurality of data blocks distributed by the CPU;
performing multiple DP iterations by multiple general purpose processing units (GPUs) arranged in a logical ring to train the AI model;
The multiple DP iterations are
For each DP iteration ,
In a first DP cycle, each of the plurality of GPUs performs a data compression operation in parallel on one of the plurality of data blocks to generate a respective first compressed data block ;
In a second DP cycle, each of the plurality of GPUs forwards, via an interprocessor link, a respective first compressed data block to a GPU downstream in the logical ring for further processing; and each receive, via a corresponding interprocessor link or CCIX connection, from an upstream GPU in the logical ring for further processing a second compressed data block generated by the upstream GPU performing a data compression operation; death,
In the third DP cycle, each of the plurality of GPUs processes the first compressed data block processed by the current GPU and the second compressed data block processed by the corresponding upstream GPU and received from that GPU. performing a summation operation on the blocks to produce a first DP result;
In a fourth DP cycle, the plurality of GPUs each perform a data decompression operation on the first DP result, and the data blocks obtained by decompression are used for the next DP iteration. including
The summing operation pipelines the first compressed data block and the second compressed data block such that the bits in the bitmask field of one compressed data block are replaced by the bits in the bitmask field of the other compressed data block. A non-transitory machine-readable medium that is an operation that compares the bits of and outputs the result of the comparison. A computer program,
A computer program for implementing the method of any one of claims 1 to 3 when said computer program is executed by a processor.

Images