TWI845081B - Graphics processor - Google Patents

Abstract

A graphics processor includes a plurality of stream multiprocessors. Each of the stream multiprocessors includes N stream processors and a tensor processing module. The N stream processors are configured to operate in an SIMT architecture, wherein each of the N stream processors includes a product-accumulation unit. The tensor processing module is connected with the N product-accumulation units, and is configured to arrange the N product-accumulation units as a systolic array that performs a matrix multiplication.

Description

Graphics processor

The present invention relates to the technical field of graphics processors, and in particular to a graphics processor that implements a pulse array.

The application of neural networks is growing rapidly. With the focus on information security and the need for real-time applications, the application of artificial intelligence has spread from the cloud to the terminal, which has also prompted many hardware accelerators designed to accelerate neural network operations. Generally speaking, matrix multiplication is the basic operation in most network layers. Therefore, if the operation of matrix multiplication can be accelerated, the operation of neural networks can be accelerated. For this reason, major manufacturers such as NVIDIA and Google have proposed hardware accelerators for accelerating neural network operations.

NVIDIA has added Tensor Core to the Graphics Processing Unit (GPU) of the Volta architecture. It is an independent hardware unit that can perform a multiplication operation of two 4x4 matrices and then perform an accumulation operation with a 4x4 matrix to complete a Fused Multiply-Add (FMA). Tensor Core and Cuda Core in the GPU are different computing units, so they can be executed simultaneously to improve instruction level parallelism.

The TPU (Tensor Processing Unit) developed by Google is a chip plugged into the PCIe bus. It is used in cloud servers to handle the large amount of computing required for artificial intelligence applications, rather than hardware designed inside the GPU like NVIDIA's Tensor core. Google TPU has launched many versions so far, but the computing units all use the systolic array hardware architecture. TPU cannot run alone and needs to be controlled by the CPU through the PCIe bus to send TPU CISC-like instructions.

Whether it is NVIDIA Tensor Core or Google TPU, they both require huge power consumption to meet the demand for computing performance, and are not considered for terminal application design. However, on terminal devices with limited area and power consumption, the cost of adding a matrix computing unit is very high. Therefore, how to improve the performance of matrix computing without increasing hardware costs too much is the current technical bottleneck.

One of the purposes of the present invention is to provide a hardware architecture for accelerating matrix multiplication operations using a graphics processor.

4。張量處理模組連接該等N個乘積累加運算單元，且經組態以將該等N個乘積累加運算單元配置為執行矩陣乘法運算的脈動陣列。 To achieve the above-mentioned object, the present invention provides a graphics processor, which includes a plurality of stream multiprocessors. Each of the stream multiprocessors includes N stream processors and a tensor operation module. The N stream processors are configured to operate in a single instruction to multiple threads (Single Instruction Multiple Threads, SIMT) architecture, wherein each of the N stream processors includes a multiply-accumulate operation unit, wherein N is a positive integer and N

4. The tensor processing module is connected to the N product-accumulation operation units and is configured to configure the N product-accumulation operation units as a pulse array for performing matrix multiplication operations.

In one embodiment of the present invention, the dimension (D) of the pulse array conforms to the following formula,

2。 Where N=2 ⁿ , n is a positive integer and n

2.

In one embodiment of the present invention, N=4, the four multiply-accumulate operation units are MAC ₀ , MAC ₁ , MAC ₂ , and MAC ₃ , which belong to four of the N stream processors in sequence, and the four multiply-accumulate operation units are configured in the following manner to form the pulse array:

4時，該等N個乘積累加運算單元被配置為下列方式以形成該脈動陣列：

In one embodiment of the present invention, the N multiply-accumulate units are MAC ₀ , MAC ₁ , ..., MAC _(N-1) and belong to the N stream processors in sequence, wherein in response to n being an even number and n being

4, the N multiplication and accumulation units are configured in the following manner to form the pulse array:

In one embodiment of the present invention, N=8, the eight multiply-accumulate operation units are MAC ₀ , MAC ₁ , MAC ₂ , ..., MAC ₇ , which belong to eight of the N stream processors in sequence, and the eight multiply-accumulate operation units are configured in the following manner to form the pulse array:

5時，該等N個乘積累加運算單元被配置為下列方式以形成該脈動陣列：

In one embodiment of the present invention, the N multiply-accumulate units are MAC ₀ , MAC ₁ , ..., MAC _(N-1) and belong to the N stream processors in sequence, wherein in response to n being an odd number and n being

5, the N product accumulation units are configured in the following manner to form the pulse array:

In one embodiment of the present invention, the tensor processing module includes a first buffer unit and a second buffer unit, wherein the first buffer unit is configured to transmit data to the multiplication and accumulation unit corresponding to the row in the pulse array, and the second buffer unit is configured to transmit data to the multiplication and accumulation unit corresponding to the column in the pulse array.

In one embodiment of the present invention, each of the streaming multiprocessors further includes a local memory and a read/store module. The local memory is connected to the tensor processing module and is configured to store data. The read/store module is connected to the local memory and the N streaming processors and is configured to read data from the local memory or store data in the local memory.

In one embodiment of the present invention, in response to performing a first operation of matrix multiplication, the tensor processing module accesses data from the local memory and transmits the data to the multiplication and accumulation unit in the pulse array to perform the first operation.

In one embodiment of the present invention, in response to performing a second operation that is not a matrix multiplication, the read/store module accesses data from the local memory and transmits the data to the N stream processors to perform the second operation.

The graphics processor proposed in the present invention configures the multiplication and accumulation unit in the stream processor as a pulse array for performing matrix multiplication operations. In addition to some changes in the hardware architecture design, the integration of the computing platform and the optimization of the neural network at the software layer are also considered. Therefore, without increasing the cost of hardware too much, the graphics processor of the present invention can effectively accelerate the operation of matrix multiplication.

100: Graphics processor

110: Internet module

120: Streaming multiprocessor

121: Local memory

123: Read/save module

125: Stream Processor

127: Tensor processing module

1271: First temporary unit

1273: Second temporary unit

129: Allocation module

130: Task Scheduling Module

140:Memory

PE ₀ ~PE _(N-1) : Stream Processor

MAC ₀ ~MAC _(N-1) : Multiplication and Accumulation Unit

L ₀ ~L _(N-1) : Channel

Aout[0], Aout[1], Aout[2], Aout[3], Bout[0], Bout[1], Bout[2], Bout[3]: data

Figure 1A is a block system schematic diagram of a graphics processor according to a preferred embodiment of the present invention.

Figure 1B is a schematic diagram of a pulse array implemented by a graphics processor according to a preferred embodiment of the present invention.

Figure 2 is a schematic diagram of a pulse array realized when n=5 according to a preferred embodiment of the present invention.

Figure 3 is a timing diagram of the tensor processing module transmitting data according to a preferred embodiment of the present invention.

In order to make the above and other purposes, features and advantages of the present invention more clearly understood, the following will specifically cite the preferred embodiments of the present invention and provide a detailed description with reference to the attached drawings.

Please refer to FIG. 1A, which is a block diagram of a graphics processor 100 according to a preferred embodiment of the present invention. The graphics processor 100 is a single instruction multiple threads (SIMT) architecture, which includes an interconnection network module 110, a plurality of streaming multiprocessors (SMs) 120, a task scheduling module 130, and a memory 140. The interconnection network module 110 is electrically connected to each of the streaming multiprocessors 120, the task scheduling module 130, and the memory 140, and is configured to transmit data between these components. The streaming multiprocessor 120 is configured to perform calculations and execute instructions. The task scheduling module 130 is configured to communicate with an external central processor (not shown), receive tasks assigned from the central processor, and schedule the tasks to the streaming multiprocessor 120 for execution.

Each streaming multiprocessor 120 includes a local memory 121, a read/storage module 123, N streaming processors (PE) 125, a tensor processing module 127, and a dispatching module 129. The local memory 121 is connected to the memory 140 to store data to be calculated and data that has been calculated. The read/storage module 123 is connected to the local memory 121 and the dispatching module 129, and is configured to access the data to be calculated from the local memory 121 and transmit the data to the dispatching module 129. The dispatching module 129 then distributes the data to the corresponding streaming processor 125 for calculation according to the type of the data. The N stream processors 125 each include a respective multiply-accumulate unit (MAC) MAC ₀ to MAC _N for performing multiply-accumulate operations. Specifically, the N stream processors 125 of the stream multiprocessor 120 perform operations based on a SIMT architecture, that is, for the same instruction, the N stream processors 125 perform operations simultaneously in thread units, thereby achieving a parallel processing effect.

In the graphics processor 100 of the present invention, each stream multiprocessor 120 further includes a tensor processing module 127, which is connected to N multiplication and accumulation units MAC ₀ ~MAC _N in the N stream processors 125. The tensor processing module 127 is configured to configure the N multiplication and accumulation units MAC ₀ ~MAC _N as a systolic array for performing matrix multiplication operations, as shown in FIG. 1B, which is a schematic diagram of the graphics processor 100 implementing a systolic array according to a preferred embodiment of the present invention. In the example of FIG. 1B, N=16. The concept of pulse array is to make data flow in the array of operation units, reduce the number of accesses, make the structure more complete, make the wiring more uniform, and increase the flow frequency. The pulse array itself is just a structure with data flow. Different data will flow and different flow directions depending on different applications.

Although FIG. 1B shows an array (MAC ₀ ~ MAC ₁₅ ), in reality, the tensor processing module 127 does not really arrange the N multiplication and accumulation units MAC ₀ ~ MAC _N into an actual pulse array. Instead, it transmits data at different timings to allow these multiplication and accumulation units MAC ₀ ~ MAC _N to perform pulse array operations. For example, L ₀ represents the channel of the multiplication and accumulation unit MAC ₀ connected to the stream processor PE ₀ , and L ₁ represents the channel of the multiplication and accumulation unit MAC ₁ connected to the stream processor PE ₁ , and so on. In this way, the tensor processing module 127 can transmit the multiplication and accumulation units MAC 0 ~ MAC N through the channels L ₀ ~ L ₁₅ passes the data to the multiplication and accumulation units MAC ₀ ~ MAC ₁₅ for operation. However, for ease of understanding, the following is presented in the form of a matrix.

In one embodiment of the present invention, the dimension (Dimension, D) of the pulse array conforms to the following formula,

4，及n為正整數且n

2。 Where N is the number of MPUs in the streaming multiprocessor 120 (i.e., the number of streaming processors 125), N=2 ⁿ , N is a positive integer and N

4, and n is a positive integer and n

2.

Specifically, the pulse array implemented by the present invention changes the dimension of the pulse array that can be implemented according to the different number of product-accumulation operation units in each stream multiprocessor 120 (that is, different N(n)). Please refer to Figure 2, which is a schematic diagram of the pulse array implemented when n=5 according to a preferred embodiment of the present invention. When n=5 (that is, the number of product-accumulation operation units N is 32), n is an odd number. According to the above formula, the pulse array implemented by the graphics processor of the present invention is a 4x4+4x4 matrix, wherein the direction of the arrow in the matrix represents the direction of data transmission when performing matrix multiplication operation. The following will introduce the positions of the pulse arrays corresponding to each product-accumulation unit MAC ₀ ~MAC _N under different N(n) conditions.

In the present embodiment, the minimum dimension of the pulse array is 2x2, that is, n=2 (that is, the number N of the multiplication and accumulation operation units is 4). In the case of n=2 (N=4), each stream multiprocessor 120 may include four multiplication and accumulation operation units, for example, MAC ₀ , MAC ₁ , MAC ₂ , and MAC ₃ , which respectively belong to four stream processors PE ₀ to PE ₃ connected to the sequential channels L ₀ to L ₃ , and these four multiplication and accumulation operation units are configured in the following manner to form the pulse array:

4的情況下，每個串流多處理器120包含的乘積累加運算單元可例如為MAC₀、MAC₁、...、MAC_(N-1)共N個，分別屬於對應連接於依序的通道L₀~L_(N-1)的N個串流處理器PE₀~PE_(N-1)，N個乘積累加運算單元被配置為下列方式以形成該脈動陣列：

When n is even and n

4, each stream multiprocessor 120 may include N multiplication and accumulation units, for example, MAC ₀ , MAC ₁ , ..., MAC _(N-1) , which belong to N stream processors PE ₀ ~PE ( _N-1) connected to the sequential channels L ₀ ~L _(N-1) , respectively. The N multiplication and accumulation units are configured in the following manner to form the pulse array:

In the case of n=3 (N=8), each stream multiprocessor 120 may include eight multiplication and accumulation units, for example, MAC ₀ , MAC ₁ , MAC ₂ , ..., MAC ₇ , which belong to eight stream processors PE ₀ -PE ₇ connected to the sequential channels L ₀ -L ₇ , respectively, and these eight multiplication and accumulation units are configured in the following manner to form the pulse array:

5的情況下，每個串流多處理器120包含的乘積累加運算單元可例如為MAC₀、MAC₁、...、MAC_(N-1)共N個，分別屬於對應連接於依序的通道L₀~L_(N-1)的N個串流處理器PE₀~PE_(N-1)，N個乘積累加運算單元被配置為下列方式以形成該脈動陣列：

When n is odd and n

5, each stream multiprocessor 120 may include N multiplication and accumulation units, for example, MAC ₀ , MAC ₁ , ..., MAC _(N-1) , which belong to N stream processors PE ₀ ~PE ( _N-1) connected to the sequential channels L ₀ ~L _(N-1) , respectively. The N multiplication and accumulation units are configured in the following manner to form the pulse array:

According to the above-mentioned implementation methods, the graphics processor 100 proposed by the present invention considers various executable pulse array methods under different numbers of multiplication and accumulation units, so it only needs to use the built-in multiplication and accumulation unit of the stream processor 125 itself, without setting up additional hardware (such as Tensor core or TPU), and the tensor processing module 127 can use the data scheduling to realize the pulse array of accelerated matrix multiplication operation.

Please refer to FIG. 3, which is a timing diagram of the tensor processing module 127 transmitting data according to a preferred embodiment of the present invention. In this example, n=4. As shown in FIG. 3, the tensor processing module 127 includes a first buffer unit 1271 and a second buffer unit 1273. The tensor processing module 127 can retrieve the data of the matrix to be multiplied (e.g., the first matrix and the second matrix) from the local memory 121 and store them in the first buffer unit 1271 (e.g., the data of the first matrix) and the second buffer unit 1273 (e.g., the data of the second matrix) according to the corresponding row and column data. The first buffer unit 1271 is configured to transmit the data of the first matrix (e.g., Aout[0], Aout[1], Aout[2], Aout[3]) to the multiplication and accumulation unit corresponding to the row in the pulse array. The second buffer unit 1273 is configured to transmit the data of the second matrix (e.g., Bbout[0], Bout[1], Bout[2], Bout[3]) to the multiplication and accumulation unit corresponding to the row in the pulse array.

Initially, the tensor processing module 127 accesses the matrix data to be multiplied from the local memory 121 to the first buffer unit 1271 and the second buffer unit 1273. Then, the first buffer unit 1271 and the second buffer unit 1273 respectively transmit the matrix data to be operated corresponding to the first row (e.g., Aout[0]) and the first column (e.g., Bout[0]) through the channel _L0 to the multiplication and accumulation unit _MAC1 for operation. After the operation is completed, Aout[0] and Bout[1] are transmitted to the multiplication and accumulation unit MAC ₄ through channel L ₄ for operation, and Bout[0] and Aout[1] are transmitted to the multiplication and accumulation unit MAC ₁ through channel L ₁ for operation. After the operation is completed, Aout[0] and Bout[2] are then passed to the multiplication and accumulation unit MAC ₈ through channel L ₈ for operation, Bout[0] and Aout[2] are passed to the multiplication and accumulation unit MAC ₂ through channel L ₂ for operation, and Aout[1] and Bout[1] are passed to the multiplication and accumulation unit MAC ₅ through channel L ₅ for operation, and so on, until all the matrix data to be operated are completed. The matrix multiplication operation can be completed.

In some embodiments, the number of buffer units that the tensor processing module 127 may include is not limited to two, and may include more than two buffer units depending on the size of the matrix data to be multiplied.

Furthermore, the tensor processing module 127 in this embodiment can first access the matrix data to be operated on the matrix multiplication from the local memory 121, and temporarily store the matrix data in the first buffer unit 1271 and the second buffer unit 1273. When the matrix multiplication is to be performed, the data can be directly accessed from the first buffer unit 1271 and the second buffer unit 1273 and provided to the corresponding product-accumulation operation unit through the corresponding channel, without the need to access the data from the local memory 121 through the read/storage module 123 and then the allocation module 129 performs the operation of allocating the data. Therefore, the time for accessing data from the external memory can be greatly reduced, thereby further reducing the time for performing matrix multiplication operations. In other words, in response to performing matrix multiplication operations, the streaming multiprocessor 120 can use the tensor processing module 127 to directly access data from the local memory 121 and store it in the first buffer unit 1271 and the second buffer unit 1273, and then transmit the data from the first buffer unit 1271 and the second buffer unit 1273 to the pulse through the channel. The multiplication and accumulation unit in the array performs the operation; when responding to the execution of non-matrix multiplication operations, the streaming multiprocessor 120 uses the read/store module 123 to access data from the local memory 121 and transmit the data to the allocation module 129, and then the allocation module 129 distributes the data to N streaming processors 125 to perform the operation.

The graphics processor proposed in the present invention configures the built-in multiplication and accumulation unit of the stream processor itself as a pulse array for performing matrix multiplication operations. In addition to some changes in the hardware architecture design, the integration of the computing platform and the optimization of the neural network at the software layer are also considered. Therefore, without increasing the cost of hardware too much, the graphics processor of the present invention can effectively accelerate the operation of matrix multiplication.

Although the present invention has been disclosed with preferred embodiments, it is not intended to limit the present invention. Anyone skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of the patent application attached hereto.

100: Graphics processor

110: Internet module

120: Streaming multiprocessor

121: Local memory

123: Read/save module

125: Stream Processor

127: Tensor processing module

129: Allocation module

130: Task Scheduling Module

140:Memory

PE ₀ ~PE _(N-1) : Stream Processor

MAC ₀ ~MAC _(N-1) : Multiplication and Accumulation Unit

L ₀ ~L _(N-1) : Channel

Claims (9)

A graphics processor includes: a plurality of stream multiprocessors, each of the stream multiprocessors includes: N stream processors configured to operate with a single instruction corresponding to a plurality of execution threads, wherein each of the N stream processors includes a multiply-accumulate operation unit, wherein N is a positive integer and N

4; and a tensor processing module connected to the N product-accumulation operation units and configured to configure the N product-accumulation operation units into a pulse array for performing matrix multiplication operations, wherein the dimension (D) of the pulse array meets the following formula,

Where N=2 ⁿ , n is a positive integer and n

2. The graphics processor as claimed in claim 1, wherein N=4, the four multiply-accumulate operation units are MAC ₀ , MAC ₁ , MAC ₂ , and MAC ₃ , which belong to four of the N stream processors in sequence, and the four multiply-accumulate operation units are configured in the following manner to form the pulse array:

The graphics processor of claim 1, wherein the N multiply-accumulate units are MAC ₀ , MAC ₁ , ..., MAC _(N-1) and belong to the N stream processors in sequence, wherein in response to n being an even number and n

4, the N multiplication and accumulation units are configured in the following manner to form the pulse array:

The graphics processor as claimed in claim 1, wherein N=8, the eight multiply-accumulate operation units are MAC ₀ , MAC ₁ , MAC ₂ , . . . , MAC ₇ , which belong to eight of the N stream processors in sequence, and the eight multiply-accumulate operation units are configured in the following manner to form the pulse array:

The graphics processor of claim 1, wherein the N multiply-accumulate units are MAC ₀ , MAC ₁ , ..., MAC _(N-1) and belong to the N stream processors in sequence, wherein in response to n being an odd number and n

5, the N product accumulation units are configured in the following manner to form the pulse array:

A graphics processor as described in claim 1, wherein the size of the pulse array is AxB, wherein the tensor processing module includes a first buffer unit and a second buffer unit, wherein the first buffer unit is configured to transmit data to a corresponding row of the B row of the pulse array, and the second buffer unit is configured to transmit data to a corresponding row of the A row of the pulse array. A graphics processor as described in claim 1, wherein each of the streaming multiprocessors further comprises: a local memory connected to the tensor processing module and configured to store data; and a read/store module connected to the local memory and the N streaming processors and configured to read data from the local memory or store data in the local memory. A graphics processor as described in claim 7, wherein in response to performing a first operation of matrix multiplication, the tensor processing module accesses data from the local memory and transmits the data to the N multiply-accumulate operation units in the pulse array to perform the first operation. A graphics processor as described in claim 7, wherein in response to performing a second operation that is not a matrix multiplication, the read/store module accesses data from the local memory and transmits the data to the N stream processors to perform the second operation.

Images