WO2025092447A1 - Parallelization strategy optimization method and system, device, and medium - Google Patents

Abstract

The present invention provides a parallelization strategy optimization method and system, a device, and a medium. The method comprises the following steps: replacing a scaling factor in a softmax function in a parallelization strategy with a maximum preset fixed value, to obtain an optimized softmax function; and performing exponential operation processing and sequence sum operation processing in parallel on the optimized softmax function, performing matrix multiplication processing after the exponential operation processing is completed, and correcting a matrix multiplication processing result by using a sequence sum operation processing result, so as to complete parallelization strategy optimization. The present application can improve the parallelization strategy operation efficiency of a large language model.

Description

A parallel strategy optimization method, system, device and medium

Cross-references

This application claims priority to Chinese patent application 202311456221.5 filed on November 3, 2023, the entire contents of which are incorporated herein by reference in their entirety.

Technical Field

The present invention belongs to the field of deep learning technology, and specifically relates to a parallelization strategy optimization method, system, device and medium.

Background Art

A large language model is a deep learning model trained with a large amount of text data that can generate natural language text or understand the meaning of language text. Large language models can handle a variety of natural language tasks, such as text classification, question answering, and dialogue, and are an important path to artificial intelligence.

As large language models become more and more important in various fields, the performance of large language model reasoning is crucial for large-scale large language model applications. A lot of work has been done to optimize large language model reasoning; as shown in Figure 2, a large language model composed of Transformer can be divided into two stages: Prefill and Decode. The main difference between the two stages is the size of the input matrix Q. The executed data flow is similar, both consisting of multiple Transformer layers, each of which can be divided into linear operations and attention mechanism operations, where the attention mechanism operation includes two general matrix multiplications and one softmax operation.

In the process of large language model inference, in order to improve the parallelism of calculations and reduce the overhead of reading and writing data, the existing work FlashAttention changed the original overall calculation method in the process of calculating the attention mechanism as shown in Figure 3(a). It chose to split the attention matrix and then perform partial softmax calculation on each part as shown in Figure 3(b). Therefore, the calculation process needs to complete the synchronization of current information and past information, and complete the update operation of existing results.

Currently, in the large language model inference calculation flow, the problem with the attention mechanism calculation pipeline is: the current common attention mechanism calculation pipeline adopts partial softmax operation, which uses partial matrix data to calculate the result. Because the data obtained from each part is different, it is necessary to synchronize information and update the results of the calculation of each part. This partial softmax synchronous update calculation will cause nearly 20% additional overhead.

Therefore, a parallelization strategy optimization method that can improve the operational efficiency of the parallelization strategy for large language models is desired.

Summary of the invention

In view of the problems existing in the prior art, the present invention provides a parallelization strategy optimization method, system, device and medium, which at least partially solve the problems existing in the prior art.

In a first aspect, an embodiment of the present disclosure provides a parallelization strategy optimization method, comprising the following steps:

Replace the scaling factor in the softmax function in the parallelization strategy with the maximum preset fixed value to obtain the optimized softmax function;

The optimized softmax function is processed by exponential operation and sequence and operation in parallel, wherein matrix multiplication operation is performed after the exponential operation is completed, and the result of the sequence and operation is used to correct the result of the matrix multiplication operation to complete the parallelization strategy optimization.

According to a specific implementation of the embodiment of the present disclosure, the softmax function in the parallelization strategy is:

Where x is the input data; is a scaling factor, which is a maximum preset fixed value; R is a real number; i is the number of input data; _xi is the i-th input data; e is the Napier constant; and _xd is the d-th input data.

According to a specific implementation of the embodiment of the present disclosure, the process of obtaining the maximum preset fixed value is:

Execute the model inference record preprocessing phase multiple times to get the input data of the softmax function;

Analyze the statistical distribution of the input data to obtain a maximum preset fixed value, which satisfies:

Most of the input data counted by this model do not satisfy: input data x _i >> maximum preset fixed value Or input data x _i ＜＜maximum preset fixed value situation.

According to a specific implementation of an embodiment of the present disclosure, most of the input data counted by the model is 99.99% of the input data.

According to a specific implementation of the embodiment of the present disclosure, the value range of the maximum preset fixed value is:

-100<Maximum preset fixed value

According to a specific implementation method of the embodiment of the present disclosure, after the matrix multiplication operation result is corrected using the sequence and operation processing result, an inner loop operation processing of optimizing the softmax function result and the feature matrix is performed.

According to a specific implementation of the embodiment of the present disclosure, the inner loop operation processing is to perform an optimized softmax function operation processing on the feature vector of each sample in the feature matrix to obtain the probability distribution of the sample.

According to a specific implementation manner of the embodiment of the present disclosure, during the inner loop operation processing, the input data of the optimized softmax function and the feature matrix are processed asynchronously separately.

According to a specific implementation manner of the embodiment of the present disclosure, there is an outer accumulation in the inner loop operation processing, and after all partial vectors are processed, the outer accumulation is performed.

According to a specific implementation of the embodiment of the present disclosure, the characteristic matrix is a V matrix, and the process of the inner loop operation processing is:

Where x is the input data; is a scaling factor, which is a maximum preset fixed value; R is a real number; is the data i-th dimension of the input data x ^(j) vector; _xi is the i-th dimension of the input vector; e is the Napier constant; _xd is the d-th dimension of the input vector; p is the number of input data x ^(j) vectors; j is the j-th vector of the input data; d/p is the number of dimensions of the x ^(j) vector; is the i-th dimension of the j-th column vector in the V matrix; For input data The result of scaling and exponential operation.

According to a specific implementation of the embodiment of the present disclosure, in the process of the inner loop operation processing, without loss of generality, it is assumed that each x _i like or When , the asynchronous softmax calculation of the vector x to which _xi belongs is terminated, and then the synchronous softmax method is used to recalculate the value of the optimized softmax function.

In a second aspect, an embodiment of the present disclosure provides a parallelization strategy optimization system, the system comprising:

The preprocessing unit is configured as

Replace the scaling factor in the softmax function in the parallelization strategy with the maximum preset fixed value to obtain the optimized softmax function;

Output unit, configured as

The optimized softmax function is processed by exponential operation and sequence and operation in parallel, wherein matrix multiplication operation is performed after the exponential operation is completed, and the result of the sequence and operation is used to correct the result of the matrix multiplication operation to complete the parallelization strategy optimization.

The present disclosure also provides an electronic device, the electronic device comprising:

at least one processor; and,

a memory communicatively connected to the at least one processor; wherein,

The memory stores instructions that can be executed by the at least one processor. When the instructions are executed by the at least one processor, the at least one processor executes the method for parallelization strategy optimization in the aforementioned first aspect or any implementation of the first aspect.

In a fourth aspect, an embodiment of the present disclosure further provides a non-transitory computer-readable storage medium, which stores computer instructions, and when the computer instructions are executed by at least one processor, the at least one processor executes the method for parallelization strategy optimization in the aforementioned first aspect or any implementation of the first aspect.

In the fifth aspect, the embodiments of the present disclosure also provide a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the method for parallelization strategy optimization in the aforementioned first aspect or any implementation of the first aspect.

Other optional features and technical effects of the embodiments of the present invention are partially described below, and partially can be understood by reading this document.

Compared with the prior art, the present invention has the following beneficial technical effects:

The present invention provides a parallelization strategy optimization method, system, device and medium, comprising the following steps: replacing the scaling factor in the softmax function in the parallelization strategy with a maximum preset fixed value to obtain an optimized softmax function; performing exponential operation processing and sequence sum operation processing on the optimized softmax function in parallel, wherein matrix multiplication operation processing is performed after the exponential operation processing is completed, and the matrix multiplication operation processing result is corrected using the sequence sum operation processing result to complete the parallelization strategy optimization; the application can improve the parallelization strategy operation efficiency of a large language model.

BRIEF DESCRIPTION OF THE DRAWINGS

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. The elements shown are not limited to the proportions shown in the accompanying drawings. The same or similar reference numerals in the accompanying drawings represent the same or similar elements, wherein:

FIG1 is a schematic diagram of a flow chart of a parallelization strategy optimization method according to an embodiment of the present disclosure;

FIG2 is a schematic diagram of a large language model reasoning calculation process in the prior art;

FIG3 is a schematic diagram showing a comparison of different softmax calculation methods in the prior art;

FIG4 is a schematic diagram of a method flow chart of a maximum preset fixed value according to an embodiment of the present disclosure;

FIG5 is a schematic diagram of a process of recalculating all partial vectors according to an embodiment of the present disclosure;

FIG6 is a diagram showing the effect of asynchronous softmax improvement in the Prefill stage according to an embodiment of the present disclosure;

FIG7 is a diagram showing the effect of asynchronous softmax improvement in the Decode stage according to an embodiment of the present disclosure;

FIG8 is a parallelization strategy optimization system according to an embodiment of the present disclosure; and

FIG. 9 is a diagram of a parallelization strategy optimization device according to an embodiment of the present disclosure.

DETAILED DESCRIPTION

The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

The following describes the embodiments of the present disclosure through specific examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all of the embodiments. The present disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present disclosure.

It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein may be embodied in a wide variety of forms, and any specific structure and/or function described herein is merely illustrative. Based on the present disclosure, it should be understood by those skilled in the art that an aspect described herein may be implemented independently of any other aspect, and two or more of these aspects may be combined in various ways. For example, any number of aspects described herein may be used to implement the device and/or practice the method. In addition, other structures and/or functionalities other than one or more of the aspects described herein may be used to implement this device and/or practice this method.

It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present disclosure. The drawings only show components related to the present disclosure rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

Additionally, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, one skilled in the art will appreciate that the aspects described may be practiced without these specific details.

In order to solve the problem that existing circuit simulation methods cannot take both simulation accuracy and simulation speed into consideration, the present invention proposes a method for modular circuit behavior simulation.

The method for modular circuit behavior simulation proposed in the present invention includes two parts: a software interface for hardware behavior modeling and a program for executing the simulation process, wherein the software interface for hardware behavior modeling takes tasks as basic units, and each task includes a start event and an end event.

Next, the parallelization strategy optimization method, system, device and medium according to the embodiments of the present disclosure will be described with reference to FIGS. 1 to 9 .

FIG1 shows a parallelization strategy optimization method 100 of the present embodiment. As shown in FIG1 , the method includes the following steps: At step S101 , the scaling factor in the softmax function in the parallelization strategy is replaced with a maximum preset fixed value to obtain an optimized softmax function.

In an embodiment of the present invention, the softmax function in the parallelization strategy is:

Where x is the input data; is a scaling factor, which is a maximum preset fixed value; R is a real number; i is the number of input data; _xi is the i-th input data; e is the Napier constant; and _xd is the d-th input data.

In an embodiment of the present invention, a method 200 for obtaining the maximum preset fixed value is shown in FIG4 , and includes the following steps:

At step S210, the model inference record preprocessing stage is executed multiple times to obtain input data of the softmax function.

Next, go to step S220.

In step S220, the statistical distribution of the input data is analyzed to obtain a maximum preset fixed value, which satisfies:

Most of the input data counted by this model do not satisfy: input data x _i >> maximum preset fixed value Or input data x _i ＜＜maximum preset fixed value situation.

In an embodiment of the present invention, the majority of input data counted by the model is 99.99% of the input data.

In an embodiment of the present invention, the maximum preset fixed value has a value range of:

-100<Maximum preset fixed value

Next, go to step S120.

At step S120, the optimized softmax function is subjected to exponential operation processing and sequence sum operation processing in parallel, wherein the exponential operation processing is completed and the matrix multiplication operation processing is performed, and the sequence sum operation processing result is used to multiply the matrix. Correct the result of matrix multiplication operation and complete the parallelization strategy optimization; It should be noted that in matrix multiplication, we usually use the sequence and operation results to correct the result of matrix multiplication operation. This process can be regarded as first performing a regular matrix multiplication, and then correcting the result of multiplication according to the sequence and the result of the operation; Specifically, suppose we have two matrices A and B, and we want to calculate A*B. After performing regular matrix multiplication, we will get a preliminary result C, and then we will correct C with the result of a sequence and operation to get the final multiplication result D; This correction process can effectively improve the accuracy and stability of matrix multiplication, especially when processing large-scale and complex data.

In an embodiment of the present invention, after the matrix multiplication operation result is corrected using the sequence and operation processing result, an inner loop operation processing of optimizing the softmax function result and the feature matrix is performed.

In an embodiment of the present invention, the inner loop operation processing is to perform an optimized softmax function operation processing on the feature vector of each sample in the feature matrix to obtain the probability distribution of the sample.

In an embodiment of the present invention, during the inner loop operation processing, the input data of the optimized softmax function and the feature matrix are processed asynchronously separately; the asynchronous processing is a processing method that does not need to wait for the processing to be completed before performing subsequent operations. The core logic of asynchronous processing is that it does not block the current thread to wait for the processing to be completed, but allows subsequent operations until other threads complete the processing and call back to notify this thread. This processing method is similar to SMS communication, and there is no need to maintain a waiting state after sending a message; for example, in programming, asynchronous processing can be used to process long-running operations, such as network requests or file IO operations, to improve the program's responsiveness and concurrency. In natural language processing, asynchronous processing can be used for computationally intensive tasks such as training language models to make full use of computing resources and improve training efficiency.

In an embodiment of the present invention, there is an outer accumulation in the inner loop operation processing, and the outer accumulation performs external accumulation processing after all partial vectors are processed; it should be noted that the outer accumulation usually refers to an accumulation operation on an external variable during a loop or iteration process, and this external variable is usually used to calculate a certain accumulation sum, or is used to calculate the sum of all loop iterations after the loop ends; in each iteration of the loop, the external variable will be accumulated with an internal variable in the loop, thereby gradually increasing the value of the external variable; when the loop ends, the value of the external variable is the accumulation sum of all loop iterations; the outer accumulation is usually used to count the number of times an event occurs, or to calculate the sum of a variable in the loop. This accumulation operation can easily calculate the sum of the results of all iterations in the loop, and can avoid repeated accumulation operations within the loop, thereby improving the efficiency and readability of the code.

In an embodiment of the present invention, the characteristic matrix is a V matrix, and the inner loop operation processing process is:

Where x is the input data; is a scaling factor, which is a maximum preset fixed value; R is a real number; is the data i-th dimension of the input data x ^(j) vector; _xi is the i-th dimension of the input vector; e is the Napier constant; _xd is the d-th dimension of the input vector; p is the number of input data x ^(j) vectors; j is the j-th vector of the input data; d/p is the number of dimensions of the x ^(j) vector; is the i-th dimension of the j-th column vector in the V matrix; For input data The result of scaling and exponential operation.

In the embodiment of the present invention, during the inner loop operation process, without loss of generality, it is assumed that each x _i like or When , the asynchronous softmax calculation of the vector x to which _xi belongs is terminated, and then the synchronous softmax method is used to recalculate the value of the optimized softmax function.

It should be noted that the purpose of optimizing the softmax function and the V matrix for inner loop operations is to obtain a set of probability distributions. The softmax function is a commonly used function that can map any real number to a value between [0,1], and the sum of these values is equal to 1, so it can be interpreted as a probability distribution, and the V matrix is usually a feature matrix, each row represents the feature vector of a sample; therefore, the inner loop operation of the softmax function and the V matrix can be understood as: performing a softmax function operation on the feature vector of each sample to obtain the probability distribution of the sample, and this probability distribution can be used to represent the probability of the sample belonging to each category, thereby providing a basis for subsequent tasks such as classification or clustering.

In an embodiment of the present invention, FIG. 5 shows an example of a parallelization strategy optimization method; a=-3, b=3 are preset, The two vectors x and y are calculated by Q·K ^T and divided into two local vectors; at the same time, the process from Q·K ^T to these local vectors is omitted. For each x _i , there is For the first partial vector of x, use and Process the first partial vector of x. There are two asynchronous threads, each thread performs corresponding calculations, respectively:

and

The two threads synchronize after processing all partial vectors and perform the final division operation. For y, the first partial vector is processed similarly, but Then both threads will be terminated and the first thread will recalculate all partial vectors based on the calculation results.

FIG5 shows the process of recalculating all partial vectors in this embodiment:

Figure 5(a) shows that each partial softmax result is processed separately without synchronous update, and Figure 5(b) shows that when overflow occurs, all partial softmax calculations need to be recalculated.

The experimental results in the embodiments of the present invention are:

In this embodiment, the optimization of the softmax function can also be called an asynchronous softmax solution, which can be applied to both the Prefill stage and the Decode stage. The proposed solution is tested against the most advanced attention implementation solution ^. The test results on GPU are shown in Figures 6 and 7. In the Prefill stage, the proposed scheme achieves an average speedup of 1.52 times and 1.19 times compared with xformers[5] and FlashAttetion2, respectively. In the Decode stage, the proposed scheme outperforms the xformers implementation of customized decoding, denoted as xformers-decoder in Figure 8, and is 2.02 times faster than the prior art FlashDecoding in the case of long context.

FIG8 shows a parallelization strategy optimization system 300 provided by the present invention. The system 300 includes: a preprocessing unit 310 and an output unit 320

A preprocessing unit 310 is configured to replace the scaling factor in the softmax function in the parallelization strategy with a maximum preset fixed value to obtain an optimized softmax function;

The output unit 320 is configured to perform exponential operation processing and sequence sum operation processing on the optimized softmax function in parallel, wherein matrix multiplication operation processing is performed after the exponential operation processing is completed, and the matrix multiplication operation processing result is corrected using the sequence sum operation processing result to complete the parallelization strategy optimization.

FIG9 shows a schematic diagram of an electronic device 1000 that can implement a method or implement an embodiment of the present invention, which may include more or fewer electronic devices than shown in the figure in some embodiments. In some embodiments, it can be implemented using a single or multiple electronic devices. In some embodiments, it can be implemented using cloud or distributed electronic devices.

As shown in FIG9 , the electronic device 1000 includes a processor 1001, which can perform various appropriate operations and processes according to the programs and/or data stored in the read-only memory (ROM) 1002 or the programs and/or data loaded from the storage part 1008 to the random access memory (RAM) 1003. The processor 1001 can be a multi-core processor or can include multiple processors. In some embodiments, the processor 1001 can include a general main processor and one or more special coprocessors, such as a central processing unit (CPU), a graphics processing unit (GPU), a neural network processor (NPU), a digital signal processor (DSP), etc. Various programs and data required for the operation of the electronic device 1000 are also stored in the RAM 1003. Processor 1001, ROM 1002 and RAM 1003 are connected to each other via a bus 1004. To the bus 1004, an input/output (I/O) interface 1005 is also connected.

The processor and the memory are used together to execute the program stored in the memory. When the program is executed by the computer, the methods, steps or functions described in the above embodiments can be implemented.

The following components are connected to the I/O interface 1005: an input section 1006 including a keyboard, a mouse, a touch screen, etc.; an output section 1007 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN card, a modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I/O interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1010 as needed, so that a computer program read therefrom is installed into the storage section 1008 as needed. FIG. 9 only schematically shows some components, which does not mean that the computer system 1000 only includes the components shown in FIG. 9.

The systems, devices, modules or units described in the above embodiments may be implemented by a computer or its associated components. The computer may be, for example, a mobile terminal, a smart phone, a personal computer, a laptop computer, a vehicle-mounted human-computer interaction device, a personal digital assistant, a media player, a navigation device, a game console, a tablet computer, a wearable device, a smart TV, an Internet of Things system, a smart home, an industrial computer, a server or a combination thereof.

Although not shown, in an embodiment of the present invention, a storage medium is provided, the storage medium storing a computer program, the computer program being configured to execute any file difference-based compilation method of the embodiments of the present invention when executed.

Storage media in embodiments of the present invention include permanent and non-permanent, removable and non-removable items that can be used to store information by any method or technology. Examples of storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

The methods, programs, systems, devices, etc. of the embodiments of the present invention may be executed or implemented in a single or multiple networked computers, or may be practiced in a distributed computing environment. In the embodiments of this specification, in these distributed computing environments, tasks may be performed by remote processing devices connected via a communication network.

Those skilled in the art should understand that the embodiments of the present specification may be provided as methods, systems or computer program products. Therefore, those skilled in the art may imagine that the functional modules/units or controllers and related method steps described in the above embodiments may be implemented in software, hardware or a combination of software/hardware.

Unless explicitly stated, the actions or steps of the methods, programs, and embodiments of the present invention do not have to be performed in a specific order and can still achieve the desired results. In some implementations, multitasking and parallel processing are also possible or may be advantageous.

In this article, multiple embodiments of the present invention are described, but for the sake of brevity, the description of each embodiment is not exhaustive, and the same or similar features or parts between the embodiments may be omitted. In this article, "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" are meant to be applicable to at least one embodiment or example according to the present invention, but not all embodiments. The above terms do not necessarily mean to refer to the same embodiment or example. In the absence of contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples.

The exemplary systems and methods of the present invention have been specifically shown and described with reference to the above embodiments, which are merely examples of the best modes for implementing the present systems and methods. It will be appreciated by those skilled in the art that various changes may be made to the embodiments of the systems and methods described herein when implementing the present systems and/or methods without departing from the spirit and scope of the present invention as defined in the appended claims.

Claims (15)

A parallel strategy optimization method, characterized in that it comprises the following steps: Replace the scaling factor in the softmax function in the parallelization strategy with the maximum preset fixed value to obtain the optimized softmax function; The optimized softmax function is processed by exponential operation and sequence and operation in parallel, wherein matrix multiplication operation is performed after the exponential operation is completed, and the result of the sequence and operation is used to correct the result of the matrix multiplication operation to complete the parallelization strategy optimization. The parallelization strategy optimization method according to claim 1 is characterized in that the softmax function in the parallelization strategy is:
Where x is the input data; is a scaling factor, which is a maximum preset fixed value; R is a real number; i is the number of input data; _xi is the i-th input data; e is the Napier constant; and _xd is the d-th input data. The parallelization strategy optimization method according to claim 1 is characterized in that the process of obtaining the maximum preset fixed value is: Execute the model inference record preprocessing phase multiple times to get the input data of the softmax function; Analyze the statistical distribution of the input data to obtain a maximum preset fixed value, which satisfies: Most of the input data counted by this model do not satisfy: input data x _i >> maximum preset fixed value Or input data x _i ＜＜maximum preset fixed value situation. The parallelization strategy optimization method according to claim 3 is characterized in that the majority of the input data counted by the model is 99.99% of the input data. The parallelization strategy optimization method according to claim 1 is characterized in that the value range of the maximum preset fixed value is:
-100<Maximum preset fixed value The parallelization strategy optimization method according to claim 1 is characterized in that after the matrix multiplication operation processing result is corrected using the sequence and operation processing result, an inner loop operation processing of optimizing the softmax function result and the feature matrix is performed. The parallelization strategy optimization method according to claim 6 is characterized in that the inner loop operation processing is to optimize the softmax function operation processing of the feature vector of each sample in the feature matrix to obtain the sample The probability distribution of . The parallelization strategy optimization method according to claim 6 is characterized in that, during the inner loop operation processing, the input data of the optimized softmax function and the feature matrix are processed asynchronously separately. The parallelization strategy optimization method according to claim 6 is characterized in that there is an outer accumulation in the inner loop operation processing, and the outer accumulation is performed after all partial vectors are processed. The parallelization strategy optimization method according to claim 6 is characterized in that the characteristic matrix is a V matrix, and the process of the inner loop operation processing is:
Where x is the input data; is a scaling factor, which is a maximum preset fixed value; R is a real number; is the data i-th dimension of the input data x ^(j) vector; _xi is the i-th dimension of the input vector; e is the Napier constant; _xd is the d-th dimension of the input vector; p is the number of input data x ^(j) vectors; j is the j-th vector of the input data; d/p is the number of dimensions of the x ^(j) vector; is the i-th dimension of the j-th column vector in the V matrix; For input data The result of scaling and exponential operation. The parallelization strategy optimization method according to claim 10 is characterized in that, in the process of the inner loop operation processing, without loss of generality, it is assumed that each x _i like or When , the asynchronous softmax calculation of the vector x to which _xi belongs is terminated, and then the synchronous softmax method is used to recalculate the value of the optimized softmax function. A parallelization strategy optimization system, characterized in that the parallelization strategy optimization method according to any one of claims 1 to 11 comprises: The preprocessing unit is configured as Replace the scaling factor in the softmax function in the parallelization strategy with the maximum preset fixed value to obtain the optimized softmax function; Output unit, configured as The optimized softmax function is processed by exponential operation and sequence and operation in parallel, wherein matrix multiplication operation is performed after the exponential operation is completed, and the result of the sequence and operation is used to correct the result of the matrix multiplication operation to complete the parallelization strategy optimization. A computer device, the electronic device comprising: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and when the instructions are executed by the at least one processor, the at least one processor executes the parallelization strategy optimization method according to any one of claims 1 to 11. A non-transitory computer-readable storage medium storing computer instructions, which, when executed by at least one processor, cause the at least one processor to perform the parallelization strategy optimization method according to any one of claims 1 to 11. A computer program product, comprising a computing program stored on a non-transitory computer-readable storage medium, wherein the computer program comprises program instructions, and when the program instructions are executed by a computer, the computer is caused to execute the parallelization strategy optimization method as described in any one of claims 1 to 11.