CN113449842B - A distributed automatic differentiation method and related device - Google Patents

Abstract

The embodiment of the application provides a distributed automatic differentiation method and a related device, wherein the method comprises the following steps: acquiring a forward propagation calculation graph of the neural network model; based on that the model parameters of the neural network model stored by the computing nodes are the same as the model parameters stored by at least one computing node except the computing nodes in the distributed computing system, inserting a target operator between the input end of the model parameters of the computing nodes and an operator for computing the model parameters so as to obtain a target computational graph; the target operator is used for transmitting the model parameters through, and the derivative algorithm is used for summing the derivatives of the model parameters of all the calculation nodes with the same model parameters; a back propagation computation graph is generated based on a derivative algorithm of the operators in the target computation graph. By implementing the embodiment of the application, the derivation of the back propagation process of the user intervention model can be avoided, the development efficiency is improved, and the development difficulty is reduced.

Description

A distributed automatic differentiation method and related device

Technical Field

The present application relates to the field of artificial intelligence (AI), and in particular to a distributed automatic differentiation method and related devices.

Background Art

With the development of deep learning, neural network models are becoming larger and larger, making it impossible for a single machine to complete model storage and calculation, and a distributed computing system is needed. A distributed computing system includes multiple computing nodes that can perform parallel computing. The training of a neural network model involves two computing processes: forward propagation and backpropagation. Forward propagation refers to the sequence of the neural network from the input layer to the output layer, sequentially calculating and storing the intermediate variables (including output) of each layer of the neural network. Backpropagation refers to the sequence of the neural network from the output layer to the input layer, sequentially calculating and storing the intermediate variables of each layer of the neural network and the derivatives (or gradients) of the parameters. There are usually several ways to train neural network models using distributed computing systems: data parallelism, model parallelism, and hybrid parallelism.

The basic idea of data parallelism is that different computing nodes have the same model parameters to be trained (or model copies), but each computing node is responsible for a different part of the training data. Therefore, it is suitable for scenarios with large sample data volumes and small training models.

Model parallelism means that different computing nodes have the same sample data to be trained (or data copies), but each computing node is responsible for a different part of the neural network model, and the model parameters can be different. Therefore, it is suitable for scenarios where the training model is large but the amount of sample data is small.

As sample data sets and models become larger and larger, traditional data parallelism or model parallelism will not be applicable due to the memory constraints of a single node, and a hybrid parallel approach is needed to solve the problem. The hybrid parallel approach can be seen as a combination of data parallelism and model parallelism. The difference is that the forward propagation calculation process of the hybrid parallel approach may involve communication operators (such as all-reduce, all-gather, send/receive, etc.).

In the implementation of back-propagation calculation, for networks that do not involve cross-node communication, the industry usually uses automatic differentiation technology to perform automatic differentiation (AD) on all operators from the input layer to the output layer to obtain the derivatives of the parameters. However, for the hybrid parallel method, if the forward propagation process involves communication operators, in the back propagation, along the order from the output layer to the input layer, automatic differentiation is performed for the part where there is no communication operator. The part involving communication operators requires developers to manually deduce the derivation logic and manually insert related operators to transfer derivatives across nodes, which is time-consuming and labor-intensive, and the correctness cannot be strictly guaranteed, resulting in low development efficiency and difficulty.

Summary of the invention

The embodiments of the present application provide a distributed automatic differentiation method and related devices, which can automatically process the derivation of various distributed networks such as data parallelism, model parallelism and hybrid parallelism in back propagation, avoid user intervention in the derivation, thereby improving development efficiency and reducing development difficulty.

In a first aspect, an embodiment of the present application provides a distributed automatic differentiation method, which is applied to a computing node in a distributed computing system, including: obtaining a forward propagation computation graph of a neural network model; wherein the forward propagation computation graph includes multiple operators, the multiple operators including local operators and communication operators, the local operators representing operators that can complete calculations in the computing node, and the communication operators representing operators that rely on communication between the computing node and at least one computing node other than the computing node in the distributed computing system to complete calculations; based on the model parameters of the neural network model stored in the computing node being the same as the model parameters stored in at least one computing node other than the computing node in the distributed computing system, a target operator is inserted between the input end of the model parameters of the computing node and the operator used to calculate the model parameters to obtain a target computation graph; wherein the target operator is used to transmit the model parameters, and the derivative algorithm of the target operator is used to sum the derivatives of the model parameters of all computing nodes with the same model parameters; based on the derivative algorithms of some or all operators in the target computation graph, a back propagation computation graph of the neural network model is generated.

Among them, an operator represents a basic calculation method, and each operator corresponds to a derivative algorithm in back propagation, that is, the derivative algorithm can be obtained by differentiating the operator. For each derivative algorithm itself, the derivative algorithm may be a basic calculation method or a combination of multiple basic calculation methods.

Among them, the target operator can also be called the mirror operator. After inserting the mirror in the forward propagation calculation graph, the target calculation graph is formed. The input of the mirror operator is the model parameter, and the output of the mirror operator is used as the input of the operator originally used to calculate the parameter. Since the role of the mirror operator in forward propagation is to pass through the model parameters, the output of the mirror operator is also the model parameter, that is, the model parameter will eventually be input into the operator originally used to calculate the parameter.

It can be seen that the embodiment of the present application designs a set of general distributed differentiation solutions. Developers only need to implement a distributed forward propagation network, without paying attention to how the distributed network backpropagation is carried out, and without paying attention to which computing nodes the model parameters are mirrored (i.e., there are model copies). The computing nodes of the distributed computing system can automatically calculate the derivative algorithm of the communication operator, and call the derivative algorithm of the communication operator to complete the transmission (communication) of the derivative results across nodes, and automatically implement the insertion of the mirror operator, the generation and output of the backpropagation calculation graph, thereby avoiding the drawbacks of the prior art that requires users to intervene in the derivation, saving manpower and material costs, improving development efficiency, and reducing development difficulty. The present application solution is suitable for solving the current mainstream data parallelism, model parallelism, hybrid parallelism and other distributed network automatic differentiation problems, and improving the user experience.

Based on the first aspect, in a possible embodiment, the back propagation calculation graph of the neural network model is generated based on the derivative algorithm of some or all operators in the target calculation graph, including: obtaining the derivative algorithm of the communication operator, the derivative algorithm of the local operator and the derivative algorithm of the target operator in the target calculation graph; generating the back propagation calculation graph according to the derivative algorithm of the communication operator, the derivative algorithm of the local operator and the derivative algorithm of the target operator.

Specifically, the derivative algorithm of each local operator/communication operator/possible mirror operator can be called from the database in sequence from the output layer to the input layer of the neural network model to automatically generate a complete back-propagation calculation graph.

The derivative algorithm of the local operator may be pre-stored in local storage.

Based on the first aspect, in a possible embodiment, the derivative algorithm of the local operator is stored in the local storage of the computing node; the method also includes: performing a derivative operation on the communication operator in the forward propagation computation graph to obtain the derivative algorithm of the communication operator.

That is to say, the computing nodes of the distributed computing system can automatically calculate the derivative algorithm of the communication operator, and call the derivative algorithm of the communication operator to complete the transmission (communication) of the derivative results across nodes, and automatically realize the insertion of the mirror operator, the generation and output of the back propagation calculation graph, thereby avoiding the disadvantages of the existing technology that requires users to intervene in the derivation, improving development efficiency and reducing development difficulty.

Based on the first aspect, in a possible embodiment, the method also includes: determining the coefficient (sens) of the input data of the back propagation calculation graph by identifying whether the output layer of the neural network model is repeatedly calculated by two or more computing nodes in the distributed computing system (i.e., identifying whether the output layer is repeatedly calculated in the forward propagation calculation graph).

Based on the first aspect, in a possible embodiment, the sens of the input data of the back-propagation calculation graph is determined by identifying whether the loss of the output layer of the neural network model is repeatedly calculated by two or more computing nodes in the distributed computing system, including: if it is repeatedly calculated, modifying the value of sens according to the number of computing nodes that repeatedly calculate; that is, the value of the coefficient sens of the input data can be set to 1/n of the original value. Exemplarily, the original value of sens can default to 1.0.

That is to say, if the loss is calculated repeatedly, each computing node takes the positive loss value (that is, the output of the forward propagation of the neural network) as the starting point, and each computing node collaborates to complete the reverse propagation of the overall calculation. In the first layer of reverse propagation, sens needs to be set to 1/n, that is, each machine contributes 1/n at the starting point of the reverse derivation.

If it is not repeatedly calculated, the value of sens is maintained unchanged, that is, the original value is maintained. For example, the original value of sens may be defaulted to 1.0.

Based on the first aspect, in a possible embodiment, inserting a target operator between the input end of the model parameters of the computing node and the operator used to calculate the model parameters to obtain a target calculation graph includes: determining that the model parameters of the neural network model stored in the computing node are the same as the model parameters stored in at least one computing node other than the computing node in the distributed computing system; the computing nodes storing the same model parameters are located in the same communication group; inserting the target operator between the model parameters of the computing node and the operator used to calculate the model parameters to obtain the target calculation graph; wherein the output of the target operator is equal to the input of the target operator, and the derivative algorithm of the target operator is to sum the derivatives of the model parameters of all computing nodes in the communication group.

In the embodiment of the present application, the Mirror operator can be pre-defined to correspond to which parameters in the model parameters. In the forward propagation of the Mirror operator, the input is the model parameter copy w, and the output is equal to the input w. It does not perform actual calculations in the forward propagation, but only identifies which computing nodes the parameters are mirrored on. In the reverse propagation, the Mirror operator takes as input the derivative dout calculated by the next layer of the neural network, and the output dw is equal to an all-reduce sum of dout on the communication group.

Based on the first aspect, in a possible embodiment, the derivative algorithm of the target operator is an all_reduce summation operator.

Based on the first aspect, in a possible embodiment, after obtaining the derivative algorithm of each operator in the target calculation graph to generate the back propagation calculation graph of the neural network model, the method also includes: training the neural network model according to the forward propagation calculation graph and the back propagation calculation graph.

Based on the first aspect, in a possible embodiment, the distributed computing system may include multiple computing devices, each of which may serve as a computing node, and different computing devices may be used to complete the forward and reverse calculations of the neural network model in parallel. The computing device may be a terminal or a server.

In a second aspect, an embodiment of the present application provides a computing node, which is applied to a distributed computing system, including:

A single-machine automatic differentiation module, used to obtain a forward propagation calculation graph of a neural network model; wherein the forward propagation calculation graph includes multiple operators, the multiple operators include local operators and communication operators, the local operators represent operators that can complete calculations in the computing node, and the communication operators represent operators that rely on communication between the computing node and at least one computing node other than the computing node in the distributed computing system to complete calculations;

A mirror operator module, which is used to insert a target operator between the input end of the model parameter of the computing node and the operator used to calculate the model parameter based on the model parameter of the neural network model stored in the computing node being the same as the model parameter stored in at least one computing node other than the computing node in the distributed computing system, so as to obtain a target calculation graph; wherein the target operator is used to transparently transmit the model parameter, and the derivative algorithm of the target operator is used to sum the derivatives of the model parameters of all computing nodes having the same model parameter;

The single-machine automatic differentiation module is also used to generate a back-propagation calculation graph of the neural network model based on a derivative algorithm of some or all operators in the target calculation graph.

It can be seen that the embodiment of the present application designs a set of general distributed differentiation solutions. Developers only need to implement a distributed forward propagation network, without paying attention to how the distributed network backpropagation is carried out, and without paying attention to which computing nodes the model parameters are mirrored (i.e., there are model copies). The stand-alone automatic differentiation module of the computing node can automatically calculate the derivative algorithm of the communication operator, and call the derivative algorithm of the communication operator to complete the transmission (communication) of the derivative results across nodes. The mirror operator module can automatically implement the insertion of the mirror operator, and the stand-alone automatic differentiation module can generate and output the reverse propagation calculation graph, thereby avoiding the drawbacks of the prior art that requires users to intervene in the derivation, saving manpower and material costs, improving development efficiency, and reducing development difficulty. The present application solution is suitable for solving the current mainstream data parallelism, model parallelism, hybrid parallelism and other distributed network automatic differentiation problems, and improving the user experience.

Based on the second aspect, in a possible embodiment, the single-machine automatic differentiation module is specifically used to: obtain the derivative algorithm of the communication operator, the derivative algorithm of the local operator, and the derivative algorithm of the target operator in the target calculation graph; generate the back propagation calculation graph according to the derivative algorithm of the communication operator, the derivative algorithm of the local operator, and the derivative algorithm of the target operator.

Based on the second aspect, in a possible embodiment, the derivative algorithm of the local operator is stored in the local storage of the computing node; the computing node also includes a communication operator derivation module, and the communication operator derivation module is used to: before obtaining the derivative algorithm of each operator in the target calculation graph, perform a derivation operation on the communication operator in the forward propagation calculation graph to obtain the derivative algorithm of the communication operator.

Based on the second aspect, in a possible embodiment, the computing node also includes a sens processing module: the sens processing module is used to determine the coefficients (sens) of the input data of the back-propagation calculation graph by identifying whether the output layer of the neural network model is repeatedly calculated by two or more computing nodes in the distributed computing system.

Based on the second aspect, in a possible embodiment, the sens processing module is specifically used to modify the value of the sens according to the number of the computing nodes that are repeatedly calculated when the computing nodes are repeatedly calculated; and to maintain the value of the sens unchanged when the computing nodes are not repeatedly calculated.

Based on the second aspect, in a possible embodiment, the mirror operator module is specifically used to: determine that the model parameters of the neural network model stored in the computing node are the same as the model parameters stored in at least one computing node other than the computing node in the distributed computing system; the computing nodes storing the same model parameters are located in the same communication group; inserting a target operator between the model parameters of the computing node and the operator used to calculate the model parameters to obtain the target calculation graph; wherein the output of the target operator is equal to the input of the target operator, and the derivative algorithm of the target operator is to sum the derivatives of the model parameters of all computing nodes in the communication group.

Based on the second aspect, in a possible embodiment, the derivative algorithm of the target operator is an all_reduce summation operator.

In a third aspect, an embodiment of the present application provides a computing node, comprising: a memory, a communication interface, and a processor coupled to the memory and the communication interface; the memory is used to store program instructions, the processor is used to execute the program instructions, and the communication interface is used to communicate with other computing nodes of a distributed computing system under the control of the processor; when the processor executes the program instructions, the steps in the method described in any embodiment of the first aspect are performed.

In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method described in any embodiment of the first aspect are implemented.

In a fifth aspect, an embodiment of the present invention provides a computer program product. The computer program product includes program instructions, and when the computer program product is executed by a computing device, the computing device executes the method provided by any possible embodiment of the first aspect.

It can be seen that the embodiment of the present application designs a set of general distributed differentiation solutions, which are applicable to the automatic differentiation problems of various types of distributed networks such as current mainstream data parallelism, model parallelism, and hybrid parallelism. Developers only need to implement the distributed forward network, without paying attention to how the distributed network is carried out in reverse, and without paying attention to which nodes the parameters are copied. The computing nodes of the distributed computing system can automatically calculate the derivative algorithm of the communication operator, and call the derivative algorithm of the communication operator to complete the transmission (communication) of the derivative results across nodes, and automatically implement the insertion of the mirror operator, Sens calculation, and the generation and output of the back propagation calculation graph, thereby avoiding the drawbacks of the prior art that requires users to intervene in the derivation, saving manpower and material costs, improving development efficiency, reducing development difficulty, and thus improving the user experience.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

FIG1 shows an example diagram of a forward propagation computation graph;

FIG2 shows an example diagram of a back-propagation computation graph;

FIG3 is a schematic diagram of a system architecture provided in an embodiment of the present application;

FIG4 is a schematic diagram of a system architecture of a computing node provided in an embodiment of the present application;

FIG5 is a schematic diagram of the structure of a distributed automatic differentiation device provided in an embodiment of the present application;

FIG6 is a schematic diagram of a communication operator derivation module for derivation of a communication operator provided in an embodiment of the present application;

FIG7 is a schematic diagram of a method flow chart applied to a mirror operator module provided in an embodiment of the present application;

FIG8 is a diagram showing an example of a scenario for performing mirror operator insertion provided by an embodiment of the present application;

FIG9 is a schematic flow chart of a method applied to a Sens processing module provided in an embodiment of the present application;

FIG10 is a diagram showing an example of a scenario for executing Sens processing provided by an embodiment of the present application;

FIG11 is a schematic flow chart of a method for applying a single-machine automatic differentiation module provided in an embodiment of the present application;

FIG12 is a flow chart of a distributed automatic differentiation method provided in an embodiment of the present application;

FIG13 is a flow chart of another distributed automatic differentiation method provided in an embodiment of the present application;

14 is a schematic diagram of a computing node interacting with a client and a business person provided by an embodiment of the present invention;

FIG. 15 is a schematic diagram of the structure of a computing node provided in an embodiment of the present invention.

DETAILED DESCRIPTION

The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

It should be noted that the terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the present application. The singular forms of "a", "said" and "the" used in the embodiments of the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and/or" used in the embodiments of the present application refers to and includes any or all possible combinations of one or more associated listed items.

In order to facilitate a better understanding of the technical solutions described in the present application, the technical terms involved in the embodiments of the present application are first explained below.

(1) Computational graph and operator:

A computational graph is a graph structure that describes the computational process of a neural network model. If the computation has obvious modularity and there are obvious temporal and logical dependencies between modules, a directed graph structure can usually be used to describe it, that is, the computational process of a neural network model can be abstracted as a directed graph structure composed of tensor data and operators. In practical applications, there are two basic elements of a directed graph structure: the connection relationship between operators, which indicates the order in which operators are executed.

Generally speaking, describing the neural network model using a computational graph is conducive to an overall grasp of the entire neural network computing task. At the same time, the expression of the computational graph is also convenient for scheduling and parallel execution of computing tasks. Since the calculation of the neural network model involves the forward propagation (also known as forward propagation) calculation process and the reverse propagation calculation process, in the application embodiment, the computational graph used in the forward propagation calculation process may be referred to as the forward propagation calculation graph, and the computational graph used in the reverse propagation calculation process may be referred to as the reverse propagation calculation graph. FIG1 shows an example of a forward propagation calculation graph, and FIG2 shows an example of a reverse propagation calculation graph.

In the computational graph of the neural network model, multiple operators may be included, and one operator represents a basic computational method, such as addition, subtraction, multiplication, and division, which are four operators. For the forward propagation computational graph, multiple operators may be included, as shown in Figure 1, which shows three operators (operators A1, B1, and C1). The computational rule is the order from the input layer to the output layer of the neural network, and the arrow indicates the flow of data. Each operator may correspond to a derivative algorithm in the back propagation, that is, the derivative algorithm can be obtained by deriving the operator. For each derivative algorithm itself, the derivative algorithm may be a basic computational method or a combination of multiple basic computational methods. These derivative algorithms may form a back propagation computational graph. As shown in Figure 2, three derivative algorithms (derivative algorithm A2, derivative algorithm B2, and derivative algorithm C3) are shown, and the computational rule is the order from the output layer to the input layer of the neural network, and the arrow indicates the flow of data.

In the embodiment of the present application, operators can also be classified into local operators and communication operators. Local operators refer to operators that can complete calculations in local computing nodes, that is, operators that complete calculations in a single computing node; communication operators refer to operators that need to use the communication between the local computing node and at least one other computing node in the distributed computing system to complete the calculation, that is, operators that need to rely on other computing nodes other than a computing node to complete the calculation. For example, addition, subtraction, multiplication, and division operations are local operators, while operators such as send (indicating sending to other nodes), receive (indicating receiving from other nodes), all-reduce, and all-gather can be regarded as communication operators. There may be communication operators in the forward propagation calculation graph of the hybrid parallel scheme of the neural network model.

(2) Neural network model:

A deep learning framework typically uses a computational graph as the main data structure to describe a neural network model. The neural network model involved in the embodiments of the present application may include a deep neural network model (DNN), a convolutional neural network model (CNN), an extreme learning machine model (ELM), a convolutional neural network for target detection (Region-CNN, RCNN) model, a long short-term memory neural network (LSTM) model, a fully connected neural network or other neural network models, which are not limited in the embodiments of the present application.

For example, for CNN, the various types of operators involved may include neural network operators. For example, common neural network operators include: convolution/deconvolution operators, pooling operators, activation operators, softmax (classifier) operators, fully connected operators, etc. Forward propagation is used to receive input data from the input layer of CNN, and output data to the output layer after processing through the forward propagation calculation graph; back propagation is used to calculate the gradient of its input according to the output derivative (gradient) of the output layer, and pass it to the input layer after processing through the back propagation calculation graph.

(3) Distributed computing system:

In the embodiment of the present application, the distributed computing system is a system that uses multiple computing nodes for parallel computing. In a specific implementation, the distributed computing system may include multiple computing devices, each of which may serve as a computing node, and different computing devices may be used to complete the forward and reverse calculations of the neural network model in parallel. The computing device may be a terminal or a server, which is not limited in the present application.

In another implementation, when a computing device has a multi-core processor, each computing core can also be a computing node. The most common structure currently used by multi-core processors is a multi-core structure based on storage sharing. The processor contains multiple computing cores, each computing core has an independent cache, register file, computing unit and instruction control unit, and all computing cores share the same global storage. Multiple computing cores can be used to process computing tasks with a high degree of parallelism. For example, different computing cores can be used to complete the forward and reverse calculations of a neural network model in parallel.

In another specific implementation, a computing node can also be a virtual machine, container, computing instance, or process running on abstract hardware resources. For example, in a cloud server or data center, multiple virtual machines are deployed, and multiple virtual machines communicate through virtual switches to form a distributed computing system. Each virtual machine acts as a computing node and executes a specific computing method.

See Figure 3, which is a system architecture 300 provided in an embodiment of the present application. The execution device 210 may be a distributed computing system described in an embodiment of the present application. The execution device 210 may be implemented by one or more computing devices (such as servers), and optionally, in conjunction with other devices, such as data storage, routers, load balancers and other devices; the execution device 210 may be arranged at one physical site, or distributed at multiple physical sites. The computing device in the execution device 210 may use the data in the data storage system 250, or call the program code in the data storage system 250 to implement the method described in an embodiment of the present application. In this process, the computing device uses operators as specific elements for executing computing tasks, and provides a function (function) executed on a CPU or an artificial intelligence processor for each operator. According to the computational graph, the computing device executes the kernel function corresponding to each operator in the computational graph to complete the calculation of the neural network model.

Users can operate their respective user devices (e.g., local device 301 and local device 302) to interact with execution device 210. Each local device can represent any computing device, such as a personal computer, a computer workstation, a smart phone, a tablet computer, a smart camera, a smart car or other type of cellular phone, a media consumption device, a wearable device, a set-top box, a game console, etc.

The local device of each user can interact with the execution device 210 through a communication network of any communication mechanism/communication standard. The communication network can be a wide area network, a local area network, a point-to-point connection, etc., or any combination thereof.

In another implementation, one or more aspects of the execution device 210 may be implemented by each local device. For example, the local device 301 may provide local data or feedback calculation results to the execution device 210 .

It should be noted that all functions of the execution device 210 can also be implemented by the local device. For example, the local device 301 implements the functions of the execution device 210 and provides services to its own user, or provides services to the user of the local device 302.

Referring to FIG. 4 , the following is an exemplary description of the system architecture of a computing node in an embodiment of the present application. The computing node may be a computing node in a distributed computing system. As shown in FIG. 4 , the system architecture includes hardware at the bottom layer (e.g., hardware such as NPU, ARM, GPU, chips, etc.) and software modules and databases at the upper layer. The software modules specifically include a distributed forward propagation network module 41 and a distributed automatic differentiation device 42. The database includes a local operator library 43, a derivative algorithm library 44 of a local operator, a communication operator library 45 (or collective communication 45), and a derivative algorithm library 46 of a communication operator. Among them, the local operator library 43 can be used to store various local operators, the communication operator library 45 can be used to store various communication operators, the derivative algorithm library 44 of the local operator can be used to store the derivative algorithms of various local operators, and the derivative algorithm library 46 of the communication operator can be used to store the derivative algorithms of various communication operators. Among them, each local operator can be mapped to a derivative algorithm, and each communication operator can also be mapped to a derivative algorithm.

In a specific implementation, the distributed automatic differentiation device 42 in the embodiment of the present application can be integrated into an artificial intelligence (AI) computing framework (not shown in the figure), and the AI computing framework can call the operators in the above-mentioned database, or call the forward propagation calculation graph for use by the distributed automatic differentiation device.

The distributed forward propagation network module 41 is used to provide a forward propagation calculation graph of the neural network model at the computing node. The forward propagation calculation graph includes multiple operators, including local operators and communication operators. The local operator represents an operator that can complete the calculation in the computing node, and the communication operator represents an operator that needs to use the communication between the computing node and at least one other computing node in the distributed computing system to complete the calculation.

The distributed automatic differentiation device 42 is used to obtain the forward propagation calculation graph to automatically perform reasoning and automatically complete the differential calculation of the back propagation to generate the back propagation calculation graph of the neural network model at the calculation node.

In a possible embodiment, the computing nodes may be used to perform calculations on a distributed neural network model in a data parallel or model parallel or hybrid parallel manner.

In a possible embodiment, the computing node may also be used to train a neural network model according to a forward propagation computation graph and a back propagation computation graph.

Referring to FIG. 5 , FIG. 5 is a schematic diagram of the structure of a distributed automatic differentiation device provided in an embodiment of the present application. The distributed automatic differentiation device may be a distributed automatic differentiation device 42 applied to the computing node shown in FIG. 4 . As shown in FIG. 5 , the device includes: a communication operator derivation module 51, a mirror operator module 52, a Sens processing module 53, and a stand-alone automatic differentiation module 54. Some or all of these modules may be implemented in the form of software code. Among them:

The communication operator derivation module 51 is used to generate the reverse logic of each communication operator at the granularity of the communication operator, that is, to generate the derivative algorithm of the communication operator. Each communication operator has its corresponding reverse logic, and the communication operator derivation module has the ability to "derive" all communication operators. The calculated derivative algorithm of the communication operator can be saved in the derivative algorithm library of the communication operator shown in FIG4.

The mirror operator module 52 is used to insert a target operator between the model parameters of the current computing node and the operator used to calculate the model parameters when the model parameters of the neural network model of the current computing node are the same as the model parameters of at least one other computing node. The target operator described in the embodiment of the present application may also be referred to as a mirror operator. Specifically, it is possible to identify which model parameters are repeatedly distributed in multiple computing nodes in the forward propagation calculation graph, and insert a Mirror operator at the corresponding position of the forward propagation calculation graph.

That is to say, when the model parameters (or training parameters) of the neural network model are mirrored on multiple computing nodes in a distributed computing system (such as data parallelism, model parallelism, or hybrid parallelism), the Mirror operator can be introduced on the forward propagation calculation graph to form a target calculation graph. The algorithm of the Mirror operator is forward direct, transparently transmitting the model parameters, and the derivative algorithm of the Mirror operator is the all_reduce (SUM) operator, that is, the gradient all_reduce is performed in reverse.

The Sens processing module 53 is used to identify the number n of computing nodes with the same computing process in the distributed computing system, that is, to identify whether the loss value of the last operator in the forward propagation process of each computing node is repeatedly calculated (that is, to identify whether the output layer is repeatedly calculated in the forward propagation calculation graph). If it is repeatedly calculated, the coefficient sens of the input gradient of the back propagation is divided by n.

The stand-alone automatic differentiation module 54 is used to perform reverse automatic differentiation according to the forward propagation calculation graph or the target calculation graph (if a Mirror operator exists) of the neural network model, and automatically insert the derivative algorithm corresponding to the local operator and the derivative algorithm corresponding to the communication operator to generate the reverse propagation calculation graph of the neural network model at this computing node, thereby realizing distributed automatic differentiation in a data parallel or hybrid parallel manner.

In order to better understand the technical solution of the present application, the functions of each module in the distributed automatic differentiation device are further described in detail below.

For the communication operator derivation module: see Figure 6. Each communication operator in the forward propagation calculation graph corresponds to a section of reverse logic (a combination of several operators). For each communication operator, the communication operator derivation module can define and derive the corresponding "derivative" for the communication operator, thereby obtaining the derivative algorithm of the communication operator, and save the derivative algorithm to the derivative algorithm library for subsequent use.

For example, the communication operator and its corresponding derivative algorithm may be shown in Table 1 below:

Table 1

It should be noted that the above Table 1 is only used to explain the present application scheme and is not intended to be limiting.

For the mirror operator module: When performing data parallelism, model parallelism, or hybrid parallelism, if the training sample data is split, and the same model parameter w is mirrored on multiple computing nodes, that is, distributed on multiple computing nodes at the same time, each computing node performs forward propagation calculations based on different training samples and the same copy of the model parameter. Mathematically, the derivative of the parameter needs to be summed up for the derivatives of each copy of all computing nodes.

Mirroring the model parameter copies on each computing node is also an operation, but this mirroring operation is generally not placed in the scope of forward propagation (that is, before the forward calculation begins, the copies of the parameters on each computing node already exist), while the reverse propagation is reverse derivatives within the scope of forward propagation. Therefore, in distributed automatic differentiation, it is necessary to describe which parameters are mirrored on which nodes.

The mirror operator module can identify which model parameters are repeatedly distributed on multiple computing nodes in the forward propagation, and insert the Mirror operator at the corresponding position of the forward propagation calculation graph. Referring to FIG. 7 , in one implementation, the process may include the following steps:

In S401, the mirror operator module determines the model parameters of the neural network model that the local computing node needs to derive.

At S402, for each parameter in the model parameter, it is analyzed whether there are copies in multiple computing nodes. If there are copies, the identification (ID) of the relevant computing node is recorded.

In S403, a communication group is created for the computing nodes corresponding to the multiple replicas.

In S404, a mirror operator is created for the parameter with multiple copies, and the corresponding communication group is recorded in the mirror operator.

In S405, the mirror operator is inserted between the input end of the corresponding parameter and the operator originally used to calculate the parameter, thereby obtaining the target calculation graph. In other words, the input of the mirror operator is the parameter, and the output of the mirror operator is used as the input of the operator originally used to calculate the parameter. Since the role of the mirror operator in forward propagation is to pass through the parameter, the output of the mirror operator is also the parameter, that is, the parameter will eventually be input into the operator originally used to calculate the parameter.

In the embodiment of the present application, the mirror operator module can pre-define which parameters in the model parameters the Mirror operator corresponds to. In the forward propagation of the Mirror operator, the input is the model parameter copy w, and the output is equal to the input w. It does not perform actual calculations in the forward propagation, but only identifies which computing nodes the parameters are mirrored on. In the reverse propagation of the Mirror operator, the input is the derivative dout calculated by the next layer of the neural network, and the output dw is equal to an all-reduce sum of dout on the communication group;

As shown in Figure 8, if in a distributed computing system, the communication group corresponding to the model parameter w includes two computing nodes, whose communication numbers are 0 and 1 respectively (i.e., machine 0 and machine 1 in the figure). The two computing nodes have a copy of the same model parameter w at the same time, but each has a different set of training sample data a1 and a2. Then, in the forward propagation calculation graph, the mirror operator module can insert the position of the mirror operator between the parameter w and the original operator OP. During the forward propagation calculation, the output of the mirror operator is equal to the input, that is, the input of the mirror operator is the parameter w, and the output of the mirror operator is used as the input of the operator OP. Since the output of the mirror operator is also the parameter w, the parameter w will eventually be input into the operator OP.

During the back-propagation calculation, the reverse output of the mirror is equal to an all-reduce summation operation on the input dout on machine 0 and machine 1 (i.e., each member of the communication group).

For the Sens processing module: The Sens processing module can be used to determine the coefficients sens of the input data of the back propagation calculation graph. Referring to FIG. 9 , in one implementation, the process may include the following steps:

At S501.Sens processing module identifies whether the output layer of the neural network is repeatedly calculated.

In S502, if the calculation is repeated, the Sens processing module determines the number n of computing nodes to be repeated, and divides the coefficient sens of the input data of the back propagation by n. The result is used as the new coefficient sens of the input data. That is to say, the value of the coefficient sens of the input data can be set to 1/n of the original value. For example, the original value of sens can be defaulted to 1.0. The input data is usually the derivative of loss (the result is, for example, 1). For example, when n=2, the final input of the back propagation is 1/2.

In S503, if there is no repeated calculation, the Sens processing module maintains the coefficient sens of the back-propagated input data unchanged, that is, maintains the original value. For example, the original value of sens may be defaulted to 1.0.

In an embodiment of the present application, the Sens processing module can identify whether the output layer of the forward propagation (generally loss) is repeatedly calculated. If the loss is repeatedly calculated, each computing node uses the forward loss value (that is, the output of the forward propagation of the neural network) as the starting point, and each computing node collaborates to complete the reverse propagation of the overall calculation. In the first layer of reverse propagation, sens needs to be set to 1/n, that is, each machine contributes 1/n at the starting point of the reverse derivation.

For example, referring to FIG10 , if in a distributed computing system, machine 0 and machine 1 work together to complete the forward propagation calculation.

In the forward propagation calculation process of machine 0, the input is data b, and b is broadcasted through the broadcast operator so that machine 0 and machine 1 each have a copy, denoted as c (b and c are the same). The multiplication operator M is used to calculate the product of c and local data d1 to obtain cd1; the results of machine 0 and machine 1 are merged once through the communication operator all_gather operator to obtain [cd1, cd2], and finally the square sum of [cd1, cd2] is calculated through the operator Reduce-SUM to obtain the loss value of the output layer.

In the forward propagation calculation process of machine 1, machine 1 receives the data c broadcast by computing node 0, and calculates the product of c and local data d2 through the multiplication operator M to obtain cd2; the results of machine 0 and machine 1 are merged once through the communication operator all_gather operator to obtain [cd1, cd2], and finally the square sum of [cd1, cd2] is calculated through the operator Reduce-SUM to obtain the loss value of the output layer.

It can be seen that for the forward propagation process of these two machines, the final output loss is the same, that is, the process of calculating loss for machine 0 and machine 1 is repeated, and the same loss value is obtained. Therefore, during the reverse propagation process, the Sens processing module can recognize that the loss is repeatedly calculated, and the number of computing nodes for repeated calculation is 2 (that is, n = 2), so the reverse propagation sens is divided by the number of computing nodes for repeated calculation 2. For example, the default value of sens is 1.0, so in the reverse propagation process in Figure 10, the input of machine 0 and machine 1 is 1/2.

For a stand-alone automatic differentiation module: the stand-alone automatic differentiation module can generate a back propagation calculation graph according to the reverse logic (derivative algorithm) of each operator (including local operators, communication operators, and possible mirror operators) in the forward propagation calculation graph (or target calculation graph). Referring to FIG. 11 , in one implementation, the process may include the following steps:

In S601, the stand-alone automatic differentiation module calls the derivative algorithm of each operator in sequence along the order from the output layer to the input layer of the input neural network. Among them, if the operator in the forward propagation is a local operator, then in S602, the stand-alone automatic differentiation module finds the derivative algorithm corresponding to the local operator from the derivative algorithm library of the local operator. If the operator in the forward propagation is a communication operator, then in S603, the derivative algorithm corresponding to the communication operator is found from the derivative algorithm library of the communication operator. If the operator in the forward propagation is a mirror operator, then in S604, the all_reduce (SUM) operator determined by the mirror operator module is found. Finally, in S605, the stand-alone automatic differentiation module outputs the back-propagation calculation graph.

It should be noted that, for the sake of convenience, the various method embodiments described in the embodiments of the present application are expressed as a combination of a series of action steps, but those skilled in the art should be aware that the specific implementation of the technical solution of the present application is not limited by the order of the described series of action steps.

Based on the system architecture and various functional modules described above, the following further describes the distributed automatic differentiation method provided in the embodiment of the present application. The method can be applied to computing nodes in a distributed computing system. See FIG. 12 , which is a flow chart of a distributed automatic differentiation method. The method includes but is not limited to the following steps:

S101. The computing node obtains the forward propagation calculation graph of the neural network model.

Among them, the forward propagation calculation graph includes multiple operators, including local operators and communication operators. The local operators represent operators that can complete calculations in the computing node, and the communication operators represent operators that need to utilize communication between the computing node and at least one remaining computing node in the distributed computing system to complete calculations.

Specifically, the forward propagation computation graph of the neural network model may be pre-configured, for example, obtained from a distributed forward network configured by a user, and the content of the forward propagation computation graph is stored in the local storage of the computing node.

The forward propagation computation graph, local operators, and communication operators have been described in the previous article and will not be repeated here.

S102. When the model parameters of the neural network model owned by the computing node are the same as the model parameters owned by at least one other computing node, the computing node inserts a target operator between the model parameters of the computing node and the operator used to calculate the model parameters to obtain a target computing graph.

Among them, the algorithm of the target operator is to pass through the model parameters in the forward propagation, that is, its input is equal to the output, and the derivative algorithm of the target operator is to sum the derivatives of each computing node with the same model parameters at the model parameters. For example, the derivative algorithm of the target operator is the all_reduce (SUM) operator.

Specifically, the embodiment of the present application can define a mirror operator through a mirror operator module, whose attribute is a communication group (group). In forward propagation, its output is equal to the input, and in reverse propagation, its output is equal to the input in the communication group. The distributed forward propagation network input by the user is analyzed, and for the model parameters driven by the demand, it is identified on which computing nodes have a common model copy, and a communication group is created for the corresponding computing nodes. In the forward propagation calculation graph, the mirror operator is inserted between the corresponding model parameter and the operator originally used to calculate the model parameter, thereby obtaining the target calculation graph.

The specific implementation process of this step can refer to the relevant description of the embodiments of Figures 7 and 8, which will not be repeated here.

S103: The computing node obtains a derivative algorithm for each operator in the target computing graph.

Specifically, the computing node may obtain the derivative algorithm of each communication operator in the target computing graph, the derivative algorithm of each local operator, and the derivative algorithm of the target operator. The derivative algorithm of the local operator is stored in the local storage of the computing node. The computing node may use the communication operator derivation module to perform derivation operations on each communication operator in the forward propagation computing graph to obtain the derivative algorithm of each communication operator, and save it to the derivative algorithm library of the communication operator.

The implementation method of derivation of the communication operator has been described in the previous article and will not be repeated here.

S104. The computing node generates a back propagation computing graph of the neural network model according to the derivative algorithm of each operator (including local operators, communication operators, and mirror operators) in the target computing graph.

Specifically, the computing node can use the single-machine automatic differentiation module to call the derivative algorithm of each local operator/communication operator/possible mirror operator from the database in turn from the output layer to the input layer according to the forward propagation process (such as the target calculation graph) on the computing node, and automatically generate a complete back-propagation calculation graph.

The specific implementation process of this step can refer to the relevant description of the embodiment of Figure 11, which will not be repeated here.

It can be seen that the embodiment of the present application designs a set of general distributed differentiation solutions. Developers only need to implement a distributed forward propagation network, without paying attention to how the distributed network backpropagation is carried out, and without paying attention to which computing nodes the model parameters are mirrored (i.e., there are model copies). The computing nodes of the distributed computing system can automatically calculate the derivative algorithm of the communication operator, and call the derivative algorithm of the communication operator to complete the transmission (communication) of the derivative results across nodes, and automatically implement the insertion of the mirror operator, the generation and output of the backpropagation calculation graph, thereby avoiding the drawbacks of the prior art that requires users to intervene in the derivation, saving manpower and material costs, improving development efficiency, and reducing development difficulty. The present application solution is suitable for solving the current mainstream data parallelism, model parallelism, hybrid parallelism and other distributed network automatic differentiation problems, and improving the user experience.

Based on the system architecture and various functional modules described above, see FIG. 13 , which is a flow chart of another distributed automatic differentiation method, which can be applied to computing nodes in a distributed computing system, and includes but is not limited to the following steps:

S201, at each computing node, pre-save the derivative algorithm of each local operator to the derivative algorithm library of the local operator, derive the derivative algorithm of each communication operator, and save the derivative algorithm of each communication operator to the derivative algorithm library of the communication operator. Save the forward propagation calculation graph of the neural network model.

Specifically, each computing node can pre-configure a local operator library, a derivative algorithm library of the local operator library, and a communication operator library in local storage. For each communication operator in the communication operator library, the communication operator derivation module provided in the embodiment of the present application can be used to define its "derivative" for each communication operator, describe the reverse logic corresponding to each communication operator, and then generate a derivative library of the communication operator.

Specifically, each computing node is used together to assist in the calculation of a complete neural network model, and in the distributed computing of the model (such as data parallelism, or model parallelism, or hybrid parallelism), the forward propagation computation graphs corresponding to each computing node may be different, such as model parameters, input data, and the calculation rules of the computation graph. Each computing node can obtain the forward propagation computation graph corresponding to the computing node itself through the distributed forward propagation network. The distributed forward propagation network can be pre-configured, for example, it can be provided by the user through the distributed computing system. The interface can be in various forms, such as a web interface, a command line tool, a REST interface, etc., which are not limited here.

S202: When a computing node needs to perform reasoning of a back-propagation computing graph, the forward propagation computing graph of the computing node is obtained from local storage.

S203, the computing node identifies which model parameters are repeatedly distributed on multiple computing nodes, and inserts the Mirror operator at the corresponding position of the forward propagation computing graph to obtain the target computing graph.

Specifically, the embodiment of the present application can define a mirror operator through a mirror operator module, whose attribute is a communication group (group). In forward propagation, its output is equal to the input, and in reverse propagation, its output is equal to the input in the communication group. The distributed forward propagation network input by the user is analyzed, and for the model parameters driven by the demand, it is identified on which computing nodes have a common model copy, and a communication group is created for the corresponding computing nodes. In the forward propagation calculation graph, the mirror operator is inserted between the corresponding model parameter and the operator originally used to calculate the model parameter, thereby obtaining the target calculation graph.

It can be understood that the target computation graph includes the content of the original forward propagation computation graph and the content of the newly added mirror operator. The target computation graph can be used as a logical intermediate variable for obtaining the final reverse propagation computation graph and is not visible to the outside.

The specific implementation process of this step can refer to the relevant description of the embodiments of Figures 7 and 8, which will not be repeated here.

S204. The computing node determines the coefficient sens of the input data of the back-propagation computing graph.

Specifically, the computing node can analyze the forward propagation process (target calculation graph) of each computing node in the system after the mirror operator is inserted through the Sens processing module, and then identify whether the output layer of the neural network is repeatedly calculated. If it is repeatedly calculated, the coefficient sens of the input data of the back propagation calculation graph of the current node is determined according to the number of computing devices that are repeatedly calculated.

The specific implementation process of this step can refer to the relevant description of the embodiments of Figures 9 and 10, which will not be repeated here.

S205. The computing node generates and outputs a reverse propagation computation graph according to the derivative algorithm of each operator (including local operators, communication operators, and possible mirror operators) in the forward propagation computation graph (or target computation graph).

Specifically, the computing node can use the single-machine automatic differentiation module to call the derivative algorithm of each local operator/communication operator/possible mirror operator from the database in turn from the output layer to the input layer according to the forward propagation process (such as the target calculation graph) on the computing node, and automatically generate a complete back-propagation calculation graph.

The specific implementation process of this step can refer to the relevant description of the embodiment of Figure 11, which will not be repeated here.

S206. Optionally, each computing node trains the neural network model according to the forward propagation calculation graph and the back propagation calculation graph, thereby realizing a distributed model training process.

It is understandable that after each computing node obtains its corresponding back propagation calculation graph, the forward propagation calculation graph, the back propagation calculation graph and the training sample data of the computing node can be used to train the neural network model. The specific training process is well known to those skilled in the art and will not be expanded here.

It can be seen that the embodiment of the present application designs a set of general distributed differentiation solutions, which are applicable to the automatic differentiation problems of various types of distributed networks such as current mainstream data parallelism, model parallelism, and hybrid parallelism. Developers only need to implement the distributed forward network, without paying attention to how the distributed network is carried out in reverse, and without paying attention to which nodes the parameters are copied. The computing nodes of the distributed computing system can automatically calculate the derivative algorithm of the communication operator, and call the derivative algorithm of the communication operator to complete the transmission (communication) of the derivative results across nodes, and automatically implement the insertion of the mirror operator, Sens calculation, and the generation and output of the back propagation calculation graph, thereby avoiding the drawbacks of the prior art that requires users to intervene in the derivation, saving manpower and material costs, improving development efficiency, reducing development difficulty, and thus improving the user experience.

Based on the same inventive concept, the following further provides related devices of the embodiments of the present invention.

14 is a schematic diagram of a computing node 700 interacting with a client 740 and a business person according to an embodiment of the present invention. The computing node 700 may include multiple processors 710 and multiple memories 720. The computing node 700 may be a computing node described in any of the above embodiments.

The memory 720 can be used to store program codes and data (such as sample data, model parameters, databases, codes of various functional modules, etc.). The computing node 700 also provides an external interface, which can be in various forms, such as a web interface, a command line tool, a REST interface, etc., which are not limited here.

The processor 710 may be used to run the various functional modules described above. Specifically, the processor 710 may call program instructions in the memory 720 to execute the method described in any of the method embodiments described above.

It should be understood that computing node 700 is only an example provided in an embodiment of the present application, and computing node 700 may have more or fewer components than those shown, may combine two or more components, or may have different configurations of components.

Referring to FIG. 15 , FIG. 15 is a schematic diagram of the structure of another computing node 800 provided in an embodiment of the present invention. The computing node 800 includes one or more processors 811, a communication interface 812, and a memory 813. The processor 811, the communication interface 812, and the memory 813 may be connected via a bus 814. In some possible implementations, the computing node 800 may be deployed in, for example, a single application server or a terminal or a server cluster.

The communication interface 812 may be a wired interface (eg, an Ethernet interface) for communicating and interacting with other computing nodes or users.

The memory 813 may include a volatile memory, such as a random access memory (RAM); the memory may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD). The memory may also include a combination of the above-mentioned types of memory. The memory 813 may store program code and data (such as sample data, model parameters, databases, codes of various functional modules, etc.).

The processor 811 includes one or more general-purpose processors, wherein the general-purpose processor can be any type of device capable of processing electronic instructions, including a central processing unit (CPU), a microprocessor, a microcontroller, a main processor, a controller, and an ASIC (Application Specific Integrated Circuit). The processor 811 executes various types of digital storage instructions, such as software or firmware programs stored in the memory 813, which enables the computing node 800 to provide a wide variety of services. For example, the processor 811 can execute a program or process data to perform at least a portion of the method embodiments disclosed in the present application.

In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one network site, computer, server or data center to another network site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, microwave, etc.) means. The computer-readable storage medium can be any available medium that can be accessed by a computer, or it can be a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape, etc.), an optical medium (e.g., a DVD, etc.), or a semiconductor medium (e.g., a solid-state hard disk), etc.

In the above embodiments, the description of each embodiment has different emphases. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

Claims (23)

1. A distributed automatic differentiation method, the method being applied to a computing node in a distributed computing system, characterized in that it comprises: Obtain a forward propagation calculation graph of a neural network model; wherein the forward propagation calculation graph includes a plurality of operators, the plurality of operators including a local operator and a communication operator, the local operator represents an operator that can complete calculation in the computing node, and the communication operator represents an operator that relies on communication between the computing node and at least one computing node other than the computing node in the distributed computing system to complete calculation; Based on the model parameters of the neural network model stored in the computing node being the same as the model parameters stored in at least one computing node other than the computing node in the distributed computing system, a target operator is inserted between the input end of the model parameter of the computing node and the operator used to calculate the model parameter to obtain a target calculation graph; wherein the target operator is used to transparently transmit the model parameter, and the derivative algorithm of the target operator is used to sum the derivatives of the model parameters of all computing nodes having the same model parameters; A back propagation computation graph of the neural network model is generated based on a derivative algorithm of some or all operators in the target computation graph. 2. The method according to claim 1, characterized in that the generating the back propagation calculation graph of the neural network model based on the derivative algorithm of some or all operators in the target calculation graph comprises: Obtaining a derivative algorithm of the communication operator, a derivative algorithm of the local operator, and a derivative algorithm of the target operator in the target computation graph; The back-propagation computation graph is generated according to the derivative algorithm of the communication operator, the derivative algorithm of the local operator, and the derivative algorithm of the target operator. 3. The method according to claim 2, characterized in that the derivative algorithm of the local operator is stored in the local storage of the computing node; the method further comprises: A derivative operation is performed on the communication operator in the forward propagation computation graph to obtain a derivative algorithm of the communication operator. 4. The method according to any one of claims 1 to 3, characterized in that the method further comprises: The coefficient sens of the input data of the back-propagation computation graph is determined by identifying whether the output layer of the neural network model is repeatedly computed by two or more computing nodes in the distributed computing system. 5. The method according to claim 4, characterized in that the step of determining the sens of the input data of the back-propagation computation graph by identifying whether the output layer of the neural network model is repeatedly computed by two or more computing nodes in the distributed computing system comprises: If it is repeatedly calculated, the value of sens is modified according to the number of computing nodes that are repeatedly calculated; If it is not recalculated, the value of sens remains unchanged. 6. The method according to any one of claims 1 to 3 and 5, characterized in that inserting a target operator between an input terminal of a model parameter of the computing node and an operator for computing the model parameter to obtain a target computing graph comprises: Determining that the model parameters of the neural network model stored in the computing node are the same as the model parameters stored in at least one computing node other than the computing node in the distributed computing system; the computing nodes storing the same model parameters are located in the same communication group; Inserting a target operator between the model parameters of the computing node and the operator used to calculate the model parameters to obtain the target computation graph; wherein the output of the target operator is equal to the input of the target operator, and the derivative algorithm of the target operator is to sum the derivatives of the model parameters of all computing nodes in the communication group. 7. The method according to claim 4, characterized in that inserting a target operator between an input terminal of a model parameter of the computing node and an operator for computing the model parameter to obtain a target computing graph comprises: Determining that the model parameters of the neural network model stored in the computing node are the same as the model parameters stored in at least one computing node other than the computing node in the distributed computing system; the computing nodes storing the same model parameters are located in the same communication group; Inserting a target operator between the model parameters of the computing node and the operator used to calculate the model parameters to obtain the target computation graph; wherein the output of the target operator is equal to the input of the target operator, and the derivative algorithm of the target operator is to sum the derivatives of the model parameters of all computing nodes in the communication group. 8. The method according to claim 6 is characterized in that the derivative algorithm of the target operator is an all_reduce summation operator. 9. The method according to claim 7, characterized in that the derivative algorithm of the target operator is an all_reduce summation operator. 10. The method according to any one of claims 1-3, 5, 7-9, characterized in that after obtaining the derivative algorithm of each operator in the target calculation graph to generate the back propagation calculation graph of the neural network model, the method further comprises: The neural network model is trained according to the forward propagation calculation graph and the back propagation calculation graph. 11. The method according to claim 4, characterized in that after obtaining the derivative algorithm of each operator in the target calculation graph to generate the back propagation calculation graph of the neural network model, the method further comprises: The neural network model is trained according to the forward propagation calculation graph and the back propagation calculation graph. 12. The method according to claim 6, characterized in that after obtaining the derivative algorithm of each operator in the target calculation graph to generate the back propagation calculation graph of the neural network model, the method further comprises: The neural network model is trained according to the forward propagation calculation graph and the back propagation calculation graph. 13. A computing node, applied to a distributed computing system, comprising: A single-machine automatic differentiation module, used to obtain a forward propagation calculation graph of a neural network model; wherein the forward propagation calculation graph includes multiple operators, the multiple operators include local operators and communication operators, the local operators represent operators that can complete calculations in the computing node, and the communication operators represent operators that rely on communication between the computing node and at least one computing node other than the computing node in the distributed computing system to complete calculations; A mirror operator module, which is used to insert a target operator between the input end of the model parameter of the computing node and the operator used to calculate the model parameter based on the model parameter of the neural network model stored in the computing node being the same as the model parameter stored in at least one computing node other than the computing node in the distributed computing system, so as to obtain a target calculation graph; wherein the target operator is used to transparently transmit the model parameter, and the derivative algorithm of the target operator is used to sum the derivatives of the model parameters of all computing nodes having the same model parameter; The single-machine automatic differentiation module is also used to generate a back-propagation calculation graph of the neural network model based on a derivative algorithm of some or all operators in the target calculation graph. 14. The computing node according to claim 13, wherein the stand-alone automatic differentiation module is specifically used for: Obtaining a derivative algorithm of the communication operator, a derivative algorithm of the local operator, and a derivative algorithm of the target operator in the target computation graph; The back-propagation computation graph is generated according to the derivative algorithm of the communication operator, the derivative algorithm of the local operator, and the derivative algorithm of the target operator. 15. The computing node according to claim 14, characterized in that the derivative algorithm of the local operator is stored in the local storage of the computing node; the computing node further comprises a communication operator derivation module, and the communication operator derivation module is used to: Before obtaining the derivative algorithm of each operator in the target computation graph, a derivative operation is performed on the communication operator in the forward propagation computation graph to obtain the derivative algorithm of the communication operator. 16. The computing node according to any one of claims 13 to 15, characterized in that the computing node further comprises a sens processing module: the sens processing module is used to: The coefficient sens of the input data of the back-propagation computation graph is determined by identifying whether the output layer of the neural network model is repeatedly computed by two or more computing nodes in the distributed computing system. 17. The computing node according to claim 16 is characterized in that the sens processing module is specifically used to modify the value of the sens according to the number of computing nodes that are repeatedly calculated when the computing node is repeatedly calculated; and maintain the value of the sens unchanged when the computing node is not repeatedly calculated. 18. The computing node according to any one of claims 13 to 15 and 17, wherein the mirror operator module is specifically used for: Determining that the model parameters of the neural network model stored in the computing node are the same as the model parameters stored in at least one computing node other than the computing node in the distributed computing system; the computing nodes storing the same model parameters are located in the same communication group; Inserting a target operator between the model parameters of the computing node and the operator used to calculate the model parameters to obtain the target computation graph; wherein the output of the target operator is equal to the input of the target operator, and the derivative algorithm of the target operator is to sum the derivatives of the model parameters of all computing nodes in the communication group. 19. The computing node according to claim 16, wherein the mirror operator module is specifically used for: Determining that the model parameters of the neural network model stored in the computing node are the same as the model parameters stored in at least one computing node other than the computing node in the distributed computing system; the computing nodes storing the same model parameters are located in the same communication group; Inserting a target operator between the model parameters of the computing node and the operator used to calculate the model parameters to obtain the target computation graph; wherein the output of the target operator is equal to the input of the target operator, and the derivative algorithm of the target operator is to sum the derivatives of the model parameters of all computing nodes in the communication group. 20. The computing node according to claim 18, characterized in that the derivative algorithm of the target operator is an all_reduce summation operator. 21. The computing node according to claim 19, characterized in that the derivative algorithm of the target operator is an all_reduce summation operator. 22. A computing device, characterized in that the computing device comprises: a memory, a communication interface, and a processor coupled to the memory and the communication interface; the memory is used to store program instructions, the processor is used to execute the program instructions, and the communication interface is used to communicate with other computing nodes of a distributed computing system under the control of the processor; When the processor executes the program instructions, the processor performs the steps of the method according to any one of claims 1 to 12. 23. A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 12 are implemented.