WO2023093689A1 - Computational graph optimization method and apparatus, and device - Google Patents

Abstract

A computational graph optimization method, comprising: obtaining a second computational graph from a first computational graph on the basis of a data dependency relationship between parameters and nodes in the first computational graph; then combining the second computational graph and the first computational graph to obtain a new computational graph; fusing operators in the new computational graph to obtain an optimized computational graph; and finally executing the optimized computational graph. In this way, the computational graph is optimized in a mode of combining re-computation and operator fusion, such that large re-computation overhead is not introduced while memory occupation is remarkably reduced, and the problem of incapability of execution of a network having one or more super-large tensors is solved.

Description

A calculation graph optimization method, device and equipment

This application claims the priority of the Chinese patent application filed with the State Intellectual Property Office of China on November 29, 2021, with the application number 202111431551.X and the application name "A Computational Graph Optimization Method, Device and Equipment", the entire content of which Incorporated in this application by reference.

technical field

The present application relates to the technical field of artificial intelligence, and in particular to a calculation graph optimization method, device and equipment.

Background technique

Artificial intelligence (AI) is a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is the branch of computer science that attempts to understand the nature of intelligence and produce a new class of intelligent machines that respond in ways similar to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that the machines have the functions of perception, reasoning and decision-making. Research in the field of artificial intelligence includes robotics, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, basic AI theory, etc.

At present, with the continuous development of computer technology, AI networks have also been widely used. Moreover, the AI network is becoming more and more complex, and the scale of its network model is showing a trend of increasing. For example, the number of network layers, parameters, and data sets of the AI network are increasing, which makes the network model of the AI network The corresponding memory consumption is getting bigger and bigger. Since the memory of the current computer hardware is relatively small, the common one is 16 gigabytes (GB) or 32GB, etc., which makes the memory consumption of the network model of the AI network more and more large, the current computer hardware It will be difficult to support the network model of AI network. Therefore, how to reduce the memory consumption of the network model of the AI network is a technical problem that needs to be solved urgently.

Contents of the invention

The present application provides a calculation graph optimization method, device, equipment, computer storage medium, computer program product, and chip. The calculation graph is optimized by combining recomputation and operator fusion, achieving a significant reduction in memory usage. At the same time, it solves the problem that networks with one or more very large tensors cannot perform without introducing large heavy calculation overhead.

In the first aspect, the present application provides a computation graph optimization method, including: obtaining a second computation graph from the first computation graph based on parameters and data dependencies between nodes in the first computation graph, the parameters including operator fusion rules , one or more of the memory threshold of the memory occupied by the tensor output by a single node in the first computation graph, the peak memory threshold corresponding to the first computation graph, and the data mutation threshold corresponding to the node in the first computation graph, the peak memory The threshold is the threshold of the memory occupied by all the tensors to be used at the moment of execution of a node during the execution of the calculation graph, and the data mutation threshold is the threshold of data mutation generated by at least one node in the first calculation graph; the second calculation graph and the second calculation graph A computation graph is merged to obtain a third computation graph, wherein, in the third computation graph, there is a first directed edge between the first node that outputs the first tensor and the second node that inputs the first tensor, and the output There is a second directed edge between the third node of the second tensor and the fourth node of the input second tensor, the first directed edge is from the first node to the second node, and the second directed edge has the third node pointing to the second Four nodes, the first node and the fourth node correspond to the nodes in the first calculation graph, the second node and the third node correspond to the nodes in the second calculation graph; the operators in the third calculation graph are fused to obtain the first Four computation graphs; execute the fourth computation graph. In this way, based on the parameters, the second calculation graph that needs to be recalculated is screened out from the first calculation graph, and the second calculation graph is merged with the first calculation graph to obtain a new calculation graph, and the new calculation graph Carry out operator fusion and execute the fused calculation graph. Thus, the calculation graph is optimized through the combination of recalculation and operator fusion, which achieves a significant reduction in memory usage without introducing large recalculation overhead. , which solves the problem that networks with one or more very large tensors cannot perform.

In a possible implementation, the second computation graph is obtained from the first computation graph based on parameters and data dependencies between nodes in the first computation graph, which specifically includes: based on the parameters and the nodes in the first computation graph The data dependency among them is obtained from the producers included in the first calculation graph, and N first node sets are obtained, N is a positive integer greater than or equal to 1, and the first node set includes at least one node, wherein, one first The nodes contained in the node set are all directly or indirectly related to a tensor output by a producer in the first calculation graph, and are directly or indirectly related to at least one tensor input by a producer in the first calculation graph; Recomputation is performed on each first node set to obtain N recalculation subgraphs, and the N recomputation subgraphs constitute a second computation graph. In this way, the second computation graph can be obtained from the first computation graph according to the data dependencies and parameters among the nodes in the first computation graph.

In a possible implementation manner, the parameter is a memory threshold; the memory occupied by the output tensor corresponding to the first node set is higher than the memory threshold. In this way, the node set corresponding to the output tensor that occupies too much memory can be automatically used as the required node set, thereby solving the problem of memory occupied by a single operator and/or tensor.

In a possible implementation, the parameter is the peak memory threshold; there is an indirect path between the producer and the consumer corresponding to the first node set, and each node between the producer and the consumer corresponding to the first node set executes The peak memory when is above the peak memory threshold. In this way, the node set corresponding to the high peak memory usage can be automatically used as the required node set, thereby solving the problem of memory usage of a single operator and/or tensor.

In a possible implementation, the parameter is a data mutation threshold; the deviation value between the memory occupied by the output tensor corresponding to the first node set and the memory occupied by at least one input tensor corresponding to the first node set is high at the data mutation threshold. In this way, the node set with data mutation can be automatically used as the required node set, thereby eliminating the problem of excessive memory usage caused by data mutation.

In a possible implementation, the parameter is an operator fusion rule; the first node set and the consumers corresponding to the first node set can be fused; or, the first node set and the consumers corresponding to the first node set cannot be fused, And there is an indirect path between the producer and the consumer corresponding to the first node set. In this way, the required node set can be filtered out through operator fusion rules.

In a possible implementation, the parameters are the memory threshold and the peak memory threshold; there is no indirect path between the producer and the consumer corresponding to the first node set, and the memory occupied by the output tensor corresponding to the first node set higher than the memory threshold; and/or, there is an indirect path between the producer and consumer corresponding to the first node set, and the peak memory of each node between the producer and consumer corresponding to the first node set is higher than Peak memory threshold. In this way, the required node set can be filtered out by combining the memory threshold and the peak memory threshold, improving the efficiency and accuracy of screening.

In a possible implementation, the parameters are memory threshold and data mutation threshold; the memory occupied by the output tensor corresponding to the first node set is higher than the memory threshold, and/or, the output tensor corresponding to the first node set occupies The deviation value between the occupied memory and the memory occupied by at least one input tensor corresponding to the first node set is higher than the data mutation threshold. In this way, the required node set can be screened out by combining the memory threshold and the data mutation threshold, improving the efficiency and accuracy of screening.

In a possible implementation, the parameters are the memory threshold and the operator fusion rule; the first node set meets one or more of the following conditions: the memory occupied by the output tensor corresponding to the first node set is higher than the memory Threshold, or, the first node set and the consumers corresponding to the first node set can be fused, or, the first node set and the consumers corresponding to the first node set cannot be fused, and the producers and consumers corresponding to the first node set There is an indirect path between them. In this way, the required node set can be screened out through the combination of memory threshold and operator fusion rules, improving the efficiency and accuracy of screening.

In a possible implementation, the parameters are the peak memory threshold and the data mutation threshold; there is an indirect path between the producer and the consumer corresponding to the first node set, and there is an indirect path between the producer and the consumer corresponding to the first node set The peak memory of each node during execution is higher than the peak memory threshold; and/or, the difference between the memory occupied by the output tensor corresponding to the first node set and the memory occupied by at least one input tensor corresponding to the first node set The deviation value is higher than the data mutation threshold. In this way, the required node set can be screened out by combining the peak memory threshold and the data mutation threshold, improving the efficiency and accuracy of screening.

In a possible implementation, the parameters are memory thresholds and operator fusion rules; the first node set meets one or more of the following conditions: there is an indirect path between the producer and the consumer corresponding to the first node set , and the peak memory of each node between the producer and consumer corresponding to the first node set is higher than the peak memory threshold during execution, or the first node set and the consumer corresponding to the first node set can be fused, or, the first node set A node set and the consumer corresponding to the first node set cannot be merged, and there is an indirect path between the producer and the consumer corresponding to the first node set. In this way, the required node set can be screened out by combining the peak memory threshold and operator fusion rules, improving the efficiency and accuracy of screening.

In a possible implementation, the parameters are data mutation thresholds and operator fusion rules; the first node set meets one or more of the following conditions: the memory occupied by the output tensor corresponding to the first node set is the same as the first node set The deviation value between the memory occupied by at least one input tensor corresponding to a node set is higher than the data mutation threshold, or, the first node set and the consumers corresponding to the first node set can be fused, or, the first node set and The consumers corresponding to the first node set cannot be merged, and there is an indirect path between the producer and the consumer corresponding to the first node set. In this way, the required node set can be screened out by combining the data mutation threshold and operator fusion rules, improving the efficiency and accuracy of screening.

In a possible implementation, the parameters are memory threshold, peak memory threshold, and data mutation threshold; the first node set meets one or more of the following conditions: between the producer and the consumer corresponding to the first node set There is no indirect path, and the memory occupied by the output tensor corresponding to the first node set is higher than the memory threshold, or, there is an indirect path between the producer and the consumer corresponding to the first node set, and the first node set corresponds to The peak memory of each node between the producer and the consumer is higher than the peak memory threshold, or the memory occupied by the output tensor corresponding to the first node set is equal to the memory occupied by at least one input tensor corresponding to the first node set The deviation value between the memory is higher than the data mutation threshold. In this way, the required node set can be screened out by combining the memory threshold, peak memory threshold, and data mutation threshold, improving the efficiency and accuracy of screening.

In a possible implementation, the parameters are memory threshold, peak memory threshold and operator fusion rule; the first node set meets one or more of the following conditions: the relationship between the producer and consumer corresponding to the first node set There is no indirect path between, and the memory occupied by the output tensor corresponding to the first node set is higher than the memory threshold, or, there is an indirect path between the producer and the consumer corresponding to the first node set, and the first node set corresponds to The peak memory of each node between the producer and consumer is higher than the peak memory threshold, or the first node set and the consumer corresponding to the first node set can be merged, or the first node set and the first node set The consumers corresponding to the set cannot be merged, and there is an indirect path between the producer and the consumer corresponding to the first set of nodes. In this way, the required node set can be screened out by combining the memory threshold, peak memory threshold, and operator fusion rules, improving the efficiency and accuracy of screening.

In a possible implementation, the parameters are memory threshold, data mutation threshold and operator fusion rule; the first node set meets one or more of the following conditions: the output tensor corresponding to the first node set occupies The memory is higher than the memory threshold, or, the deviation value between the memory occupied by the output tensor corresponding to the first node set and the memory occupied by at least one input tensor corresponding to the first node set is higher than the data mutation threshold, or, The first node set and the consumers corresponding to the first node set can be fused, or the first node set and the consumers corresponding to the first node set cannot be fused, and there is an indirect relationship between the producer and the consumer corresponding to the first node set path. In this way, the required node set can be screened out by combining the memory threshold, data mutation threshold, and operator fusion rules, improving the efficiency and accuracy of screening.

In a possible implementation, the parameters are peak memory threshold, data mutation threshold and operator fusion rule; the first node set meets one or more of the following conditions: the producers and consumers corresponding to the first node set There is an indirect path between them, and the peak memory of each node between the producer and consumer corresponding to the first node set is higher than the peak memory threshold during execution, or the memory occupied by the output tensor corresponding to the first node set is the same as The deviation value between the memory occupied by at least one input tensor corresponding to the first node set is higher than the data mutation threshold, or, the first node set and the consumers corresponding to the first node set can be fused, or, the first node set Consumers corresponding to the first set of nodes cannot be fused, and there is an indirect path between producers and consumers corresponding to the first set of nodes. In this way, the required node set can be screened out by combining the peak memory threshold, data mutation threshold, and operator fusion rules, improving the efficiency and accuracy of screening.

In a possible implementation, the parameters are memory threshold, peak memory threshold, data mutation threshold and operator fusion rule; the first node set meets one or more of the following conditions: the producer corresponding to the first node set There is no indirect path between the consumer and the consumer, and the memory occupied by the output tensor corresponding to the first node set is higher than the memory threshold, or there is an indirect path between the producer and the consumer corresponding to the first node set, and the first The peak memory of each node between the producer and consumer corresponding to a node set is higher than the peak memory threshold during execution, or the memory occupied by the output tensor corresponding to the first node set is at least one of the corresponding to the first node set The deviation value between the memory occupied by the input tensor is higher than the data mutation threshold, or, the first node set and the consumers corresponding to the first node set can be fused, or, the consumption of the first node set and the first node set corresponding cannot be merged, and there is an indirect path between the producers and consumers corresponding to the first set of nodes. In this way, the required node set can be screened out through a combination of memory threshold, peak memory threshold, data mutation threshold, and operator fusion rules, improving the efficiency and accuracy of screening.

In a possible implementation, at least one first node set is obtained from producers included in the first computation graph based on parameters and data dependencies between nodes in the first computation graph, specifically including:

Based on the data dependencies between the nodes in the first calculation graph, the first list is obtained, and the first list includes the corresponding relationship between producers and consumers in the first calculation graph; according to the output of each producer in the first list Tensor, obtain the set of candidate nodes, the set of candidate nodes includes at least the first set of nodes, wherein, for any producer in the first list, the tensor output in any producer and with any producer A set of directly related and indirectly related nodes is used as a node set; based on parameters, the first node set is selected from the candidate node sets. In this way, the first set of nodes can be obtained from the producers in the first computation graph.

In a possible implementation manner, each first node set is recalculated to obtain N recalculated subgraphs, which specifically includes: for any node set in the N first node sets, copy any node The producer subgraph corresponding to the set; delete the nodes in the producer subgraph except the nodes contained in any node set, and delete the edges not related to any node set in the producer subgraph to obtain any node set The corresponding recomputation subgraph.

In a possible implementation manner, merging the second computation graph with the first computation graph to obtain the third computation graph specifically includes: respectively constructing the input of each recalculation subgraph in the N recalculation subgraphs The directed edge between the node of the first target tensor and the node outputting the first target tensor in the first computation graph, and the output second Directed edges between nodes of the target tensor and nodes of the input second target tensor in the first computation graph to obtain a third computation graph.

In a possible implementation, based on the parameters and the data dependencies between the nodes in the first computation graph, before obtaining the second computation graph that needs to be recalculated from the first computation graph, the method further includes: Operators in the calculation graph are fused. This improves the performance of operators in the calculation graph; in addition, through the first operator fusion, the number of operators can also be reduced, which can reduce the overhead of analyzing operators in the subsequent recalculation process; in addition, it can also expand In the subsequent recalculation process, the operator is analyzed to improve the effect of recalculation.

In a second aspect, the present application provides a calculation graph optimization device, including: at least one memory for storing programs; at least one processor for executing the programs stored in the memory, and when the programs stored in the memory are executed, the processor is used to Execute the method provided in the first aspect.

In a third aspect, the present application provides a device, including: at least one memory for storing a program; at least one processor for executing the program stored in the memory, and when the program stored in the memory is executed, the processor is used for executing the first methods provided in the aspect.

In a fourth aspect, the present application provides a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, and when the computer program runs on an electronic device, the electronic device executes the method provided in the first aspect.

In a fifth aspect, the present application provides a computer program product, which causes the electronic device to execute the method provided in the first aspect when the computer program product is run on the electronic device.

In a sixth aspect, the present application provides a chip, including at least one processor and an interface; the interface is used to provide program instructions or data for at least one processor; at least one processor is used to execute program instructions to implement the first aspect. provided method.

It can be understood that, for the beneficial effects of the above-mentioned second aspect to the sixth aspect, reference can be made to the related description in the above-mentioned first aspect, which will not be repeated here.

Description of drawings

The following briefly introduces the drawings used in the embodiments or the description of the prior art.

Fig. 1 is a schematic diagram of an artificial intelligence subject framework provided by an embodiment of the present application;

FIG. 2 is a schematic diagram of a training process of a network model of an AI network provided by an embodiment of the present application;

Fig. 3 is a schematic diagram of the training process of the network model of another AI network provided by the embodiment of the present application;

Fig. 4 is a schematic diagram of comparison before and after operator fusion provided by the embodiment of the present application;

Fig. 5 is a schematic diagram of peak memory comparison before and after operator fusion provided by the embodiment of the present application;

FIG. 6 is a schematic diagram of a graph-calculation fusion process under an AI open-source computing framework provided by an embodiment of the present application;

FIG. 7 is a schematic diagram of an operator fusion process provided by an embodiment of the present application;

Fig. 8 is a schematic diagram of topologically sorting operators in a calculation graph provided by an embodiment of the present application;

FIG. 9 is a schematic diagram of steps for screening recomputing node sets through memory provided by the embodiment of the present application;

FIG. 10 is a schematic diagram of another step of screening recomputing node sets through memory provided by the embodiment of the present application;

Fig. 11 is a schematic diagram of a calculation graph provided by the embodiment of the present application;

Fig. 12 is a schematic diagram of peak memory comparison before and after recalculation of the calculation graph provided by the embodiment of the present application;

Fig. 13 is a schematic diagram of the steps of screening recomputation node sets through operator fusion rules provided by the embodiment of the present application;

Fig. 14 is a schematic diagram of a process of determining a recalculation graph provided by an embodiment of the present application;

Fig. 15 is a schematic diagram of the process of merging the determined recalculation graph and the calculation graph obtained after operator fusion of the original computation graph provided by the embodiment of the present application;

FIG. 16 is a schematic diagram of an operator fusion process provided by an embodiment of the present application;

Figure 17 is a schematic diagram of the comparison of the memory size required before and after optimizing the calculation graph by means of operator fusion and recalculation provided by the embodiment of the present application;

Fig. 18 is a schematic flow chart of a calculation graph optimization method provided by an embodiment of the present application;

FIG. 19 is a schematic structural diagram of a chip provided by an embodiment of the present application.

Detailed ways

The term "and/or" in this article is an association relationship describing associated objects, which means that there can be three relationships, for example, A and/or B can mean: A exists alone, A and B exist simultaneously, and B exists alone These three situations. The symbol "/" in this document indicates that the associated object is an or relationship, for example, A/B indicates A or B.

The terms "first" and "second" and the like in the specification and claims herein are used to distinguish different objects, rather than to describe a specific order of objects. For example, the first response message and the second response message are used to distinguish different response messages, rather than describing a specific order of the response messages.

In the embodiments of the present application, words such as "exemplary" or "for example" are used as examples, illustrations or illustrations. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of the present application shall not be interpreted as being more preferred or more advantageous than other embodiments or design schemes. Rather, the use of words such as "exemplary" or "such as" is intended to present related concepts in a concrete manner.

In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more, for example, multiple processing units refer to two or more processing units, etc.; multiple A component refers to two or more components or the like.

For ease of understanding, the technical terms involved in this solution are introduced first.

(1) Graph computing fusion

Graph computing fusion is MindSpore's unique network performance optimization technology. It can automatically analyze and optimize the logic of the existing network computing graph, and combine the capabilities of the target hardware to optimize the computing graph by computing simplification and replacement, operator splitting and fusion, and operator specialization compilation to improve device computing resources. Utilization, to achieve overall optimization of network performance. Compared with traditional optimization technologies, graph computing fusion has unique advantages such as multi-operator cross-boundary joint optimization, cross-layer collaboration with MindAKG (Polyhedral-based operator compiler), and real-time compilation. In addition, graph-computing integration only requires the user to open the corresponding configuration, and the entire optimization process can be completed automatically, without additional perception by network developers, allowing users to focus on network algorithm implementation.

(2) Tensor

Tensor (Tensor) is a data structure used to represent data in the AI framework. It usually consists of shape (representing the size and dimension of data), data type (such as flaot32, int, etc.), memory address and other related attributes.

(3) operator

In the field of deep learning, an operator generally refers to a computing unit that can complete simple or complex calculations such as addition, subtraction, multiplication, and division, and is usually executed in units of operators on acceleration hardware.

(4) Memory multiplexing

Memory multiplexing is an optimization technique. In the deep learning network, the multiplexing of physical memory can be achieved by pointing Tensors of different layers to the same address, so as to achieve the purpose of reducing memory usage.

(5) Calculation graph

A computation graph is a computation function expressed by a directed graph with Operators as nodes. Computational graphs are mainly represented by nodes and edges. Nodes in a computation graph can represent operators. The edge between two operators in the computation graph can represent the relationship between the two operators. Among them, the edge can have a direction, and the direction of the edge can be the flow direction of data between operators. For example, when the edge between operator a and operator b in the calculation graph is from operator a to operator b, then the tensor output by operator a is the input of operator b. In the AI framework, the calculation function expressed by the calculation graph sequentially calls and executes the operator nodes in the directed graph to the input tensor, and obtains the final output tensor.

(6) Peak memory

Peak memory refers to the memory occupied by all the tensors that need to be used when an operator is executed during the execution of the calculation graph. For example, please refer to (A) in Figure 5. At the moment when operator D is executed, the tensors output by operator A, the tensors output by operator C, and the tensors to be output by operator D are all tensors that need to be used. Therefore, the peak memory at this time is T0+T1+T2; among them, the input tensor of operator B and the output tensor of operator C reuse one memory.

(7) Producer

A producer is a computing subgraph that generates data in two computing subgraphs with data dependencies in the computing graph, that is, an operator that generates data. For example, please refer to Figure 4, taking operator O ₁ and operator O ₂ as an example, operator O ₁ and operator O ₂ have a data dependency relationship, where the output tensor of operator O ₁ is the output tensor of operator O ₂ The input tensor, therefore, the operator O ₁ can be called a producer.

(8) Consumers

A consumer is a computation subgraph that consumes data in two computation subgraphs that have data dependencies in the computation graph, that is, an operator that consumes data. For example, please refer to Figure 4, taking operator O ₁ and operator O ₂ as an example, operator O ₁ and operator O ₂ have a data dependency relationship, where the output tensor of operator O ₁ is the output tensor of operator O ₂ The input tensor, therefore, the operator _O2 can be called a consumer.

(9) Node set

A node set refers to a node or a set of multiple nodes. The nodes contained in a node set are all directly or indirectly related to a tensor output by a producer in the calculation graph, and are directly or indirectly related to at least one tensor input by the producer. In addition, it can also be understood that when the node set includes multiple nodes, the calculation graph composed of multiple nodes can obtain at least one tensor from the outside and output a tensor to the outside, that is to say, among the multiple nodes At least one node can obtain tensors input from the outside, and only one node can output tensors to the outside, while the tensors output by other nodes can only be used inside the calculation graph.

For example, as shown in Figure 11, Gather, Scatter and Div can constitute a node set, and the calculation graph composed of these three nodes, in this calculation graph, Gather, Scatter and Div are connected in sequence, where Gather and Scatter There is an edge between Scatter and Div, and these three nodes are directly or indirectly related to the tensor B output by the corresponding producer FuseOp1, and are directly or indirectly related to the tensor A input by the producer FuseOp1; In other words, the calculation graph composed of these three nodes can obtain a tensor A from the outside through Gather, and can output a tensor B to the outside through Div. Neither Gather nor Scatter can output tensors to the outside of the calculation graph. The tensors they output can only be used inside the computation graph. As shown in Figure 14, in (B) of Figure 14, Gather can form a node set, and the calculation graph corresponding to the node set can be shown in Figure 14 (C). Quantity A, and a tensor B that can be output externally.

(10) Data dependencies between nodes

When the data required by one node needs to be provided by another node, there is a data dependency between the two nodes. For example, when a tensor output by one node is a tensor input by another node, there is a data dependency between the two nodes.

(11) Data mutation threshold

The data mutation threshold refers to the threshold at which at least one node in the calculation graph generates a data mutation. Taking a node as an example, the data mutation threshold is the threshold of the deviation value (such as difference or ratio, etc.) between the memory occupied by the tensor output by the node and the memory occupied by the tensor input by the node. Taking multiple nodes as an example, these multiple nodes can constitute the node set described above, and the data mutation threshold is the memory occupied by the output tensor corresponding to the node set formed by these multiple nodes and the node set The threshold of the deviation value (such as: difference or ratio, etc.) between the memory occupied by the corresponding at least one input tensor. Among them, the output tensor corresponding to the node set refers to the tensor that the calculation graph composed of the nodes in the node set outputs to the outside, and the input tensor corresponding to the node set refers to the calculation graph composed of the nodes in the node set. Tensor obtained externally.

Next, the technical solutions in the embodiments of the present application will be described.

Exemplarily, FIG. 1 shows a schematic diagram of an artificial intelligence main framework, which describes the overall workflow of an artificial intelligence system, and is applicable to general artificial intelligence field requirements.

The following is an elaboration on the above artificial intelligence theme framework from the two dimensions of "intelligent information chain" (horizontal axis) and "IT value chain" (vertical axis).

"Intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it can be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, intelligent execution and output. In this process, the data has undergone a condensed process of "data-information-knowledge-wisdom".

"IT value chain" reflects the value brought by artificial intelligence to the information technology industry from the underlying infrastructure of artificial intelligence, information (provided and processed by technology) to the systematic industrial ecological process.

(1) Infrastructure:

The infrastructure provides computing power support for the artificial intelligence system, realizes communication with the outside world, and realizes support through the basic platform. Communicate with the outside through sensors; computing power is provided by smart chips (CPU, NPU, GPU, ASIC, FPGA and other hardware acceleration chips); the basic platform includes distributed computing framework and network and other related platform guarantees and supports, which can include cloud storage and Computing, interconnection network, etc. For example, sensors communicate with the outside to obtain data, and these data are provided to the smart chips in the distributed computing system provided by the basic platform for calculation.

(2) data

Data from the upper layer of the infrastructure is used to represent data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data of traditional equipment, including business data of existing systems and sensory data such as force, displacement, liquid level, temperature, and humidity.

(3) Data processing

Data processing usually includes data training, machine learning, deep learning, search, reasoning, decision-making, etc.

Among them, machine learning and deep learning can symbolize and formalize intelligent information modeling, extraction, preprocessing, training, etc. of data.

Reasoning refers to the process of simulating human intelligent reasoning in a computer or intelligent system, and using formalized information to carry out machine thinking and solve problems according to reasoning control strategies. The typical functions are search and matching.

Decision-making refers to the process of decision-making after intelligent information is reasoned, and usually provides functions such as classification, sorting, and prediction.

(4) General ability

After the above-mentioned data processing is performed on the data, some general capabilities can be formed based on the results of data processing, such as algorithms or a general system, such as translation, text analysis, computer vision processing, speech recognition, image processing identification, etc.

(5) Smart products and industry applications

Intelligent products and industry applications refer to the products and applications of artificial intelligence systems in various fields. It is the packaging of the overall solution of artificial intelligence, which commercializes intelligent information decision-making and realizes landing applications. Its application fields mainly include: intelligent manufacturing, intelligent transportation, Smart home, smart medical care, smart security, automatic driving, safe city, smart terminals, etc.

It should be noted that the operator fusion process involved in this application can be located in the data processing stage in (3) above.

Exemplarily, FIG. 2 shows a training process of a network model of an AI network. The training process of the network model may include a forward calculation process and a backward calculation process. As shown in Figure 1, the part on the left side of the dotted line x is the forward calculation process, and the part on the right side of the dotted line x is the backward calculation process, and each block in the figure represents a tensor. In Figure 1, tensors in the same row reuse the same memory address, for example: tensor A and tensor A _b reuse a memory address, tensor B and tensor B _b reuse a memory address, ..., tensor F and tensor F _b multiplex a memory address etc. In addition, in Figure 1, tensor A _b , tensor B _b , tensor C _b , tensor D _b , tensor E _b , and tensor F _b are respectively related to tensor A, tensor B, tensor C, Reverse operator data corresponding to tensor D, tensor E, and tensor F. In Fig. 1, in the training process of the network model, the forward calculation can be performed first, and then the backward calculation can be performed. Among them, the tensor obtained by the forward calculation will be stored in the memory of the corresponding computing device. In the backward calculation process, the calculation of tensor A _b depends on tensor In and tensor B _b , the calculation of tensor B _b depends on tensor A and tensor C _b , and the calculation of tensor C _b depends on tensor Quantity B and tensor D _b , the calculation of tensor D _b depends on tensor C and tensor E _b , etc., where, in the backward calculation process, after a tensor is calculated, the tensor will be released The memory of the tensor obtained by the corresponding forward calculation, for example, after the calculation of the tensor B _b , the memory occupied by the tensor A is released, so that the tensor A _b obtained by the subsequent calculation can reuse the tensor A occupied memory. It can be seen that during the training process of the network model shown in FIG. 1 , the data obtained by the forward calculation will occupy memory for a long time, which will lead to excessive memory consumption of the network model.

Generally, in Figure 1, recomputation or operator fusion can be used to solve the problem of memory usage. Among them, when the recalculation method is used, the time-for-space method is adopted in Figure 1, and the memory occupied by the part of the results obtained by the forward calculation in Figure 1 is released in advance, and this part of the results is processed in the subsequent backward calculation. Perform recalculation to shorten the life cycle of this part of the data, thereby achieving the purpose of reducing memory usage. Specifically, as shown in FIG. 3 , the tensor A obtained by the forward calculation may continue to occupy memory before the tensor B _b is obtained by the backward calculation. The tensor D obtained by the forward calculation can continue to occupy memory before the tensor E _b is obtained by the backward calculation. Tensor B, Tensor C, Tensor E, and Tensor F obtained from the forward calculation. The memory occupied by these four tensors can be released immediately after being used in the forward calculation process, and this can be done during the reverse calculation. The calculation process corresponding to the four tensors is re-executed. Among them, the memory occupied by tensor B can be released after the tensor C is obtained by the forward calculation, and the memory occupied by the tensor C can be released after the tensor D is obtained by the forward calculation. When the tensor D _b is calculated backward, it can be re- The tensor C is calculated, and when the tensor C _b is calculated backward, the tensor B can be recalculated. In this way, the purpose of reducing memory usage can be achieved by recalculating. Continuing to refer to Figure 1, if each block occupies one memory in Figure 1, and the forward calculation ends when the tensor F is obtained, the data obtained after the forward calculation ends takes up 7 memory in total; while in Figure 3 Among them, since the memory occupied by tensor B, tensor C, tensor E and tensor F will be released immediately after being used in the forward calculation process, these four tensors can be considered as not occupying memory , so the data obtained after the forward calculation in Figure 3 occupies a total of 3 memories. It can be seen that the purpose of reducing memory usage can be achieved by recalculating.

However, when using the recalculation method, it is often necessary to manually specify the recalculation point or automatically calculate the recalculation point. Among them, the manual recalculation point solution often requires the user to identify the recalculation segmentation point, which requires high user capabilities, and needs to be set separately for different models and different hardware backends, which is inefficient; the automatic calculation recalculation point solution takes time to solve. Long (hour-level solution time, more than most training tasks themselves), or need to be counted at runtime, and cannot be combined with operator fusion. In addition, the recalculation method can only be recalculated for data dependencies between forward and reverse, and cannot be recalculated for data dependencies between forward and forward, and reverse and reverse, and the usage scenarios are relatively small. In addition, the method of recalculation cannot solve the problem of memory occupied by a single operator and a single super large tensor. For example, continue to refer to Figure 3, if the data volume of the tensor A is too large, the memory occupied by it will be large, and at this time, the memory will continue to be occupied by a large amount.

When using operator fusion, it can improve network performance by fusing multiple adjacent operators into one operator, and can also reduce memory usage overhead. Specifically, as shown in Figure 4, (A) in Figure 4 is the calculation method before fusion, wherein, tensor A is input to operator _O1 to obtain tensor B, and tensor B is input to operator _O2 to obtain Tensor C, tensor C is input to operator O ₃ to obtain tensor D. Operators O ₁ , O ₂ and O ₃ can be fused by using the preset fusion rules, and these three operators can be fused into one operator. At this time, tensor D obtained from tensor A is transformed into Figure 4 The way shown in (B). In (B) of FIG. 4 , tensor D can be obtained after inputting tensor A to the fused operator (O ₁ +O ₂ +O ₃ ). Comparing (A) and (B) in Figure 4, it can be seen that there are 4 tensors that need to occupy memory in (A) in Figure 4, and 2 tensors that need to be occupied in (B) in Figure 4 Memory. It can be seen that using operator fusion can reduce memory usage.

However, limited by the internal storage size and hardware processing capability, operator fusion can only process operators in adjacent spaces, and the scope and scale of its fusion are limited. However, there are a large number of calculation processes between the forward and reverse operators, which cannot be directly fused, and the results of the forward calculation still need to be temporarily stored in the memory. In addition, in some multi-input scenarios, operator fusion will extend the life cycle of different tensors to the entire fusion operator, which instead increases the memory usage. Exemplarily, as shown in Figure 5, (A) in Figure 5 is the execution sequence of each operator before operator fusion, the life cycle of each operator's input tensor and output tensor, and each operator's Peak memory during execution, (B) in Figure 5 shows the execution order of each operator after operator fusion, the life cycle of each operator's input tensor and output tensor, and the peak value of each operator during execution Memory. Wherein, in FIG. 5 , operator A, operator B, and operator C may be fused into operator E. In (A) of Figure 5, the tensor output by operator B and the tensor input by operator A share one memory, the tensor output by operator C and the tensor input by operator B share one memory, and the tensor output by operator D The tensor and the tensor input by operator C share a single memory. In (B) of Figure 5, the tensor output by operator D and a tensor input by operator E can reuse one memory. When operator C is executed, it is equivalent to the execution of operator E at this time, so At this time, memory T0, T1, and T2 cannot be reused, and only one more memory, that is, memory T3, can be applied. From (A) in Figure 5, it can be seen that before operator fusion, the peak memory when using the calculation graph is (T0+T1+T2), and from Figure 5 (B), after operator fusion, when using the calculation graph The peak memory of is (T0+T1+T2+T3). It can be seen that the peak memory after operator fusion is significantly larger than the peak memory before operator fusion.

From the above analysis, it can be seen that although the method of recomputation or operator fusion can solve the problem of part of the memory usage, it still cannot solve the problem of high peak memory usage caused by tensors with long life cycles spanning long path dependencies. The problem that cannot be executed, and the problem that a single operator and a single tensor exceeds the memory limit causes the network to fail to execute.

In view of this, the present application provides a calculation graph optimization method, which mainly obtains a recalculation graph that needs to be recalculated from the calculation graph to be optimized, and merges the obtained recalculation graph with the calculation graph to be optimized to generate A new calculation graph, and finally perform operator fusion on the new calculation graph. Therefore, using the combination of recalculation and operator fusion, while significantly reducing memory usage, without introducing large recalculation overhead (including compile time and runtime), it solves the problem of having one or more super large tensors. The network cannot execute the problem.

Exemplarily, FIG. 6 shows a schematic diagram of a graph-computing fusion process under an AI open-source computing framework. The AI open source computing framework shown in FIG. 6 may be MindSpore. As shown in Figure 6, the AI open source computing framework may include a front-end presentation layer (not shown in the figure), a MindSpore layer, and a MindSpore Auto Kernel Generator layer (MindSpore AGK layer). Wherein, the front-end presentation layer may provide at least one application programming interface (API) interface for the user. When using the AI open source computing framework, users can select the desired network model 401 at the front-end presentation layer, and configure various data through the front-end presentation layer. Afterwards, the network model can be constructed in the MindSpore layer and automatically differentiated to generate the user-configured MindSpore Expressio (ME) front-end representation. Then, in the MindSpore layer, a calculation graph can be automatically constructed based on the user-configured ME front-end representation. Then, public optimization can be performed on the calculation graph first, such as: eliminating unnecessary code, optimizing the same data that occurs multiple times, and so on. Then, perform back-end optimization on the calculation graph, for example, select a more suitable expression method according to the characteristics of different hardware. After back-end optimization of the calculation graph, an optimized calculation graph can be generated. Among them, the process of obtaining the optimized calculation graph can be understood as a graph-computing fusion process. The calculation graph optimized by graph computing fusion can contain several operator definition descriptions, which describe the calculation process of each operator in the calculation graph. At the MindSpore AGK layer, the json format can be passed to Mind AKG for compilation and optimization, and the code generator is called to generate the back-end hardware code and compiled into the back-end Kernel. Among them, the back-end Kernel can be understood as the expression of the operator on the hardware, such as the binary file corresponding to the operator. After compiling the layers and operators, the MindSpore layer can call the Kernel in the order of the calculation graph and execute the corresponding calculation graph.

Among them, this application mainly focuses on the back-end optimization part of the MindSpore layer. The back-end optimization part in the MindSpore layer can reduce the memory consumption of the network model through recalculation and operator fusion processing, so that it can be used for different types of hardware back-ends (such as graphics processors, neural network processing, etc.) processor, central processing unit, etc.) to provide performance and memory optimization.

The calculation graph optimization scheme provided by this application will be described in detail below based on the MindSporeAI open source computing framework described above and in conjunction with the accompanying drawings. It can be understood that the MindSporeAI open source computing framework described in the embodiment of this application is only a schematic illustration, and the calculation graph optimization method provided in the embodiment of this application is not limited to this framework, and can also be applied to other frameworks, where , still within the protection scope of the present application when applied to other frameworks.

Exemplarily, the calculation graph optimization solution may be run on any device, device, platform or device cluster with computing and processing capabilities, but not limited to. Among them, the computation graph optimization scheme mainly includes two steps of setting configuration item interface and computing graph optimization. Setting the configuration item interface is mainly for the user to configure some necessary parameters for subsequent use in the calculation graph optimization process; calculation graph optimization is mainly a process of combining operator fusion and recomputation, see the description below for details.

(1) Setting configuration item interface

MindSpore can be configured with a Context module and a graph-calculation fusion module. The Context module can be used to receive the user's global settings for the running process. The graph-calculation fusion module can, but is not limited to, be responsible for the graph-computation fusion of calculation graphs and operators. Among them, the graph-calculation fusion module can be, but not limited to, configured in the AI compiler module in MindSpore, and the AI compiler module can be responsible for compiling and optimizing the calculation graph and operators.

In this embodiment of the application, two configuration item interfaces can be added in the Context module, but not limited to. One configuration item interface can be used to configure the memory threshold and/or peak memory threshold, and the other configuration item interface can be used to configure the data threshold. Among them, users can set the values of the above two configuration items according to the actual network and dataset scenarios. When the user does not set it, the default value of the system can be used, or it can be automatically generated according to the hardware environment, for example: the memory size of the graphics processing unit (graphics processing unit, GPU) can be used as the memory threshold and/or the peak memory threshold.

It can be understood that the interface for setting configuration items can be set in real time or in advance, which can be determined according to actual conditions, and is not limited here.

(2) Calculation graph optimization

The process of computing graph optimization is mainly a process of combining operator fusion and recomputation. This process can be executed, but not limited to, in the graph-computing fusion module configured in MindSpore. Wherein, the graph computing fusion module can read the memory threshold and/or peak memory threshold and data threshold configured by the user in the Context module. The calculation graph optimization process mainly includes primary operator fusion, recomputation, and secondary operator fusion. See the description below for details.

(1) One operator fusion

When optimizing the calculation graph, operator fusion can be performed on the obtained calculation graph first. Exemplary, but not limited to, based on preset operator fusion rules (such as: elementwise+elementwise operator fusion rules, elementwise+reduction operator fusion rules, segment+elementwise operator fusion rules, etc.), and According to the category of operators in the calculation graph (for example: computation-intensive, memory-intensive, etc.) An operator obtained after fusion (hereinafter referred to as "fusion operator"), thereby obtaining an optimized calculation graph, that is, a primary fusion subgraph. This improves the performance of operators in the calculation graph. In addition, through the first operator fusion, the number of operators can also be reduced, which can reduce the overhead of analyzing operators in the subsequent recalculation process. In addition, it can also expand the scope of analyzing operators in the subsequent recalculation process to improve the effect of recalculation. For example, please refer to Figure 11. Assuming that there is no data mutation of operator Div before operator fusion, and after operator fusion, the output tensor B corresponding to FuseOp1 is much larger than its corresponding input tensor A, then in operator fusion Previously, it was difficult to use the operator Div as a recomputation node set in the way of data mutation, but after the operator fusion, the fusion operator FuseOp1 corresponding to the operator Div has a data mutation, so the output tensor B of the operator Div can be combined with A set of related operators is used as a set of recomputing nodes, so that operator fusion can expand the scope of analyzing operators in the process of recomputing.

For example, as shown in Figure 7, (A) in Figure 7 is the initially obtained calculation graph, which includes 6 operators, and these 6 operators are Slice, Gather, ScatterAdd, MatMul, Mul and Div. In (A) of Figure 7, the output of Slice is tensor A; the input of Gather is tensor A, and the output is tensor B; the input of ScatterAdd is tensor B, and the output is tensor C; the input of MatMul is tensor Quantity C, the output is tensor D; the input of Mul is tensor D, and the output is tensor E; the input of Div is tensor B and tensor D. If the pre-set operator fusion rules are: Gather and ScatterAdd fusion, Mul and Div fusion, then based on the operator fusion rules (A) in Figure 7, the operator fusion can be obtained as shown in Figure 7 (B). The calculation graph shown. (B) in FIG. 7 includes 4 calculation subgraphs, and these 4 calculation subgraphs correspond to 4 fusion operators, respectively: Slice, FuseOp1, MatMul and FuseOp2. In (B) of Figure 7, the output of Slice is tensor A; the input of FuseOp1 is tensor A, and the output is tensor B and tensor C; the input of MatMul is tensor C, and the output is tensor D; FuseOp2 The input of is tensor D and tensor B. It can be understood that since (B) in FIG. 7 is a calculation graph obtained after operator fusion, each operator in (B) in FIG. 7 can be called a "fusion operator".

It can be understood that, in the embodiment of the present application, one operator fusion can be selected according to the actual situation. In some embodiments, a subsequent recalculation step may be performed after performing this step. In some other embodiments, this step may not be performed, but the subsequent recalculation step is directly performed.

(2) Recalculation

The recalculation process mainly includes building a producer-consumer list, determining a recalculation candidate set, screening a recalculation point set from the recalculation candidate set, determining a recalculation graph, and generating one or more items in a new calculation graph, See description below for details.

A) Build a producer-consumer list

According to the data dependencies between the nodes in the calculation graph, the nodes in the calculation graph are topologically sorted first, for example, sorted according to the physical relationship between operators in the calculation graph, etc., and then the producer- list of consumers. Among them, the producer is the calculation subgraph that generates data in the two calculation subgraphs with data dependencies, that is, the operator that generates data, and the consumer is the calculation subgraph that uses data in the two calculation subgraphs with data dependencies , which is an operator using data.

For example, as shown in FIG. 8 , (A) in FIG. 8 is a calculation subgraph obtained after one fusion. Based on the data dependencies among the nodes in the calculation graph, each node in the calculation graph shown in (A) of FIG. 8 is topologically sorted, and the sorting result shown in (B) of FIG. 8 can be obtained. In (B) of FIG. 8 , Slice and FuseOp1 have a data dependency, FuseOp1 has a data dependency with MatMul and FuseOp2 respectively, MatMul and Div have a data dependency, and Div and FuseOp2 have a data dependency. Among them, for Slice and FuseOp1, Slice can generate data, and FuseOp1 can use the data generated by Slice, so Slice is the producer, and FuseOp1 is the consumer. Similarly, for FuseOp1 and MatMul, FuseOp1 is the producer, and MatMul is the consumer; for FuseOp1 and FuseOp2, FuseOp1 is the producer, and FuseOp2 is the consumer; for MatMul and Div, MatMu is the producer, and Div is the consumer; for Div and FuseOp2, Div is the producer and FuseOp2 is the consumer. In this way, the producer-consumer list corresponding to the computation graph in (A) in FIG. 8 can be obtained from (B) in FIG. 8 , and the list can be shown in Table 1.

Table 1

producer consumer Slices FuseOp1 FuseOp1 MatMul FuseOp1 FuseOp2 MatMul div div FuseOp2

B) Determine the set of recalculation candidates

After the producer-consumer list is constructed, for any producer in the list, a set of operators in the producer and directly and indirectly related to the tensor output by the producer can be used as recomputation candidates A piece of data in the set, which can be called a set of recomputing nodes. Thus, a recomputation candidate set is obtained. Wherein, the set of recomputing nodes may include one node, or may include multiple nodes.

For example, continue to refer to Figure 8, taking the producer FuseOp1 as an example, the tensors output by the producer are tensor B and tensor C respectively. The node directly related to tensor B is Gather, and there is no node indirectly related to tensor B. The node directly related to tensor C is ScatterAdd, and the node indirectly related to tensor C is Gather. Therefore, the set of recomputation candidates obtained by the producer FuseOp1 is [(Gather), (Gather, ScatterAdd)]. The recalculation candidate combination includes two recalculation node sets, one recalculation node set is (Gather), which includes a node, that is, node Gather, and the other recalculation node set is (Gather, ScatterAdd), which includes two nodes, namely nodes Gather and ScatterAdd.

C) Filter the recalculation node set from the recalculation candidate set

The method of screening the recalculation point set from the recalculation candidate set may include one or more methods of screening based on memory size, screening based on data mutation, and screening based on operator fusion rules. Among them, one of these three ways can be selected, or can be combined arbitrarily, which can be determined according to actual conditions, and is not limited here. The three methods are described below.

a) Filter based on memory size

When filtering based on memory size, you can filter out the required recalculation node set corresponding to the producer from the recalculation candidate set according to the memory size occupied by the tensors output by each recalculation node set in the recalculation candidate set . Specifically, as shown in FIG. 9 , at S901 , the memory occupied by the output tensor corresponding to each recalculation node set in the recalculation candidate set may be compared with the previously set memory threshold or peak memory threshold. In S902, when the memory occupied by the output tensor corresponding to the recalculation node set is greater than or equal to the memory threshold or the peak memory threshold set in the previous period, it is determined that the recalculation node set is the required recalculation node set. At this time, it can be The set of recompute nodes is preserved. In S903, when the memory occupied by the output tensor corresponding to the recalculation node set is less than the previously set memory threshold or peak memory threshold, it is determined that the recalculation node set is not the required recalculation node set, and the recalculation node set can be discarded at this time Recalculate the set of nodes. Thus, the required recalculation point set is filtered out from the recalculation candidate set. In one example, the output tensor corresponding to the recalculation node set refers to a tensor output by the producer corresponding to the recalculation node set, and each node in the recalculation node set is directly related to the tensor or indirectly related. The input tensor corresponding to the recalculation node set refers to at least one tensor input by the producer corresponding to the recalculation node set, and each node in the recalculation node set is related to each tensor in the at least one tensor directly or indirectly related.

As another possible implementation, in the process of obtaining the producer-consumer list, it is possible to record whether there is an indirect path between the producer-consumer, that is, record the tensor output by the producer except the tensor directly used as the consumer Whether the input tensor can also be indirectly used as the input tensor of this consumer. For example, continue to refer to Figure 8, for the producer-consumer FuseOp1-FuseOp2, the tensor B output by the producer FuseOp1 can be directly used as the input tensor of the consumer FuseOp2, and the tensor C output by the producer FuseOp1 can be passed MatMul and Div indirectly serve as tensors for input to FuseOp2, so there is an indirect path between FuseOp1 and FuseOp2.

Wherein, for any producer-consumer, as shown in FIG. 10, at S1001, it may be determined whether an indirect path exists between the producer-consumer. When there is no indirect path between the producer and the consumer, according to the memory size occupied by the output tensor corresponding to each recomputation node set corresponding to the producer in the recalculation candidate set, the recalculation candidate set Filter out the required set of recomputing nodes corresponding to the producer. Specifically, in S1002, the memory occupied by the output tensor corresponding to each recomputation node set corresponding to the producer in the recomputation candidate set may be compared with the previously set memory threshold or peak memory threshold. In S1003, when the memory occupied by the output tensor corresponding to the recalculation node set is greater than or equal to the memory threshold set in the previous period, it is determined that the recalculation node set is the required recalculation node set, and the recalculation node set can be reserved at this time collection of nodes. In S1004, when the memory occupied by the output tensor corresponding to the recalculation node set is less than the previously set memory threshold or peak memory threshold, it is determined that the recalculation node set is not the required recalculation node set, and the recalculation node set can be discarded at this time Recalculate the set of nodes. Thus, the required recalculation point set is filtered out from the recalculation candidate set.

For example, continue to refer to Figure 8, for the producer-consumer MatMul-Div, there is no indirect path between the two. If the memory threshold set in the early stage is 40 bytes, the memory occupied by the tensor output by the producer MatMul is 50 bytes. Since there is only a direct path between MatMul and Div, and there is no indirect path, and the memory occupied by the tensor output by the producer MatMul is larger than the memory threshold set in the previous stage, MatMul can be used as the required set of recomputing nodes.

Exemplarily, for determining the memory occupied by the tensor output by the producer, it may be determined based on, but not limited to, the data size of the tensor output by the producer. Among them, after the calculation graph is determined, the structure (such as: dimension, type) of the tensor output by each operator in the calculation graph can be determined. Therefore, according to the structure of the tensor output by each operator, etc., Determine the memory size occupied by tensors output by each operator. For example, if the dimension of the tensor is [4,4,3,2] and the type is float32, since each data type of float32 occupies 4 bytes, the memory size occupied by the tensor is: 4x4x3x2x4 = 384 bytes.

Continuing to refer to FIG. 10 , at S1005 , when there is an indirect path between the producer and the consumer, the peak memory of each operator on the indirect path between the producer and the consumer may be calculated separately during execution. Next, at S1006, it is determined whether there is at least one operator whose peak memory is greater than or equal to the previously set peak memory threshold. Next, in S1007, when there is at least one operator whose corresponding peak memory is greater than or equal to the previously set peak memory threshold, the producer-consumer can be directly related to each tensor input from the producer to the consumer The set of operators and indirectly related operators are all used as a set of required recomputing nodes. In S1008, when the peak memory corresponding to each operator on the indirect path between the producer and the consumer is smaller than the previously set peak memory threshold, the producer-consumer and the producer-consumer input can be The set of directly related and indirectly related operators of the tensor is not the required set of recomputing nodes, and these sets of recomputing nodes are discarded. Thus, the required recalculation point set is filtered out from the recalculation candidate set.

For example, as shown in FIG. 11 , FuseOp1 and FuseOp2 in FIG. 11 can form a producer-consumer. There is an indirect path between FuseOp1 and FuseOp2, and the operator on this indirect path is MatMul. If the memory when MatMul is executed (that is, the sum of the memory occupied by tensor B, tensor C, tensor D, and tensor E) is greater than the peak memory threshold set in the previous stage, it can be directly related to tensor B and indirectly The collection of related operators, and the collection of operators directly and indirectly related to tensor E are used as the required recomputation node collection. Among them, the set of operators directly and indirectly related to tensor B is Gather, and the set of operators directly and indirectly related to tensor E is (Gather, ScatterAdd, Div), so the recalculation selected at this time The node sets are (Gather) and (Gather, ScatterAdd, Div) respectively.

In some embodiments, in order to improve the accuracy of screening based on the peak memory threshold, the producer-consumer and the tensor input from the producer to the consumer (that is, the tensor output by the producer) can be combined during screening. The size between the memory occupied by the producer and the memory occupied by the input tensor related to the tensor required by the producer is used as the judgment condition of the attachment. Among them, when there is an indirect path between the producer and the consumer, when the tensor input from the producer to the consumer (that is, the tensor output by the producer) in the producer-consumer occupies more memory than the producer The required memory occupied by the input tensor related to the tensor, and the peak memory of each operator on the indirect path between the producer and the consumer during execution has at least one peak memory corresponding to the operator When it is greater than or equal to the peak memory threshold set in the previous period, the set of operators directly and indirectly related to each tensor input from the producer to the consumer in the producer-consumer can be used as a required Recalculate the set of nodes.

For example, as shown in (A) of Figure 12, for the pair of producers and consumers of operator 1 and operator 5, there is an indirect path between them, and the peak memory of operator 4 is the highest when it is executed , the peak memory at this time is (T1+T2+T3+T4). If the peak memory of operator 4 exceeds the peak memory threshold set in the previous stage, then operator 1 can be used as the required set of recomputing nodes. As shown in (B) of Figure 12, when operator 1 is used as the required recomputation node set, and the peak memory after recomputation is (T0+T2+T3+T4), if T1 is greater than T0, then At this time, after recalculation, the peak memory corresponding to the calculation graph can be reduced. If T1 is less than T0, the peak memory corresponding to the calculation graph will be increased after recalculation. If T1 is equal to T0, then after recalculation at this time The peak memory corresponding to the calculation graph has not changed, but the system overhead has been increased due to recalculation. Therefore, when T1 is greater than T0, operator 1 can be used as the required set of recomputing nodes.

b) Mutation screening based on data

When screening based on data mutation, it can be determined whether there is data mutation in the recalculation node set in the recalculation candidate set. If the deviation value (such as difference, ratio, etc.) between the size of the memory occupied by the output tensor corresponding to the recalculation node set and the size of the memory occupied by at least one input tensor corresponding to it is greater than or equal to the data set in the previous period Threshold, indicating that there is a data mutation at this time, then keep the set of recalculation points, otherwise discard the set of recalculation points. In this way, the situation of excessive memory usage caused by data mutation is eliminated.

For example, continue to refer to FIG. 11 , if the set of recalculation points in the recalculation candidate set is [(Gather), (Gather-ScatterAdd-Div)]. If the memory occupied by tensor E minus the memory occupied by tensor A is greater than the previously set data threshold, it indicates that there is a data mutation in the recalculation point set Gather, and the recalculation point set can be retained at this time. If the memory occupied by tensor B minus the memory occupied by tensor A is less than the previously set data threshold, it indicates that there is no data mutation in the recalculation point set (Gather-ScatterAdd-Div), and the recalculation point can be discarded at this time gather. The set of recalculation points thus filtered out is (Gather).

For the case of eliminating excessive memory usage through data mutation, for example, as shown in Figure 12, (A) in Figure 12 is the execution order of each operator before recomputation, the input tensor of each operator and The life cycle of the output tensor and the peak memory of each operator during execution, (B) in Figure 12 shows the execution order of each operator after recalculation, the input tensor and output tensor of each operator The lifetime and peak memory of each operator during execution. Comparing (A) and (B) in Figure 12, when operator 4 is executing, the peak memory before recalculation is T1+T2+T3+T4, and the peak memory after recalculation is T0+T2+T3 +T4. If the value obtained by subtracting T0 from T1 is greater than the previously set data threshold, and operator 1 is a recomputation node set in the determined recomputation candidate set, at this point, it can be seen that when operator 4 is executing, The peak memory after recalculation is significantly smaller than the peak memory before recalculation. Therefore, in this case, the peak memory can be significantly reduced. Therefore, operator 1 can be used as a set of filtered recalculation nodes at this time. If the value obtained by subtracting T0 from T1 is less than the previously set data threshold, and operator 1 is a set of recalculation nodes in the determined recalculation candidate set, at this time, it can be seen that when operator 4 is executing, The difference between the peak memory after recalculation and the peak memory before recalculation is small, so in this case, operator 1 can be used as the selected recalculation node set, or operator 1 can not be selected as the recalculation node set gather.

c) Screening based on operator fusion rules

When screening the recomputation node set based on the operator fusion rule, as shown in (A) of Figure 13, at S1311, based on the preset operator fusion rule, it can be determined whether the recomputation node set in the recomputation candidate set is the same as Whether the consumers corresponding to the set of recomputing nodes can be merged. In S1312, if it can be merged, the set of recomputing nodes is retained. In S1313, if the fusion cannot be performed, the recalculation node set may be discarded. In an example, S1311 can also be understood as judging whether a node that outputs tensors in the recalculation node set and its corresponding consumer can be fused.

As another possible implementation, as shown in (B) of Figure 13, at S1321, based on the preset operator fusion rules, it is possible to determine the recomputation node set in the recomputation candidate set and the recomputation node set Whether the consumers corresponding to the set can be merged. In S1322, if it can be merged, the set of recomputing nodes is retained. In S1323, if the integration is not possible, it may be determined whether there is an indirect path between the producer and the consumer corresponding to the set of recomputing nodes. Wherein, when there is an indirect path, the set of recomputing nodes may be reserved, that is, S1322 is executed. At S1324, when there is no indirect path, the recomputation node combination may be discarded. The set of recomputing nodes is thus filtered out. Among them, when the operator fusion rules are not met, but there is an indirect path, recalculation can still shorten the life cycle of tensors that occupy too much memory, thereby bringing the possibility of reducing the peak memory benefit. In an example, S1321 can also be understood as judging whether a node that outputs tensors in the recalculation node set and its corresponding consumer can be fused.

For example, continue referring to FIG. 12 , assuming that the determined recomputation node set in the recomputation candidate set is operator 1, and operator 1 and operator 5 cannot be merged. Since operator 1 and operator 5 do not comply with the fusion rules, but there is an indirect path between them, operator 1 can be used as the required recomputation node set. Comparing (A) and (B) in Figure 12, if the memory T1 occupied by the tensor output by operator 1 is too large, then in (B) in Figure 12, the tensor output by operator 1 can be greatly shortened. Life cycle, at this time the peak memory corresponding to the entire calculation graph has also been significantly reduced.

D) Determine the recalculation graph

After the recomputation node set is screened out, a recomputation graph can be determined from the determined recomputation node set. Exemplarily, an edge between a recalculation node set and a consumer corresponding to a producer to which the recalculation node set belongs may be used as a recomputation point in the calculation graph. Then, copy the producer subgraph corresponding to each recomputation point, wherein the number of the obtained producer subgraph is equal to or greater than the number of the filtered recalculation node set. Finally, delete all the nodes in the producer subgraph except for the recalculation node set corresponding to the producer subgraph, and delete the edges in the producer subgraph that are not related to the recalculation node set corresponding to the producer subgraph , so that the recomputation graph corresponding to each recomputation node set is obtained. Since the recomputation graph obtained in this way has a very small probability of duplication between recomputation node sets, the probability of recomputation is reduced, thereby reducing the amount of calculation and saving recomputation overhead. In one example, deleting all nodes in the producer subgraph except the recomputation node set corresponding to the producer subgraph can be understood as deleting the tensor output from the producer subgraph and the recalculation node set irrelevant operators.

In one example, for any set of recomputing nodes, the producers to which the set of recomputing nodes belong can also be directly copied in the computation graph, so that the producer subgraph corresponding to the set of recomputing nodes can be obtained. Then, delete all the nodes in the producer subgraph except for the recalculation node set corresponding to the producer subgraph, and delete the nodes in the producer subgraph that are not related to the recalculation node set corresponding to the producer subgraph edge, so that the recomputation graph corresponding to each recomputation node set is obtained.

For example, as shown in FIG. 14 , (A) in FIG. 14 is a fused subgraph obtained by performing operator fusion on the initial calculation graph. If the memory occupied by tensor B in (A) of FIG. 14 is higher than the memory threshold configured by the user, then the set of recalculation nodes can be determined as Gather, and the edge corresponding to tensor B can be determined as the recalculation point. Next, the producer subgraph corresponding to the recomputation point can be copied, that is, FuseOp1 can be copied to obtain the fusion operator shown in (B) of FIG. 14 . Wherein, copying the producer subgraph corresponding to the recalculation point can be understood as copying the fusion operator that outputs the tensor B from the fusion subgraph obtained by one operator fusion. Next, in (B) of Figure 14, operators other than the recalculation node set Gather can be deleted, that is, ScatterAdd can be deleted, and edges that are not related to the recalculation node set Gather can be deleted, that is, operator Scatter and operator Gather can be deleted The edge between them and the edge corresponding to the output tensor of the operator Scatter, so as to obtain the recomputation graph shown in (C) of Figure 14.

E) Generate a new calculation graph

After the recalculation graph is obtained, according to the relationship between the nodes in each recalculation graph and the nodes in the original computation subgraph, each recalculation graph is combined with the original computation graph to generate a new computation graph. Among them, there is a data dependency between the node that inputs the first tensor in the recalculation graph and the node that outputs the first tensor in the original computation subgraph, and the node that outputs the second tensor in the recalculation graph is the same as the node that outputs the second tensor in the original computation subgraph. Tensor nodes have data dependencies. It is understandable that, for the original computation subgraph, if operator fusion has been performed on the original computation graph during the execution of the recomputation step, the original computation subgraph is the subgraph obtained after operator fusion; If operator fusion is not performed on the original computation graph during the recalculation step, the original computation subgraph is the original computation graph.

Exemplarily, the directed edge between the node inputting the first target tensor in each recomputation graph and the node outputting the first target tensor in the original calculation graph can be respectively constructed, and each recomputation graph can be constructed separately The directed edge between the node outputting the second target tensor in the original calculation graph and the node inputting the second target tensor in the original calculation graph to obtain a new calculation graph.

For example, as shown in Figure 15, (A) in Figure 15 is the initial calculation graph (that is, the original calculation graph). A fusion subgraph shown. After recalculating the primary fusion subgraph shown in (B) of FIG. 15 , a recalculation graph as shown in (C) of FIG. 15 can be obtained. Since the input of the recomputation graph shown in (C) in Figure 15 is tensor A, the output is tensor B, and the operator outputting tensor A in Figure 15 (B) is operator Slice, and the input tensor B The operator of is the operator FuseOp2, therefore, the operator used to input the tensor A in the recomputation graph in (C) of Figure 15 can be connected with the operator Slice in (B) of Figure 15, that is, to construct two The directed edge between them, and the operator used to output the tensor B in the recomputation graph in (C) of Figure 15 is connected to the operator FuseOp2 in (B) of Figure 15, that is, to construct the The directed edge between. Therefore, after combining the recomputation graph shown in (C) of FIG. 15 with the primary fusion subgraph shown in (B) of FIG. 15 , a new computation graph shown in (D) of FIG. 15 can be obtained.

(3) Secondary operator fusion

Operator fusion (ie, secondary fusion) is performed on the obtained new calculation graph to obtain a new fusion subgraph, thereby obtaining an optimized calculation graph. Exemplarily, based on preset operator fusion rules, the computation graph can be divided into A plurality of calculation subgraphs, wherein each calculation subgraph may correspond to an operator obtained after fusion (hereinafter referred to as "fusion operator"). Thus, an optimized calculation graph is obtained, that is, the secondary fusion subgraph, which is the final optimized calculation graph. After that, the calculation graph obtained by the final optimization can be executed. In an example, for the calculation graph obtained by executing the final optimization, the calculation graph can be compiled first, and then the compiled calculation graph can be executed, or the calculation graph can be directly executed; either way, it is essentially The calculation graph obtained by the final optimization performed is only different in representation.

For example, as shown in FIG. 16 , (A) in FIG. 16 shows a new calculation graph obtained after recalculation. If the preset operator fusion rule is: Gather and Mul fusion, then based on the operator fusion rule, the fusion subgraph shown in (B) in Figure 16 can be obtained after performing operator fusion on (A) in Figure 16 .

In some embodiments, the nodes described in the embodiments of the present application may be, but not limited to, operators, and the operators described in the embodiments of the present application may also be, but not limited to, nodes, which may be determined according to actual conditions.

Therefore, the fusion subgraph is obtained by first performing operator fusion on the calculation graph, and then the recalculation graph that needs to be recalculated is determined from the fusion subgraph, and the determined recalculation graph is combined with the fusion subgraph to obtain Generate a new calculation graph, and finally perform operator fusion on the new calculation graph to obtain the required calculation graph, which achieves a significant reduction in memory usage without introducing large recalculation overhead, and solves the problem of having one or more The problem that networks with very large tensors cannot perform.

For ease of understanding, the following example illustrates the process of reducing memory usage through the above solution.

Exemplarily, as shown in Figure 17, in Figure 17, M and L respectively represent the memory size occupied by the tensor output by the operator, M represents a medium size, such as 56.8M, and L represents a super large size, such as 27G. The calculation graph shown in (A) of FIG. 17 is a calculation graph without any processing, that is, an initial calculation graph. In (A) of Figure 17, the operator Slice and the operator ScatterAdd reuse a memory, and the operator Gather and the operator Div reuse a memory, so the memory size required for the calculation graph shown in Figure 17 (A) It is: M1+M2+L1+L2.

The calculation graph shown in (B) of FIG. 17 is a calculation graph obtained by performing operator fusion on the calculation graph shown in FIG. 17 (A) based on a preset operator fusion rule, that is, a primary fusion subgraph. In (B) of Figure 17, the operator Slice and the operator MatMul reuse one memory, so the memory size required for the calculation graph shown in (B) of Figure 17 is: M1+M2+L1+L2.

The calculation graph shown in (C) of FIG. 17 is a calculation graph obtained after recalculating the calculation graph shown in FIG. 17 (B), that is, a new calculation graph. In (C) of Figure 17, memory cannot be reused between operators, so the memory size required for the calculation graph shown in (C) of Figure 17 is: M1+M2+M3+L1+L2.

The calculation graph shown in (D) of Figure 17 is a calculation graph obtained by performing secondary operator fusion on the calculation graph shown in Figure 17 (C) based on the preset operator fusion rules, that is, the secondary fusion subgraph , the computation graph is the desired computation graph. In (D) of Figure 17, memory cannot be reused between operators, so the memory size required for the calculation graph shown in (D) of Figure 17 is: M1+M2+M3+L1.

Assuming M1=M2=M3, L1=L2, the memory size occupied by the calculation graph shown in Figure 17 (A) is 2M+2L, and the memory size occupied by the calculation graph shown in Figure 17 (B) is 2M+ 2L, the memory size occupied by the calculation graph shown in (C) of Figure 17 is 3M+2L, and the memory size occupied by the calculation graph shown in (D) of Figure 17 is 3M+L. Since M represents medium size and L represents super large, the calculation graph shown in (D) of Figure 17 occupies less memory than the calculation graph of (A) in Figure 17 (L-M). In addition, it can also be seen from the comparison of (A) and (B) in Figure 17 that after an operator fusion, the memory size occupied by the calculation graph does not change.

Furthermore, during the recalculation process, the recalculation point can be determined based on memory size, data mutation, fusion rules, etc., which can not only effectively reduce memory usage and improve the efficiency of identifying recalculation points, but also reduce memory usage. Tensors are converted to internal storage (such as register space),`

Next, based on the calculation graph optimization scheme described above, a calculation graph optimization method provided by the embodiment of this application is introduced. It can be understood that this method is another expression of the calculation graph optimization scheme described above, and the two are combined. This method is proposed based on the calculation graph optimization scheme described above, part or all of the content of this method can be referred to but not limited to the description of the calculation graph optimization scheme above.

Please refer to FIG. 18 . FIG. 18 is a schematic flowchart of a computation graph optimization method provided by an embodiment of the present application. It can be understood that the method can be executed by any device, device, platform, or device cluster that has computing and processing capabilities. As shown in Figure 18, the calculation graph optimization method may include the following steps:

S1801. Obtain the second computation graph from the first computation graph based on the parameters and the data dependencies between the nodes in the first computation graph. The parameters include operator fusion rules and the tensor output of a single node in the first computation graph. One or more of the memory threshold of the memory, the peak memory threshold corresponding to the first computation graph, and the data mutation threshold corresponding to the nodes in the first computation graph. The threshold of the memory occupied by the tensor to be used, and the data mutation threshold is the threshold of at least one node in the first calculation graph that generates a data mutation.

Specifically, when optimizing the computation graph, the second computation graph can be obtained from the first computation graph based on parameters and data dependencies between nodes in the first computation graph. Exemplarily, the parameters include operator fusion rules, memory thresholds occupied by tensors output by a single node in the first computation graph, peak memory thresholds corresponding to the first computation graph, and data mutation thresholds corresponding to nodes in the first computation graph one or more of the . Wherein, the peak memory threshold is the threshold of memory occupied by all the tensors to be used at the time of execution of a node during the execution of the computation graph, and the data mutation threshold is the threshold of at least one node in the first computation graph that generates a data mutation. For example, when the computing framework corresponding to the computing graph is the MindSporeAI open source computing framework, the user can set the memory threshold, peak memory threshold, and data mutation threshold through the Context module configured in MindSpore. Exemplarily, the memory threshold may also be called a memory threshold, the peak memory threshold may also be called a peak memory threshold, and the data mutation threshold may also be called a data threshold. In some embodiments, a single node can be understood as a node, and can also be understood as a node obtained by merging multiple nodes.

As a possible implementation, when obtaining the second computation graph, based on the parameters and the data dependencies between the nodes in the first computation graph, N first A node set, N is a positive integer greater than or equal to 1, the first node set includes at least one node, and the nodes contained in a first node set are all directly related to a tensor output by a producer in the first calculation graph are related or indirectly related, and are directly related or indirectly related to at least one tensor input by a producer in the first computation graph. Then, recomputation is performed on each first node set to obtain N recalculation subgraphs, wherein the N recalculation subgraphs constitute the second computation graph. In this way, the second calculation graph can be obtained. Exemplarily, the second computation graph may include N recomputation subgraphs. Exemplarily, the first node set can be understood as the required recomputation node set described above, and the recalculation subgraph can be understood as the recomputation graph described above.

In an example, performing recalculation on each first node set to obtain N recalculation subgraphs may be implemented in the following manner. Specifically, for any node set in the N first node sets, the producer subgraph (also called a producer) corresponding to the any node set may be copied first. Then, delete the nodes in the producer subgraph except for the nodes contained in any node set, and delete the edges that are not related to any node set in the producer subgraph, so that the corresponding weight of any node set can be obtained Compute subgraphs. By traversing each node set in the N first node sets, N recomputation subgraphs can be obtained.

Wherein, for obtaining the first set of nodes, the first list may be obtained based on the data dependencies between the nodes in the first computation graph, and the first list includes the correspondence between producers and consumers in the first computation graph. Exemplarily, the first list can be the producer-consumer list described above, which can be shown in Table 1 above, and this step can be understood as constructing the producer-consumer list in the recalculation step described above. Steps for consumer list.

Then, according to the tensor output by each producer in the first list, a set of candidate nodes is obtained, and the set of candidate nodes includes at least the first set of nodes, wherein, for any producer in the first list, any The set of nodes in a producer that are directly and indirectly related to tensors output by any producer is regarded as a node set. Wherein, this step can be understood as a step of determining a recalculation candidate set in the recalculation step described above. Wherein, the candidate node set can be understood as the recalculation candidate set described above.

Finally, based on the parameters, the first node set is selected from the candidate node sets. Among them, this step can be understood as one or more of the above-described recalculation steps based on memory size screening, data mutation-based screening, and operator fusion rule-based screening to determine recalculation from the recalculation candidate set Diagram of steps.

In one example, the parameter is a memory threshold; the memory occupied by the output tensor corresponding to the first set of nodes is higher than the memory threshold. In this way, the node set corresponding to the output tensor that occupies too much memory can be automatically used as the required node set, thereby solving the problem of memory occupied by a single operator and/or tensor.

In one example, the parameter is the peak memory threshold; there is an indirect path between the producer and the consumer corresponding to the first node set, and the peak memory of each node between the producer and the consumer corresponding to the first node set when executing Above the peak memory threshold. In this way, the node set corresponding to the high peak memory usage can be automatically used as the required node set, thereby solving the problem of memory usage of a single operator and/or tensor.

In one example, the parameter is a data mutation threshold; the deviation value between the memory occupied by the output tensor corresponding to the first node set and the memory occupied by at least one input tensor corresponding to the first node set is higher than the data mutation threshold . In this way, the node set with data mutation can be automatically used as the required node set, thereby eliminating the problem of excessive memory usage caused by data mutation.

In one example, the parameter is an operator fusion rule; the first node set and the consumers corresponding to the first node set can be fused; or, the first node set and the consumers corresponding to the first node set cannot be fused, and the first node set There is an indirect path between the producer and consumer corresponding to the set. In this way, the required node set can be filtered out through operator fusion rules.

In one example, the parameters are memory threshold and peak memory threshold; there is no indirect path between producers and consumers corresponding to the first set of nodes, and the memory occupied by the output tensor corresponding to the first set of nodes is higher than the memory threshold and/or, there is an indirect path between the producer and the consumer corresponding to the first node set, and the peak memory of each node between the producer and the consumer corresponding to the first node set is higher than the peak memory threshold during execution. In this way, the required node set can be filtered out by combining the memory threshold and the peak memory threshold, improving the efficiency and accuracy of screening.

In one example, the parameters are a memory threshold and a data mutation threshold; the memory occupied by the output tensor corresponding to the first node set is higher than the memory threshold, and/or, the memory occupied by the output tensor corresponding to the first node set is different from A deviation value between memory occupied by at least one input tensor corresponding to the first node set is higher than a data mutation threshold. In this way, the required node set can be screened out by combining the memory threshold and the data mutation threshold, improving the efficiency and accuracy of screening.

In one example, the parameters are a memory threshold and an operator fusion rule; the first node set meets one or more of the following conditions: the memory occupied by the output tensor corresponding to the first node set is higher than the memory threshold, or, The first node set and the consumers corresponding to the first node set can be fused, or the first node set and the consumers corresponding to the first node set cannot be fused, and there is an indirect relationship between the producer and the consumer corresponding to the first node set path. In this way, the required node set can be screened out through the combination of memory threshold and operator fusion rules, improving the efficiency and accuracy of screening.

In one example, the parameters are the peak memory threshold and the data mutation threshold; there is an indirect path between the producer and the consumer corresponding to the first node set, and each node between the producer and the consumer corresponding to the first node set executes and/or, the deviation value between the memory occupied by the output tensor corresponding to the first set of nodes and the memory occupied by at least one input tensor corresponding to the first set of nodes is higher than Data mutation threshold. In this way, the required node set can be screened out by combining the peak memory threshold and the data mutation threshold, improving the efficiency and accuracy of screening.

In one example, the parameters are the memory threshold and operator fusion rules; the first node set meets one or more of the following conditions: there is an indirect path between the producer and the consumer corresponding to the first node set, and the first The peak memory of each node between the producer and the consumer corresponding to the node set is higher than the peak memory threshold during execution, or the first node set and the consumer corresponding to the first node set can be merged, or the first node set and The consumers corresponding to the first node set cannot be merged, and there is an indirect path between the producer and the consumer corresponding to the first node set. In this way, the required node set can be screened out by combining the peak memory threshold and operator fusion rules, improving the efficiency and accuracy of screening.

In one example, the parameters are data mutation threshold and operator fusion rule; the first node set meets one or more of the following conditions: the memory occupied by the output tensor corresponding to the first node set corresponds to the first node set The deviation value between the memory occupied by at least one input tensor of is higher than the data mutation threshold, or, the first node set and the consumers corresponding to the first node set can be fused, or, the first node set and the first node set The corresponding consumers cannot be fused, and there is an indirect path between the producers and consumers corresponding to the first set of nodes. In this way, the required node set can be screened out by combining the data mutation threshold and operator fusion rules, improving the efficiency and accuracy of screening.

In one example, the parameters are a memory threshold, a peak memory threshold, and a data mutation threshold; the first set of nodes meets one or more of the following conditions: there is no indirect path between producers and consumers corresponding to the first set of nodes , and the memory occupied by the output tensor corresponding to the first node set is higher than the memory threshold, or there is an indirect path between the producer and the consumer corresponding to the first node set, and the producer and consumer corresponding to the first node set The peak memory of each node between them is higher than the peak memory threshold, or the memory occupied by the output tensor corresponding to the first node set is between the memory occupied by at least one input tensor corresponding to the first node set The deviation value of is higher than the data mutation threshold. In this way, the required node set can be screened out by combining the memory threshold, peak memory threshold, and data mutation threshold, improving the efficiency and accuracy of screening.

In one example, the parameters are memory threshold, peak memory threshold, and operator fusion rules; the first set of nodes meets one or more of the following conditions: there is no indirect relationship between producers and consumers corresponding to the first set of nodes path, and the memory occupied by the output tensor corresponding to the first node set is higher than the memory threshold, or, there is an indirect path between the producer and the consumer corresponding to the first node set, and the producer and the consumer corresponding to the first node set correspond to The peak memory of each node between consumers is higher than the peak memory threshold, or the first node set and the consumers corresponding to the first node set can be merged, or the consumption of the first node set and the first node set cannot be merged, and there is an indirect path between the producers and consumers corresponding to the first set of nodes. In this way, the required node set can be screened out by combining the memory threshold, peak memory threshold, and operator fusion rules, improving the efficiency and accuracy of screening.

In one example, the parameters are memory threshold, data mutation threshold and operator fusion rule; the first node set meets one or more of the following conditions: the memory occupied by the output tensor corresponding to the first node set is higher than the memory Threshold, or, the deviation value between the memory occupied by the output tensor corresponding to the first node set and the memory occupied by at least one input tensor corresponding to the first node set is higher than the data mutation threshold, or, the first node set Consumers corresponding to the first node set can be fused, or the first node set and consumers corresponding to the first node set cannot be fused, and there is an indirect path between producers and consumers corresponding to the first node set. In this way, the required node set can be screened out by combining the memory threshold, data mutation threshold, and operator fusion rules, improving the efficiency and accuracy of screening.

In one example, the parameters are peak memory threshold, data mutation threshold, and operator fusion rules; the first node set meets one or more of the following conditions: there is an indirect path, and the peak memory of each node between the producer and consumer corresponding to the first node set is higher than the peak memory threshold during execution, or the memory occupied by the output tensor corresponding to the first node set is the same as the first node set The deviation value between the memory occupied by the corresponding at least one input tensor is higher than the data mutation threshold, or the first node set and the consumers corresponding to the first node set can be fused, or the first node set and the first node set The consumers corresponding to the set cannot be merged, and there is an indirect path between the producer and the consumer corresponding to the first set of nodes. In this way, the required node set can be screened out by combining the peak memory threshold, data mutation threshold, and operator fusion rules, improving the efficiency and accuracy of screening.

In one example, the parameters are memory threshold, peak memory threshold, data mutation threshold and operator fusion rule; the first node set meets one or more of the following conditions: the relationship between the producer and the consumer corresponding to the first node set There is no indirect path between, and the memory occupied by the output tensor corresponding to the first node set is higher than the memory threshold, or, there is an indirect path between the producer and the consumer corresponding to the first node set, and the first node set corresponds to The peak memory of each node between the producer and consumer is higher than the peak memory threshold, or the memory occupied by the output tensor corresponding to the first node set is equal to the memory occupied by at least one input tensor corresponding to the first node set The deviation value between the occupied memory is higher than the data mutation threshold, or the first node set and the consumers corresponding to the first node set can be fused, or the first node set and the consumers corresponding to the first node set cannot be fused, And there is an indirect path between the producer and the consumer corresponding to the first node set. In this way, the required node set can be screened out through a combination of memory threshold, peak memory threshold, data mutation threshold, and operator fusion rules, improving the efficiency and accuracy of screening.

In some embodiments, before S1801, operators in the first computation graph may be fused first. This improves the performance of operators in the calculation graph; in addition, through the first operator fusion, the number of operators can also be reduced, which can reduce the overhead of analyzing operators in the subsequent recalculation process; in addition, it can also expand In the subsequent recalculation process, the operator is analyzed to improve the effect of recalculation. Exemplarily, but not limited to, operators in the first computation graph may be fused based on a preset operator merging rule.

After obtaining the second calculation graph, S1802 can be executed.

S1802. Merge the second computation graph with the first computation graph to obtain a third computation graph, wherein, in the third computation graph, there is a gap between the first node that outputs the first tensor and the second node that inputs the first tensor There is a first directed edge, there is a second directed edge between the third node outputting the second tensor and the fourth node inputting the second tensor, the first directed edge is directed from the first node to the second node, and the second The directed edge has the third node pointing to the fourth node, the first node and the fourth node correspond to the nodes in the first computation graph, and the second node and the third node correspond to the nodes in the second computation graph.

Specifically, after obtaining the second computation graph, the second computation graph can be merged with the first computation graph, thus obtaining the third computation graph. Among them, in the third calculation graph, there is a first directed edge between the first node outputting the first tensor and the second node inputting the first tensor, and the third node outputting the second tensor is connected to the second node inputting the second tensor There is a second directed edge between the fourth nodes of the quantity, the first directed edge is from the first node to the second node, the second directed edge has the third node to the fourth node, the first node and the fourth node correspond to the A node in a computation graph, and the second node and the third node correspond to nodes in a second computation graph. Wherein, the process of obtaining the third calculation graph can be understood as the step of generating a new calculation graph in the recalculation step described above.

In one example, when the second computation graph is composed of N recalculation subgraphs, when merging the second computation graph with the first computation graph, each recalculation subgraph in the N recalculation subgraphs can be constructed separately The directed edge between the node inputting the first target tensor in the graph and the node outputting the first target tensor in the first computation graph, and constructing each recalculation subgraph in the N recomputation subgraphs respectively A directed edge between a node that outputs the second target tensor and a node that inputs the second target tensor in the first computation graph results in a third computation graph.

S1803. Fuse operators in the third computation graph to obtain a fourth computation graph.

Specifically, operators in the third computation graph may be fused based on but not limited to a preset operator fusion rule, so as to obtain a fourth computation graph. Exemplarily, the fourth computation graph may also be called an optimized computation graph.

S1804. Execute the fourth calculation graph.

Specifically, after obtaining the fourth calculation graph, the calculation graph can be executed.

In one example, for the execution of the fourth calculation graph, the calculation graph can be compiled first, and then the compiled calculation graph can be executed, or the calculation graph can be directly executed; either way, it is essentially executed The fourth calculation graph is only in a different form.

In this way, the recalculation graph is obtained by recomputing the desired computation graph first, and the obtained recalculation graph is merged with the original computation graph into a new computation graph, and then the new computation graph is fused with operators to obtain the optimized Computation graph; after that, the optimized computation graph can be executed. Therefore, the calculation graph is optimized by combining recalculation and operator fusion, which achieves a significant reduction in memory usage without introducing large recalculation overhead, and solves the problem of networks with one or more large tensors. Unable to implement the problem.

Based on the methods described in the foregoing embodiments, embodiments of the present application further provide a chip. Please refer to FIG. 19 . FIG. 19 is a schematic structural diagram of a chip provided by an embodiment of the present application. As shown in FIG. 19 , a chip 1900 includes one or more processors 1901 and an interface circuit 1902 . Optionally, the chip 1900 may also include a bus 1903 . in:

The processor 1901 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method may be completed by an integrated logic circuit of hardware in the processor 1901 or instructions in the form of software. The above-mentioned processor 1901 may be a general-purpose processor, a digital communicator (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components . Various methods and steps disclosed in the embodiments of the present application may be implemented or executed. A general-purpose processor may be a microprocessor, or the processor may be any conventional processor, or the like.

The interface circuit 1902 can be used for sending or receiving data, instructions or information. The processor 1901 can use the data, instructions or other information received by the interface circuit 1902 to process, and can send the processing completion information through the interface circuit 1902 .

Optionally, the chip further includes a memory, which may include a read-only memory and a random access memory, and provides operation instructions and data to the processor. A portion of the memory may also include non-volatile random access memory (NVRAM).

Optionally, the memory stores executable software modules or data structures, and the processor can execute corresponding operations by calling operation instructions stored in the memory (the operation instructions can be stored in the operating system).

Optionally, the interface circuit 1902 may be used to output the execution result of the processor 1901 .

It should be noted that the corresponding functions of the processor 1901 and the interface circuit 1902 can be realized by hardware design, software design, or a combination of software and hardware, which is not limited here.

It should be understood that each step in the foregoing method embodiments may be implemented by logic circuits in the form of hardware or instructions in the form of software in the processor.

It can be understood that the processor in the embodiments of the present application may be a central processing unit (central processing unit, CPU), and may also be other general processors, digital signal processors (digital signal processor, DSP), application specific integrated circuits (application specific integrated circuit, ASIC), field programmable gate array (field programmable gate array, FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. A general-purpose processor can be a microprocessor, or any conventional processor.

The method steps in the embodiments of the present application may be implemented by means of hardware, or may be implemented by means of a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in random access memory (random access memory, RAM), flash memory, read-only memory (read-only memory, ROM), programmable read-only memory (programmable rom) , PROM), erasable programmable read-only memory (erasable PROM, EPROM), electrically erasable programmable read-only memory (electrically EPROM, EEPROM), register, hard disk, mobile hard disk, CD-ROM or known in the art any other form of storage medium. An exemplary storage medium is coupled to the processor such the processor can read information from, and write information to, the storage medium. Of course, the storage medium may also be a component of the processor. The processor and storage medium can be located in the ASIC.

In the above embodiments, all or part of them may be implemented by software, hardware, firmware or any combination thereof. When implemented using software, it may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on the computer, the processes or functions according to the embodiments of the present application will be generated in whole or in part. The computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in or transmitted via a computer-readable storage medium. The computer instructions may be transmitted from one website site, computer, server, or data center to another website site by wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) , computer, server or data center for transmission. The computer-readable storage medium may be any available medium that can be accessed by a computer, or a data storage device such as a server or a data center integrated with one or more available media. The available medium may be a magnetic medium (such as a floppy disk, a hard disk, or a magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (solid state disk, SSD)), etc.

It can be understood that the various numbers involved in the embodiments of the present application are only for convenience of description, and are not used to limit the scope of the embodiments of the present application.

Claims (26)

A computational graph optimization method, characterized in that the method comprises: Based on the parameters and the data dependencies between the nodes in the first calculation graph, the second calculation graph is obtained from the first calculation graph, the parameters include operator fusion rules, and the output of a single node in the first calculation graph One or more of the memory threshold of the memory occupied by the tensor, the peak memory threshold corresponding to the first computation graph, and the data mutation threshold corresponding to the node in the first computation graph, the peak memory threshold being the computation graph The threshold value of the memory occupied by all tensors to be used at the moment of execution of a node during the execution process, the data mutation threshold is the threshold value of data mutation generated by at least one node in the first calculation graph; Merging the second computation graph with the first computation graph to obtain a third computation graph, wherein, in the third computation graph, the first node that outputs the first tensor is the same as the first node that inputs the first tensor There is a first directed edge between the second node of the quantity, and there is a second directed edge between the third node outputting the second tensor and the fourth node inputting the second tensor, and the first directed edge consists of The first node points to the second node, the second directed edge has the third node pointing to the fourth node, and the first node and the fourth node correspond to the first calculation graph The node in, the second node and the third node correspond to the nodes in the second calculation graph; Fusing operators in the third computation graph to obtain a fourth computation graph; Execute the fourth computation graph. The method according to claim 1, wherein the second calculation graph is obtained from the first calculation graph based on parameters and data dependencies between nodes in the first computation graph, specifically comprising: Based on the parameters and the data dependencies between the nodes in the first calculation graph, N first node sets are obtained from the producers included in the first calculation graph, where N is a positive integer greater than or equal to 1 , the first set of nodes includes at least one node, wherein each node included in the first set of nodes is directly or indirectly related to a tensor output by a producer in the first computation graph, and are all directly or indirectly related to at least one tensor input by a producer in the first computation graph; Recomputation is performed on each of the first node sets to obtain N recalculation subgraphs, and the N recomputation subgraphs constitute the second computation graph. The method according to claim 2, wherein the parameter is the memory threshold; The memory occupied by the output tensor corresponding to the first node set is higher than the memory threshold. The method according to claim 2, wherein the parameter is the peak memory threshold; There is an indirect path between the producer and the consumer corresponding to the first node set, and the peak memory of each node between the producer and the consumer corresponding to the first node set is higher than the peak memory threshold during execution . The method according to claim 2, wherein the parameter is the data mutation threshold; A deviation value between the memory occupied by the output tensor corresponding to the first node set and the memory occupied by at least one input tensor corresponding to the first node set is higher than the data mutation threshold. The method according to claim 2, wherein the parameter is the operator fusion rule; The first node set and the consumers corresponding to the first node set can be merged; Alternatively, the first set of nodes and consumers corresponding to the first set of nodes cannot be merged, and an indirect path exists between producers and consumers corresponding to the first set of nodes. The method according to claim 2, wherein said parameter is said memory threshold and said peak memory threshold; There is no indirect path between the producer and the consumer corresponding to the first node set, and the memory occupied by the output tensor corresponding to the first node set is higher than the memory threshold; And/or, there is an indirect path between the producer and the consumer corresponding to the first node set, and the peak memory of each node between the producer and the consumer corresponding to the first node set is higher than the specified The peak memory threshold described above. The method according to claim 2, wherein the parameters are the memory threshold and the data mutation threshold; The memory occupied by the output tensor corresponding to the first node set is higher than the memory threshold, and/or, the memory occupied by the output tensor corresponding to the first node set is equal to the memory corresponding to the first node set A deviation value between memory occupied by at least one input tensor is higher than the data mutation threshold. The method according to claim 2, wherein the parameters are the memory threshold and the operator fusion rule; The first set of nodes meets one or more of the following conditions: The memory occupied by the output tensor corresponding to the first node set is higher than the memory threshold, or, the first node set and the consumers corresponding to the first node set can be fused, or, the first node set The node set and the consumer corresponding to the first node set cannot be merged, and an indirect path exists between the producer and the consumer corresponding to the first node set. The method according to claim 2, wherein the parameters are the peak memory threshold and the data mutation threshold; There is an indirect path between the producer and the consumer corresponding to the first node set, and the peak memory of each node between the producer and the consumer corresponding to the first node set is higher than the peak memory threshold during execution ; And/or, the deviation value between the memory occupied by the output tensor corresponding to the first node set and the memory occupied by at least one input tensor corresponding to the first node set is higher than the data mutation threshold. The method according to claim 2, wherein the parameters are the memory threshold and the operator fusion rule; The first set of nodes meets one or more of the following conditions: There is an indirect path between the producer and the consumer corresponding to the first node set, and the peak memory of each node between the producer and the consumer corresponding to the first node set is higher than the peak memory threshold during execution , or, the first node set and the consumers corresponding to the first node set can be fused, or, the first node set and the consumers corresponding to the first node set cannot be fused, and the first There are indirect paths between producers and consumers corresponding to a set of nodes. The method according to claim 2, wherein the parameters are the data mutation threshold and the operator fusion rule; The first set of nodes meets one or more of the following conditions: The deviation value between the memory occupied by the output tensor corresponding to the first node set and the memory occupied by at least one input tensor corresponding to the first node set is higher than the data mutation threshold, or, the The first node set and the consumers corresponding to the first node set can be fused, or the first node set and the consumers corresponding to the first node set cannot be fused, and the production corresponding to the first node set There is an indirect path between buyers and consumers. The method according to claim 2, wherein the parameters are the memory threshold, the peak memory threshold and the data mutation threshold; The first set of nodes meets one or more of the following conditions: There is no indirect path between the producer and the consumer corresponding to the first node set, and the memory occupied by the output tensor corresponding to the first node set is higher than the memory threshold, or the first node There is an indirect path between the producer and the consumer corresponding to the set, and the peak memory of each node between the producer and the consumer corresponding to the first node set is higher than the peak memory threshold during execution, or, the A deviation value between the memory occupied by the output tensor corresponding to the first node set and the memory occupied by at least one input tensor corresponding to the first node set is higher than the data mutation threshold. The method according to claim 2, wherein the parameters are the memory threshold, the peak memory threshold and the operator fusion rule; The first set of nodes meets one or more of the following conditions: There is no indirect path between the producer and the consumer corresponding to the first node set, and the memory occupied by the output tensor corresponding to the first node set is higher than the memory threshold, or the first node There is an indirect path between the producer and the consumer corresponding to the set, and the peak memory of each node between the producer and the consumer corresponding to the first node set is higher than the peak memory threshold during execution, or, the The first node set and the consumers corresponding to the first node set can be fused, or the first node set and the consumers corresponding to the first node set cannot be fused, and the production corresponding to the first node set There is an indirect path between buyers and consumers. The method according to claim 2, wherein the parameters are the memory threshold, the data mutation threshold and the operator fusion rule; The first set of nodes meets one or more of the following conditions: The memory occupied by the output tensor corresponding to the first node set is higher than the memory threshold, or the memory occupied by the output tensor corresponding to the first node set is at least one of the memory corresponding to the first node set The deviation value between the memory occupied by the input tensor is higher than the data mutation threshold, or, the first node set and the consumers corresponding to the first node set can be fused, or, the first node set Consumers corresponding to the first node set cannot be merged, and an indirect path exists between producers and consumers corresponding to the first node set. The method according to claim 2, wherein the parameters are the peak memory threshold, the data mutation threshold and the operator fusion rule; The first set of nodes meets one or more of the following conditions: There is an indirect path between the producer and the consumer corresponding to the first node set, and the peak memory of each node between the producer and the consumer corresponding to the first node set is higher than the peak memory threshold during execution , or, the deviation value between the memory occupied by the output tensor corresponding to the first node set and the memory occupied by at least one input tensor corresponding to the first node set is higher than the data mutation threshold, or , the first node set and the consumers corresponding to the first node set can be fused, or the first node set and the consumers corresponding to the first node set cannot be fused, and the first node set There are indirect paths between corresponding producers and consumers. The method according to claim 2, wherein the parameters are the memory threshold, the peak memory threshold, the data mutation threshold and the operator fusion rule; The first set of nodes meets one or more of the following conditions: There is no indirect path between the producer and the consumer corresponding to the first node set, and the memory occupied by the output tensor corresponding to the first node set is higher than the memory threshold, or the first node There is an indirect path between the producer and the consumer corresponding to the set, and the peak memory of each node between the producer and the consumer corresponding to the first node set is higher than the peak memory threshold during execution, or, the The deviation value between the memory occupied by the output tensor corresponding to the first node set and the memory occupied by at least one input tensor corresponding to the first node set is higher than the data mutation threshold, or, the first The node set and the consumers corresponding to the first node set can be fused, or the first node set and the consumers corresponding to the first node set cannot be fused, and the producers corresponding to the first node set and There are indirect paths between consumers. The method according to any one of claims 2-17, characterized in that, based on the parameters and the data dependencies between nodes in the first computation graph, from the production data included in the first computation graph At least one first node set is obtained from those, specifically including: Obtaining a first list based on data dependencies between nodes in the first computation graph, the first list including correspondences between producers and consumers in the first computation graph; According to the tensor output by each producer in the first list, a candidate node set is obtained, and the candidate node set includes at least the first node set, wherein, for any of the first list A producer, taking the set of nodes directly and indirectly related to the tensor output by any producer in any producer as a set of nodes; Based on the parameters, the first set of nodes is selected from the set of candidate nodes. The method according to any one of claims 2-18, wherein the recalculation is performed on each of the first node sets to obtain N recalculation subgraphs, specifically comprising: For any node set in the N first node sets, copy the producer subgraph corresponding to the any node set; Deleting nodes in the producer subgraph other than the nodes included in any node set, and deleting edges not related to any node set in the producer subgraph, so as to obtain the any The recomputed subgraph corresponding to the set of nodes. The method according to any one of claims 2-19, wherein the merging the second calculation graph with the first calculation graph to obtain a third calculation graph specifically includes: Respectively construct the directed relationship between the node inputting the first target tensor and the node outputting the first target tensor in the first calculation graph of each recomputing subgraph in the N recomputing subgraphs An edge, and respectively constructing the connection between the node outputting the second target tensor in each of the N recomputing subgraphs and the node inputting the second target tensor in the first calculation graph between directed edges to obtain the third computation graph. The method according to any one of claims 1-20, characterized in that, based on the parameters and the data dependencies between the nodes in the first calculation graph, the first calculation graph that needs to be recalculated is obtained from the first calculation graph Before calculating the graph, the method also includes: The operators in the first computation graph are fused. A computational graph optimization device, characterized in that it comprises: at least one memory for storing programs; At least one processor is configured to execute the program stored in the memory, and when the program stored in the memory is executed, the processor is configured to execute the method according to any one of claims 1-21. A device, characterized in that it comprises: at least one memory for storing programs; At least one processor is configured to execute the program stored in the memory, and when the program stored in the memory is executed, the processor is configured to execute the method according to any one of claims 1-21. A computer-readable storage medium, the computer-readable storage medium stores a computer program, and when the computer program is run on an electronic device, the electronic device executes the method according to any one of claims 1-21 . A computer program product, characterized in that, when the computer program product is run on an electronic device, the electronic device is made to execute the method according to any one of claims 1-21. A chip, characterized in that it includes at least one processor and an interface; said interface for providing program instructions or data to said at least one processor; The at least one processor is configured to execute the program line instructions to implement the method as claimed in any one of claims 1-21.

Images