WO2025055495A1 - Memory management method and apparatus for neural network model, device, medium and product - Google Patents

Abstract

The present application relates to the technical field of memory management, and discloses a memory management method and apparatus for a neural network model, a device, a medium and a product. The method comprises: acquiring a computational graph corresponding to a neural network model; on the basis of the computational graph, determining memory sizes to be allocated to network layer operators; and acquiring, from a free memory block list, allocation memory blocks matched with the memory sizes, and allocating the allocation memory blocks to the network layer operators, wherein the free memory block list is used for storing free memory blocks which have been allocated but occupation of which has been released, and the allocation memory blocks are memory blocks allocated to the network layer operators for storing data. By means of free memory blocks in a free memory block list, allocation memory blocks matched with memory sizes are allocated to network layer operators, reducing the memory allocated to a neural network model during operation, improving the utilization rate of memory.

Description

Memory management method, device, equipment, medium and product for neural network model

This application claims priority to Chinese patent application filed on September 11, 2023, with application number 202311165933.1 and invention name “Memory management method, device, equipment, medium and product for neural network model”, all contents of which are incorporated by reference into this application.

Technical Field

The embodiments of the present application relate to the field of memory management technology, and in particular to a memory management method, device, equipment, medium and product for a neural network model.

Background Art

With the development of artificial intelligence (AI) technology, neural network models are being used more and more frequently. In order to achieve better algorithm accuracy, neural network models are becoming more and more complex, and hardware capabilities limit the development of neural networks in a deeper direction.

In the related art, the memory required to be occupied by each network layer in the neural network model is obtained, and then the memory is allocated to the entire neural network model according to the running order of the entire neural network model. For example, during the operation of the neural network model, it is necessary to occupy a 100M memory block, a 10M memory block, and a 50M memory block in sequence. The storage cycles of the 10M memory block and the 50M memory block are intersected, that is, the occupied time of the 10M memory block and the 50M memory block has an intersection. When the neural network model applies for a 100M memory block, a 100M memory block can be allocated to the neural network, and then when the neural network model applies for a 10M memory block, it can be determined whether the 10M memory block application can reuse the above-mentioned allocated 100M memory block. If it can, no new memory block is allocated for the applied 10M memory block, and the 10M memory block application reuses the above-mentioned 100M memory block. Similarly, when the neural network model applies for a 50M memory block, it also determines whether the 50M memory block application can reuse the above-mentioned allocated 100M memory block. If it can be reused, the 50M memory block is allocated to reuse the above-mentioned allocated 100M memory block. Otherwise, a new 50M memory block is allocated for the 50M memory block application.

It can be seen from the above related technologies that when a neural network model applies for a memory block, since the storage cycles of the applied 10M memory block and the applied 50M memory block overlap, after the applied 10M memory block reuses the allocated 100M memory block, the applied 50M memory block can no longer reuse the allocated 100M memory block, and needs to apply for a new 50M memory block. Therefore, the entire neural network model needs to occupy a total of 150M memory blocks, resulting in a large amount of memory occupied by the entire neural network model. Therefore, how to reasonably manage the memory occupied by the neural network model and improve memory utilization is an important issue that needs to be solved urgently.

Summary of the invention

The present application provides a memory management method, device, equipment, medium and product for a neural network model, and the technical solution is described as follows.

According to one aspect of the present application, a memory management method for a neural network model is provided, the method being executed by a computer device and comprising the following steps.

A computation graph corresponding to a neural network model is obtained, wherein the computation graph includes at least two network layer operators, and the network layer operators are used to represent network layers in the neural network model.

Based on the computation graph, a memory size to be allocated to the network layer operator is determined, where the memory size is used to represent the memory size that the network layer operator needs to occupy when the neural network model is running.

An allocated memory block matching the memory size is obtained from a free memory block list, and the allocated memory block is allocated to the network layer operator.

The free memory block list is used to store free memory blocks that have been allocated but released, and the allocated memory blocks refer to memory blocks allocated to the network layer operator for storing data.

According to one aspect of the present application, a memory management device for a neural network model is provided, and the device includes the following steps.

An acquisition module is used to acquire a computational graph corresponding to a neural network model, wherein the computational graph includes at least two network layer operators, and the network layer operators are used to represent network layers in the neural network model.

A determination module is used to determine the memory size to be allocated to the network layer operator based on the calculation graph, and the memory size is used to represent the memory size that the network layer operator needs to occupy when the neural network model is running.

The allocation module is used to obtain an allocated memory block matching the memory size from a free memory block list, and allocate the allocated memory block to the network layer operator.

The free memory block list is used to store free memory blocks that have been allocated but released, and the allocated memory blocks refer to memory blocks allocated to the network layer operator for storing data.

According to another aspect of the present application, a computer device is provided, which includes: a processor and a memory, wherein at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor to implement the memory management method of the neural network model as described above.

According to another aspect of the present application, a computer storage medium is provided, in which at least one computer program is stored. The at least one computer program is loaded and executed by a processor to implement the memory management method of the neural network model as described above.

According to another aspect of the present application, a computer program product is provided, which includes a computer program stored in a computer-readable storage medium; the computer program is read and executed from the computer-readable storage medium by a processor of a computer device, so that the computer device executes the memory management method of the neural network model as described above.

By obtaining the calculation graph corresponding to the neural network model; based on the calculation graph, determining the memory size to be allocated to the network layer operator; based on the memory size, obtaining the allocated memory block matching the memory size from the free memory block list, and allocating the allocated memory block to the network layer operator. This application uses the free memory blocks in the free memory block list to allocate the allocated memory block matching the memory size to the network layer operator, thereby reducing the memory allocated during the operation of the neural network model and improving the memory utilization.

Since memory utilization is improved, when a smaller memory is configured, the computer device can also achieve a neural network model running effect that is similar or the same as that with a larger memory configuration, which helps to reduce the hardware requirements of the sports neural network model for the computer device.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a schematic diagram of a memory management method for a neural network model provided by an exemplary embodiment of the present application;

FIG2 is a schematic diagram of the architecture of a computer system provided by an exemplary embodiment of the present application;

FIG3 is a flow chart of a memory management method for a neural network model provided by an exemplary embodiment of the present application;

FIG4 is a flow chart of another memory management method of a neural network model provided by an exemplary embodiment of the present application;

FIG5 is a schematic diagram of a calculation graph provided by an exemplary embodiment of the present application;

FIG6 is a schematic diagram of a method for determining an allocated memory block provided by an exemplary embodiment of the present application;

FIG7 is a schematic diagram of another method for determining an allocated memory block provided by an exemplary embodiment of the present application;

FIG8 is a schematic diagram of unallocated memory provided by an exemplary embodiment of the present application;

FIG9 is a schematic diagram of releasing a memory block provided by an exemplary embodiment of the present application;

FIG10 is a schematic diagram of reshaping performed by a shape reshaping operator provided by an exemplary embodiment of the present application;

FIG11 is a schematic diagram of splicing performed by a splicing operator provided by an exemplary embodiment of the present application;

FIG12 is a schematic diagram of splitting performed by a splitting operator provided by an exemplary embodiment of the present application;

FIG13 is a schematic diagram of another splicing operator for splicing provided by an exemplary embodiment of the present application;

FIG14 is a schematic diagram of another splitting operator performing splitting provided by an exemplary embodiment of the present application;

FIG15 is a flowchart of a memory management method of another neural network model provided by an exemplary embodiment of the present application;

FIG16 is a structural diagram of an AI chip provided by an exemplary embodiment of the present application;

FIG17 is a block diagram of a memory management device for a neural network model provided by an exemplary embodiment of the present application;

FIG. 18 is a schematic diagram of the structure of a computer device provided by an exemplary embodiment of the present application.

DETAILED DESCRIPTION

In order to make the purpose, technical solution and advantages of the present application clearer, the implementation mode of the present application will be further described in detail below with reference to the accompanying drawings. Here, the exemplary embodiments will be described in detail, and examples thereof are shown in the accompanying drawings. When the description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Instead, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.

The terms used in this disclosure are for the purpose of describing specific embodiments only and are not intended to limit the disclosure. The singular forms of "a", "said" and "the" used in this disclosure and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and/or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

It should be understood that although the terms first, second, third, etc. may be used in the present disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other.

For ease of understanding, the nouns involved in the embodiments of the present application are explained below.

Computational graph: A computational graph in a broad sense is a directed graph used to represent the computational relationship between input, output, and intermediate variables, where each node in the graph represents a mathematical operation. The computational graph of the neural network model in the embodiment of the present application is used to characterize the execution order between different network layers and the data flow between network layers during the calculation process of the neural network model. In some embodiments, the computational graph also includes the life cycle of the input and output tensors of each network layer.

The computational graph of a neural network model consists of network layer operators and the edges between network layer operators. The network layer operators correspond to the network layers in the neural network model and contain the input and output tensor sizes of the network layer operators; the edges between network layer operators are used to represent the data flow between network layers.

Life cycle: The life cycle in the embodiments of the present application refers to the life cycle of a tensor, which can be an input or output tensor of a network layer. During the life cycle, the tensor is used by the current network layer, and accordingly, the memory block storing the tensor is occupied; at the end of the life cycle, the tensor will not be used by other network layers, and the memory block storing the tensor will be released.

Free Memory Block List & Allocated Memory Block List: The free memory block list is used to record the list of free memory blocks that have been allocated but released. The allocated memory block list is used to record the list of memory blocks that have been allocated and occupied. In some embodiments, both lists are maintained by the memory management unit.

Among them, after the free memory blocks in the free memory block list are reallocated, new memory blocks will be added to the allocated memory block list; after the memory blocks in the allocated memory block list are released, new free memory blocks will be added to the free memory block list.

Data processing layer operator: refers to the operator corresponding to the network layer in the computational graph that is used to adjust the data format in the neural network model but does not change the data content. The data format adjustment includes adjusting the dimension of the tensor, such as splitting a multi-channel three-dimensional tensor into multiple single-channel two-dimensional tensors, or splicing multiple single-channel two-dimensional tensors into a single-channel two-dimensional tensor, or combining multiple single-channel two-dimensional tensors into a multi-channel three-dimensional tensor.

An embodiment of the present application provides a schematic diagram of a memory management method for a neural network model, as shown in Figure 1. The method can be executed by a computer device, which can be a terminal or a server. Specifically, the method can be executed by a memory management unit (MMU) in the computer device.

Exemplarily, the computer device obtains a computational graph 10 corresponding to the neural network model; based on the computational graph 10, the computer device determines the memory size to be allocated to the network layer operator 40; the computer device obtains an allocated memory block matching the memory size from the free memory block list 20, and allocates the allocated memory block to the network layer operator 40.

Exemplarily, the memory management unit obtains a calculation graph 10 corresponding to the neural network model; based on the calculation graph 10, the memory management unit determines the memory size to be allocated to the network layer operator 40; the memory management unit obtains an allocated memory block that matches the memory size from the free memory block list 20, and allocates the allocated memory block to the network layer operator 40.

The computational graph 10 is used to represent the computational process of the neural network model.

Optionally, the computation graph 10 includes at least two network layer operators 40 and edges 50 between at least two network layer operators, the network layer operators 40 are used to represent network layers in the neural network model, and the edges 50 are used to represent data flow between network layers.

As shown in FIG1 , the computer device obtains the computational graph 10 corresponding to the neural network model. The computational graph 10 includes at least three network layer operators 40, namely: network layer operator G0, network layer operator G1 and network layer operator G2. When the neural network model is running, the operation order of the network layer operators 40 is network layer operator G0→network layer operator G1→network layer operator G2. Among them, the memory size to be allocated to the network layer operator G0 is 16M, that is, the network layer operator G0 needs to occupy 16M when running. 16M of memory (used to store the input and output tensors of the network layer corresponding to the network layer operator G0); the memory size to be allocated to the network layer operator G1 is 10M, that is, the network layer operator G1 needs to occupy 10M of memory during runtime (used to store the input and output tensors of the network layer corresponding to the network layer operator G1); the memory size to be allocated to the network layer operator G2 is 5M, that is, the network layer operator G2 needs to occupy 5M of memory during runtime (used to store the input and output tensors of the network layer corresponding to the network layer operator G2).

The memory size is used to indicate the memory size that the network layer operator 40 needs to occupy when the neural network model is running.

The free memory block list 20 is used to store free memory blocks that have been allocated but released.

The allocated memory block refers to a memory block allocated to the network layer operator 40 for storing data.

The memory size includes the size of the input tensor and/or the size of the output tensor.

The input tensor is used to represent the multi-dimensional array input to the network layer operator 40.

The output tensor is used to represent the multidimensional array output from the network layer operator 40.

In some embodiments, the computer device may also determine the life cycles corresponding to the input tensors and output tensors of the network layer operator 40 based on the computation graph 10 .

After the current network layer operator is executed and the tensor is not called by other network layer operators, the tensor ends its life cycle and the memory block occupied by the tensor can be released.

Exemplarily, the computer device obtains the arrangement order of the network layer operator 40; when the size of the input tensor corresponding to the network layer operator 40 is greater than the tensor size threshold, the computer device obtains the allocated memory block matching the input tensor from the free memory block list 20, and allocates the allocated memory block to the network layer operator in the arrangement order for storing the input tensor.

The arrangement order is used to represent the execution order of the network layer operators 40 when the neural network model is running.

In some embodiments, the method for determining the allocated memory block includes at least one of the following methods, but is not limited thereto.

(1) When the size of the input tensor corresponding to the network layer operator 40 is less than or equal to the tensor size threshold, an allocated memory block matching the size of the input tensor is directly obtained from the unallocated memory.

(2) When the size of the input tensor is greater than the tensor size threshold and there is at least one free memory block in the free memory block list 20 whose size is greater than or equal to the size of the input tensor, obtain an allocated memory block that matches the size of the input tensor from the free memory block list 20.

(3) When the size of the input tensor is greater than the tensor size threshold and the sizes of the free memory blocks in the free memory block list 20 are all smaller than the size of the input tensor, an allocated memory block that matches the size of the input tensor is obtained from the free memory block list 20 and the unallocated memory.

(4) When the size of the input tensor is greater than the tensor size threshold and there is no free memory block in the free memory block list 20, a memory block matching the size of the input tensor is allocated from the unallocated memory as an allocated memory block.

Among them, unallocated memory refers to the memory in the storage space that has not been allocated or occupied.

The tensor size threshold refers to the size of the smallest memory block that can be reused.

Optionally, the tensor size threshold may adopt at least one of a custom value and a default value, but is not limited to this, and the embodiments of the present application do not make specific limitations on this.

Exemplarily, when the size of the input tensor corresponding to the network layer operator 40 is less than or equal to the tensor size threshold, an allocated memory block matching the size of the input tensor is directly obtained from the unallocated memory, thereby avoiding memory fragmentation.

Exemplarily, when the size of the input tensor is greater than the tensor size threshold and the free memory block list includes a first free memory block with the same size as the input tensor, the first free memory block is directly allocated as an allocated memory block to the corresponding network layer operator 40 for storing the input tensor.

When the size of the input tensor is greater than the tensor size threshold and the free memory block list 20 includes a second free memory block that is larger than the size of the input tensor, a third memory block that matches the size of the input tensor is divided from the second free memory block, and the third memory block is allocated as an allocated memory block to the corresponding network layer operator 20 for storing the input tensor.

For example, the method for determining the allocated memory block shown in FIG1 is shown in (a) of FIG1 . The shaded area in the figure is the network layer operator 40 that is running, that is, the network layer operator 40 that is currently running is the network layer operator G0. The memory size that the input tensor of the network layer operator G0 needs to occupy is 16M. Then the computer device allocates a 16M memory block to the network layer operator G0. After the allocation, the allocated memory block list 30 records the allocated 16M memory block. At this time, the free memory block list 20 As shown in (b) of FIG1 , after the network layer operator G0 is completed, the 16M memory block occupied by the network layer operator G0 is released. At this time, the free memory block list 20 records the released 16M memory block.

As shown in (c) of FIG1 , when network layer operator G1 runs before network layer operator G2, a memory block is allocated to network layer operator G1 first in the order of arrangement, and then a memory block is allocated to network layer operator G2. Assuming that the tensor size threshold is 4M, when the size of the input tensors of network layer operator G1 and network layer operator G2 are both greater than the tensor size threshold (4M), it is further determined whether the allocated memory block can be obtained from the free memory block list 20.

When the size of the input tensor of the network layer operator G1 (10M) is greater than the tensor size threshold (4M), and the size of the free memory block in the free memory block list 20 (16M) is greater than the size of the input tensor of the network layer operator G1 (10M), an allocated memory block that matches the size (10M) of the input tensor corresponding to the network layer operator G1 is obtained from the free memory block list 20, that is, the allocated memory block corresponding to the network layer operator G1 obtained is (10M).

When the size of the input tensor of the network layer operator G2 (5M) is greater than the tensor size threshold (4M), and the size of the remaining free memory blocks in the free memory block list 20 (6M) is greater than the size of the input tensor of the network layer operator G2 (5M), the allocated memory block that matches the size of the input tensor corresponding to the network layer operator G2 (5M) is obtained from the free memory block list 20, that is, the allocated memory block corresponding to the obtained network layer operator G2 is (5M). At this time, the remaining 1M memory block that is reused is recorded in the free memory block list 20, and the allocated memory block (10M) of the network layer operator G1 and the allocated memory block (5M) of the network layer operator G2 are recorded in the allocated memory block list 30.

In some embodiments, when the size of the input tensor is greater than the tensor size threshold and the free memory block list 20 includes a fourth free memory block that is smaller than the size of the input tensor, the fourth free memory block is merged with the merged memory block to obtain an allocated memory block.

The merged memory block is a memory block divided from the unallocated memory, and the size of the merged memory block is the difference between the size of the input tensor and the size of the fourth free memory block.

For example, the size of the current input tensor is 10MB, and there are two free memory blocks of 2MB and 4MB in the free memory block list, among which the 4MB free memory block is at the end of the free memory block list. Then, the 4MB free memory block is taken out and a 6MB memory block is divided from the unallocated memory to merge, and a 10MB memory block is generated as an allocated memory block and allocated to the corresponding network layer operator 40 for storing the input tensor.

In summary, the method provided in this embodiment obtains the calculation graph corresponding to the neural network model; based on the calculation graph, determines the memory size to be allocated to the network layer operator; based on the memory size, obtains the allocated memory block that matches the memory size from the free memory blocks in the free memory block list, and allocates the allocated memory block to the network layer operator. This application uses the free memory blocks in the free memory block list to allocate the allocated memory blocks that match the memory size to the network layer operator, thereby reducing the memory allocated by the neural network model during operation and improving memory utilization.

Fig. 2 shows a schematic diagram of the architecture of a computer system provided by an embodiment of the present application. The computer system may include: a terminal 100 and a server 200.

The terminal 100 may be an electronic device such as a mobile phone, a tablet computer, a vehicle terminal (vehicle computer), a wearable device, a personal computer (PC), a vehicle terminal, an aircraft, an unmanned vending terminal, etc. The terminal 100 may be installed with a client that runs a target application, and the target application may be an application for memory management of a reference neural network model, or may be other applications that provide a memory management function of a neural network model, and this application does not limit this. In addition, this application does not limit the form of the target application, including but not limited to an application (Application, App) installed in the terminal 100, a mini-program, etc., and may also be in the form of a web page.

Server 200 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services, cloud database, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and cloud servers for basic cloud computing services such as big data and artificial palm image recognition platforms. Server 200 may be a backend server of the above-mentioned target application, used to provide backend services for the client of the target application.

Cloud technology refers to a hosting technology that unifies hardware, software, network and other resources within a wide area network or local area network to achieve data computing, storage, processing and sharing. The general term for network technology, information technology, integration technology, management platform technology, application technology, etc. applied in business models, which can form a resource pool and be used on demand, flexibly and conveniently. Cloud computing technology will become an important support. The background services of technical network systems require a large amount of computing and storage resources, such as video websites, picture websites and more portal websites. With the rapid development and application of the Internet industry, in the future, each item may have its own identification mark, which needs to be transmitted to the background system for logical processing. Data of different levels will be processed separately. All kinds of industry data need strong system backing support, which can only be achieved through cloud computing.

In some embodiments, the above-mentioned server can also be implemented as a node in a blockchain system. Blockchain is a new application mode of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, the platform product service layer, and the application service layer.

The terminal 100 and the server 200 may communicate with each other via a network, such as a wired or wireless network.

In the memory management method of the neural network model provided in the embodiment of the present application, the executor of each step can be a computer device, and a computer device refers to an electronic device with data calculation, processing and storage capabilities.

In some embodiments, the computer device is a device having the requirements for running a neural network model.

In some embodiments, the computer device is provided with an AI chip, which includes an AI processor, a memory, and a memory management unit, wherein the memory management unit is used to allocate memory blocks to the network layers in the running neural network model. The memory management unit is used to implement the memory management method of the neural network model described in various embodiments of the present application.

Taking the solution implementation environment shown in Figure 2 as an example, the memory management method of the neural network model can be executed by the terminal 100 (such as the client of the target application installed and running in the terminal 100 executes the memory management method of the neural network model), or the memory management method of the neural network model can be executed by the server 200, or the terminal 100 and the server 200 can interact and cooperate to execute it, and this application does not limit this.

FIG3 is a flow chart of a memory management method for a neural network model provided by an exemplary embodiment of the present application. The method can be executed by a computer device, which can be a terminal or a server. The method includes the following steps.

Step 302: Obtain a computational graph corresponding to the neural network model.

The computational graph is used to represent the computational process of the neural network model.

Optionally, the computational graph includes at least two network layer operators and edges between at least two network layer operators, the network layer operators correspond to network layers in the neural network model, one network layer operator corresponds to one network layer, and the edges are used to indicate the data flow between network layers.

Optionally, the neural network model includes at least one of a deep learning neural network model (Deep Neural Network, DNN), a convolutional neural network model (Convolutional Neural Network, CNN), an extreme learning machine model (Extreme Learning Machine, ELM) or other neural network models, but is not limited to this, and the embodiments of the present application do not make specific limitations on this.

In a possible implementation, the computation graph is determined by a computer device by analyzing the structure of the neural network model. Optionally, the computation graph is constructed in real time during the forward propagation of the neural network model.

In another possible implementation, the calculation graph is generated in advance and input into the computer device together with the neural network model. The calculation graph can be generated manually or by a program.

Step 304: Based on the computation graph, determine the memory size to be allocated to the network layer operator.

Memory size is used to indicate the memory size that the network layer operator needs to occupy when the neural network model is running.

Exemplarily, the computer device determines the memory size to be allocated to the network layer operator based on the computation graph.

Optionally, the memory size includes the size of the input tensor and/or the size of the output tensor.

Input tensors are used to represent multidimensional arrays that are input to network layer operators.

Output tensors are multidimensional arrays representing the output from network layer operators.

In an illustrative example, the input tensor [2, 3, 4] indicates that the input tensor is a 3×4 matrix with 2 channels, i.e., it contains 2*3*4=24 elements. If each element is fp32 (i.e., each element occupies 4 bytes), the memory size occupied by the input tensor is 24*4=96 bytes.

Step 306: Obtain an allocated memory block that matches the memory size from the free memory block list, and allocate the allocated memory block to the network layer operator.

A free memory block is a memory block that has been allocated but released.

The free memory block list is used to store free memory blocks that have been allocated but released. That is, the free memory blocks are not currently occupied by the network layer.

Allocated memory blocks refer to memory blocks allocated to network layer operators for storing data.

Exemplarily, the computer device adjusts the free memory blocks in the free memory block list by reusing the free memory blocks in the free memory block list to obtain allocated memory blocks that match the memory size.

In summary, the method provided in this embodiment obtains the calculation graph corresponding to the neural network model; based on the calculation graph, determines the memory size to be allocated to the network layer operator; based on the memory size, obtains the allocated memory block that matches the memory size from the free memory blocks in the free memory block list, and allocates the allocated memory block to the network layer operator. This application allocates the allocated memory block that matches the memory size to the network layer operator by reusing the free memory blocks in the free memory block list, thereby reducing the memory allocated during the operation of the neural network model and improving the memory utilization. Since the memory utilization is improved, when a smaller memory is configured, the computer device can also achieve a neural network model operation effect that is similar or the same as that configured with a larger memory, which helps to reduce the hardware requirements of the motion neural network model for the computer device.

FIG4 is a flow chart of a memory management method for a neural network model provided by an exemplary embodiment of the present application. The method can be executed by a computer device, which can be a terminal or a server. The method includes the following steps.

Step 401: Obtain a computational graph corresponding to the neural network model.

Exemplarily, a deep learning framework or a graph compilation device parses the input neural network model and generates a corresponding computational graph.

Step 402: Based on the computation graph, determine the memory size to be allocated to the network layer operator.

Take the input tensor as an example. For example, the multidimensional array corresponding to the input tensor A is [512, 32, 32]. Assuming that the data type of each element in the input tensor A is Float 32, that is, each element occupies 4 bytes, the size of the input tensor A is: 512*32*32*4=2MB.

In the embodiment of the present application, a tensor in a neural network model is a multidimensional array. Taking the Open Neural Network Exchange (ONNX) standard as an example, a tensor is represented by a (rank, shape, data type) triple. For example, when a tensor is represented by a triple, the triple of the tensor is shown in Table 1.

Table 1. Triplet of tensor

As shown in the example in the third row of Table 1, tensor = [9, 10], which represents a two-dimensional matrix with 9 rows and 10 columns.

As shown in Table 1, a tensor is a multidimensional array. In the triplet of a tensor, the rank is used to indicate the dimension of the tensor, the shape is a representation of the tensor, and the data type is used to indicate the type of element data in the tensor shape.

Taking tensor = [9, 10] as an example, the tensor has 9*10=90 elements in total. The tensor is of Float 32 type, that is, each element occupies 4Bytes, so the size of the tensor is 9*10*4=360B=0.35KB.

A tensor can be represented in the form of an array or a shape. In the embodiment of the present application, a tensor is represented by a shape.

The shape of the tensor is [], which means a scalar with dimension 0; the shape of the tensor is [10], which means a vector with dimension 1 and 10 elements; the shape of the tensor is [9, 10], which means a matrix with dimension 2, with 9 elements in the first dimension and 10 elements in the second dimension, represented as a two-dimensional matrix with 9 rows and 10 columns.

The number of numbers in the shape indicates the dimension of the tensor. For example, if there are 4 numbers in [D0, D1, D2, D3], it means that the tensor is a 4-dimensional tensor. The number in the shape of a tensor indicates the number of elements in that dimension.

Furthermore, the size of the tensor is finally obtained by multiplying the number of elements and the bits occupied by a single element.

In some embodiments, the computer device may also determine the arrangement order corresponding to the network layer operators based on the computation graph.

The arrangement order is used to indicate the execution order of network layer operators when the neural network model is running.

Exemplarily, the computer device obtains the arrangement order of the network layer operators; when the size of the input tensor corresponding to the network layer operator is greater than the tensor size threshold, the computer device obtains the allocated memory block matching the input tensor from the free memory block list, and allocates the allocated memory block to the network layer operator in the arrangement order for storing the input tensor.

In some embodiments, the computation graph also includes the life cycle of the tensor (input or output tensor) corresponding to the network layer operator. Optionally, the life cycle of the tensor can be expressed in the following form: [the first network layer operator using the tensor, the last network layer operator using the tensor].

For example, as shown in the schematic diagram of the computation graph in FIG5, the computation graph includes five network layer operators, namely G0, G1, G2, G3, and G4. Taking the output T1 of the G0 operator as an example, T1 has 2*3*4=24 elements (assuming that T1 is of type Float 32, i.e., each element of T1 occupies 4 Bytes). It can be seen that T1 requires 24*4=96 Bytes of storage space. In addition, T1 is used as an input tensor by G1 and G3, so the life cycle of T1 ends after G3 is executed. It can be seen that the life cycle of T1 is [G0, G3], and the size and life cycle of other input tensors are the same as T1. The result of sorting the network layer operators in the order of execution is: [G0, G1, G2, G3, G4]. According to the sorting result, the allocated memory block is allocated to the network layer operator for storing the input tensor.

For example, the computer device allocates memory blocks according to the sorting result of G0-G1-G2-G3-G4. First, a 96B memory block one is allocated to the G0 operator to store T0. After the execution of the G0 operator is completed, the memory block one is released. Second, a 96B memory block two is allocated to the G1 operator to store T1. Since the storage period of T1 is [G0, G3], the memory block two is not released after the execution of the G1 operator is completed. Third, a 96B memory block three is allocated to the G2 operator to store T2. After the execution of the G2 operator is completed, the memory block three is released. Fourth, the G3 operator starts to execute. After the execution of the G3 operator is completed, the memory block two is released. Fifth, a 96B memory block four is allocated to the G4 operator to store T3. After the execution of the G4 operator is completed, the memory block five is released.

Step 403: Determine whether the size of the input tensor is less than the tensor size threshold.

The tensor size threshold refers to the size of the smallest memory block that can be reused.

Optionally, the tensor size threshold may adopt at least one of a custom value and a default value, but is not limited to this, and the embodiments of the present application do not make specific limitations on this.

Exemplarily, when the size of the input tensor is smaller than the tensor size threshold, step 409 is performed; when the size of the input tensor is greater than or equal to the tensor size threshold, step 404 is performed.

Step 404: Determine whether there is a free memory block in the free memory block list.

Exemplarily, when the free memory block list includes free memory blocks, step 405 is executed; when the free memory block list does not include free memory blocks, step 408 is executed.

Step 405: Determine whether the size of the free memory block is smaller than the size of the input tensor.

Exemplarily, when there is at least one free memory block whose size is smaller than the size of the input tensor, step 407 is executed; when the sizes of the free memory blocks are all greater than or equal to the size of the input tensor, step 406 is executed.

Step 406: Get an allocated memory block that matches the size of the input tensor from the free memory block list.

In some embodiments, when the size of the input tensor is greater than a tensor size threshold and the size of a free memory block in a free memory block list is greater than or equal to the size of the input tensor, an allocated memory block that matches the size of the input tensor is obtained from the free memory block list.

Exemplarily, when the size of the input tensor is greater than a tensor size threshold, the first free memory block in the free memory list is used as the allocated memory block; wherein the size of the first free memory block is the same as the size of the input tensor.

Or, when the size of the input tensor is greater than the tensor size threshold, a third memory block matching the size of the input tensor is divided from the second free memory block in the free memory block class table, and the third memory block is used as the allocated memory block; the size of the second free memory block is greater than the size of the input tensor.

For example, as shown in FIG6 , a schematic diagram of a method for determining an allocated memory block is shown. As shown in FIG6 (a), the input tensor 601 is 2MB, and there is a 10MB free memory block in the free memory block list 602. Then, the computer device divides the 10MB free memory block into two memory blocks of 2MB and 8MB, and allocates the 2MB memory block as an allocated memory block to the corresponding network layer operator for storing the input tensor 601. As shown in FIG6 (b), the 2MB memory block is placed in the allocated memory block list 603, and the 8MB memory block is placed back in the free memory block list 602.

Step 407: Get an allocated memory block that matches the size of the input tensor from the free memory block list and the unallocated memory.

Unallocated memory refers to the memory in the storage space that has not been allocated or occupied.

In some embodiments, when the size of the input tensor is greater than a tensor size threshold and the size of a free memory block in the free memory block list is smaller than the size of the input tensor, an allocated memory block matching the size of the input tensor is obtained from the free memory block list and the unallocated memory.

Exemplarily, when the size of the input tensor is greater than the tensor size threshold and the free memory block list includes the fourth free memory block, the fourth free memory block is merged with the merged memory block to obtain the allocated memory block.

The size of the fourth free memory block is smaller than the size of the input tensor.

A merged memory block is a memory block divided from unallocated memory.

The size of the merged memory block is the difference between the size of the input tensor and the size of the fourth free memory block.

In some embodiments, when the size of the input tensor is greater than the tensor size threshold, the fourth free memory block in the free memory list is merged with the merged memory block in the unallocated memory to obtain an allocated memory block.

For example, as shown in FIG7 , a schematic diagram of a method for determining an allocated memory block is shown. As shown in (a) of FIG7 , an input tensor 701 is 10MB, and there is a 2MB free memory block and a 4MB free memory block in a free memory block list 702. The computer device takes the 4MB free memory block from the free memory block list 702, and divides it into a 6MB merged memory block from the unallocated memory. The computer device merges the 4MB free memory block from the free memory block list 702 and the 6MB merged memory block from the unallocated memory, and uses the merged memory block as an allocated memory block for storing the input tensor 701. That is, as shown in (b) of FIG7 , a 10MB allocated memory block obtained by merging the 4MB memory block and the 6MB merged memory block is put into the allocated memory block list 703.

Step 408: A memory block matching the size of the input tensor is divided from the unallocated memory as an allocated memory block.

In some embodiments, the unallocated memory includes first-level unallocated memory and second-level unallocated memory, and the allocation priority of the first-level unallocated memory is higher than the allocation priority of the second-level unallocated memory.

In some embodiments, the first-level unallocated memory belongs to the first memory, the second-level unallocated memory belongs to the second memory, and the storage priority of the first memory is higher than the storage priority of the second memory.

Optionally, a memory access speed of the first memory is greater than a memory access speed of the second memory.

Optionally, the capacity of the first memory is smaller than the capacity of the second memory, and the hardware cost per unit storage space in the first memory is higher than the hardware cost per unit storage space in the second memory.

In some embodiments, the first memory is an L2 cache and the second memory is an L3 cache.

In some embodiments, when the size of the input tensor is greater than a tensor size threshold and there is no free memory block in the free memory block list, a memory block matching the size of the input tensor is allocated from the unallocated memory as an allocated memory block.

For example, as shown in the schematic diagram of unallocated memory in FIG8 , the neural network model runs on an AI chip, which includes multiple processor core clusters, such as a first processor core cluster 801 and a second processor core cluster 802. In order to speed up memory access, a multi-level storage architecture is usually adopted. The storage level close to the processor has a larger data transmission bandwidth, but the hardware cost is higher, so the storage space is relatively limited, which is called the first-level unallocated memory 803 or L2 cache in the embodiment of the present application. The storage level far from the processor has a smaller data transmission bandwidth, but the hardware cost is low and the storage space is larger, which is called the second-level unallocated memory 804 or L3 cache in the embodiment of the present application.

It should be noted that the first-level unallocated memory and the second-level unallocated memory each have independent allocated memory block lists and free memory block lists, and both lists are empty in the initial state.

Optionally, the memory blocks in the free memory block list are sorted from small to large, and the largest memory block is arranged at the end. Accordingly, the computer device can determine whether to store the free memory block according to the size of the free memory block at the end of the free memory block list. In a free memory block that is greater than or equal to the input tensor.

Exemplarily, when the size of the input tensor is greater than the tensor size threshold and there is no free memory block in the free memory block list, a memory block matching the size of the input tensor is allocated from the first-level unallocated memory or the second-level unallocated memory as an allocated memory block.

In some embodiments, when the size of the input tensor is greater than a tensor size threshold, there are no free memory blocks in the free memory block list, and the remaining memory in the first-level unallocated memory is greater than or equal to the size of the input tensor, a memory block matching the size of the input tensor is divided from the first-level unallocated memory as an allocated memory block.

In some embodiments, when the size of the input tensor is greater than a tensor size threshold, there are no free memory blocks in the free memory block list, and the remaining memory in the first-level unallocated memory is less than the size of the input tensor, a memory block matching the size of the input tensor is divided from the second-level unallocated memory as an allocated memory block.

Step 409: Get an allocated memory block matching the size of the input tensor from the unallocated memory.

In some embodiments, when the size of the input tensor corresponding to the network layer operator is less than or equal to the tensor size threshold, an allocated memory block matching the size of the input tensor is obtained from the unallocated memory.

Exemplarily, when the size of the input tensor corresponding to the network layer operator is less than or equal to the tensor size threshold, the memory block is not reused, and the allocated memory block matching the size of the input tensor is directly obtained from the unallocated memory. If a memory block matching the size of the input tensor can be divided from the first-level unallocated memory, it is allocated from the first-level unallocated memory first. If a memory block matching the size of the input tensor cannot be divided from the first-level unallocated memory, it is allocated from the second-level unallocated memory.

Among them, the setting of the tensor size threshold is used to reduce the generation of memory blocks corresponding to smaller input tensors.

In some embodiments, based on the computational graph, the life cycles corresponding to the input tensors and output tensors of the network layer operator are determined; in response to a memory block in the allocated memory block list reaching its life cycle, the computer device releases the memory block and places the memory block in the free memory block list.

The allocated memory block list is used to store occupied memory blocks.

The lifecycle is used to indicate the time that a tensor occupies a memory block, that is, the input tensor and output tensor are no longer used by other network layer operators after reaching their lifecycle.

Exemplarily, in response to a memory block in a list of allocated memory blocks reaching its life cycle, the computer device releases a memory block; if there is a released memory block in an adjacent position of the currently released memory block, the currently released memory block is merged with the released memory block to obtain a merged released memory block; and the merged released memory block is placed in a list of free memory blocks.

The merged released memory block refers to a memory block obtained by merging the currently released memory block and the released memory block.

In a possible implementation, the life cycle of a tensor can be expressed in the following form: [the first network layer operator that uses the tensor, the last network layer operator that uses the tensor]. The computer device determines whether the currently running network operation layer is the last network layer operator that uses the tensor in the life cycle. If so, it is determined that the life cycle has been reached, and if not, it is determined that the life cycle has not been reached.

For example, as shown in the schematic diagram of releasing memory blocks in FIG9 , as shown in (a) in FIG9 , the memory blocks of size 2M and 6M (located at the end of the list) in the primary/secondary free memory block list 902 are about to be released. As shown in (b) in FIG9 , in the case where there are 2M released memory blocks and 6M released memory blocks in the adjacent position of the 2M currently released memory block, the 2M currently released memory block is merged with the 2M and 6M released memory blocks in the primary/secondary allocated memory block list 902 to obtain a 10M merged released memory block; the 10M merged released memory block is put into the primary/secondary free memory block list 902, and only two memory blocks, 2M and 4M, are left in the primary/secondary allocated memory block list 902.

In some embodiments, the network layer operators include data processing layer operators.

The data processing layer operator is used to adjust the data format in the neural network model. The network layer corresponding to the data processing layer operator is called the data transformation layer.

The data processing layer operators include at least one of a reshape operator, a concatenation operator, and a split operator, but are not limited thereto, and the embodiments of the present application do not make specific limitations on this.

The reshape operator is used to reshape the input tensor to reshape the input tensor into a target shape, but in the process of reshaping the data, the number of elements contained in the data and the arrangement of the elements in the data are not changed.

For example, the input tensor of the input reshape operator is represented in the form of a matrix. As shown in the schematic diagram of the reshape operator performing reshape, in FIG10 , the size of the matrix of the input reshape operator is [2, 3, 4] (a 3-row 4-column matrix with 2 channels), that is, the matrix of the input reshape operator is a 2×3×4 tensor, and the size of the matrix output by the reshape operator is [6, 4] (a 6-row 4-column matrix with a single channel), that is, the reshape operator is used to transform a matrix of size [2, 3, 4] into a matrix of size [6, 4].

The splicing operator is used to splice at least two input tensors. For example, the input tensor of the splicing operator is represented in the form of a matrix, as shown in FIG11 , the schematic diagram of the splicing operator for splicing, the size of the matrix of the input splicing operator is: tensor A = [1, 3, 2] (single-channel 3-row 2-column matrix), tensor B = [2, 3, 2] (3-row 4-column matrix with 2 channels), the size of the matrix output by the splicing operator is: tensor C = [3, 3, 2] (3-row 4-column matrix with 3 channels), that is, the splicing operator is used to splice two tensors of size [1, 3, 2] and [2, 3, 2] into a matrix of [3, 3, 2].

The split operator is used to split the input tensor according to the split dimension, splitting it into at least two sub-input tensors. The split operator can be understood as an inverse process of the splicing operator. If the split dimension is the highest dimension or the dimension whose first element number is not 1, the output tensor of the split operator reuses the memory block occupied by the input tensor. For example, taking tensor A = [1, 128, 32, 32] as an example, its highest dimension is dimension 0, that is, the dimension with 1 element number; the dimension whose first element number is not 1 refers to dimension 1, that is, the dimension with 128 elements.

For example, the input tensor of the input splitting operator is represented in the form of a matrix. As shown in the schematic diagram of the splitting operator splitting in Figure 12, the size of the matrix of the input splitting operator is: tensor C = [3, 3, 2] (a 3-row 2-column matrix with 3 channels), and the size of the matrix output by the splitting operator is: tensor A = [1, 3, 2] (a 3-row 4-column matrix with a single channel), tensor B = [2, 3, 2] (a 3-row 4-column matrix with 2 channels), that is, the splitting operator is used to split a tensor of size [3, 3, 2] into matrices of [1, 3, 2] and [2, 3, 2].

Exemplarily, the computer device obtains the input tensor and output tensor corresponding to the data processing layer operator; exemplary, the computer device configures the output tensor to reuse the allocated memory block occupied by the input tensor.

Optionally, the data processing layer operator includes a reshape operator, and the output tensor includes the reshape tensor. The computer device allocates the allocated memory block occupied by the input tensor to the reshape tensor based on the allocated memory block occupied by the input tensor.

The reshape operator is used to adjust the shape of the input tensor without changing the data in the input tensor. The reshape tensor refers to the tensor output by the reshape operator.

The reshape operator reshapes the input tensor and does not change the data of the input tensor in the memory block. Therefore, when the neural network model is running, the reshape operator copies the memory data of the input tensor to the memory where the output tensor is located (that is, the memory where the input tensor of the next network layer is located). In the embodiment of the present application, the computer device can eliminate the data copy operation corresponding to the reshape operator when the neural network model is running by making the output tensor of the reshape operator reuse the memory block occupied by the input tensor of the reshape operator.

For example, tensor A = [1, 1, 512, 32, 32] passes through the reshape operator, and dimensions 3 and 4 in tensor A are merged, and the output tensor B = [1, 512, 1024] (assuming that the data types of tensor A and tensor B are both Float 32). If the computer device allocates a memory block A of 512*32*32*4=2MB for the input tensor A of the reshape operator, then the computer device allocates memory block A to tensor B when allocating memory for the output tensor B of the reshape operator. When the neural network model is running, the reshape operator in this scenario does not need to perform data transfer operations, thus avoiding consumption caused by frequent memory accesses.

Optionally, the data processing layer operator includes a concatenation operator. The computer device determines an allocated memory block occupied by the output tensor; and the computer device configures at least two input tensors to offset-reuse the allocated memory block occupied by the output tensor.

The concatenation operator concatenates two or more input tensors according to the concatenation dimension.

The splicing dimension is the highest dimension or the dimension whose first number of elements is not 1. In an embodiment of the present application, if the splicing dimension specified by the splicing operator is the highest dimension or the dimension whose first number of elements is not 1, the output tensor and input tensor of the splicing operator can reuse the same memory block. The computer device can eliminate the data copy operation corresponding to the splicing operator when the neural network model is running by making multiple input tensors of the splicing operator reuse the memory block occupied by the output tensor of the splicing operator according to the offset.

For example, as shown in the schematic diagram of the splicing operator in FIG13, tensor A = [512, 32, 32], tensor B = [256, 32, 32], assuming that after the splicing operator with a splicing dimension of 0, tensor A and tensor B are spliced into tensor C = [768, 32, 32] (assuming that the data types of tensor A, tensor B and tensor C are all Float 32), if the computer device is a splicing operator The input tensor A is allocated a memory block C of 768*32*32*4=3MB. Then the memory block C is divided into two sub-memory blocks A and sub-memory blocks B with sizes of 2MB and 1MB respectively according to the sizes of the input tensors A and B, and sub-memory blocks A and sub-memory blocks B are used to store tensors A and B respectively. Then, when the neural network model is running, the splicing operator does not need to perform data transfer operations, avoiding consumption caused by frequent memory access.

Optionally, the data processing layer operator includes a splitting operator, and the output tensor includes at least two sub-output tensors. The computer device divides the allocated memory block occupied by the input tensor to obtain sub-memory blocks corresponding to the at least two sub-input tensors respectively; and the computer device allocates the sub-memory blocks to the at least two sub-output tensors.

The splitting operator is used to divide the input tensor into at least two sub-input tensors, and the sub-output tensor refers to the tensor output by the data processing layer operator.

The splitting operator can be understood as the inverse operation of the splicing operator. The splitting operator splits the input tensor according to the splitting dimension to generate multiple output tensors. If the splitting dimension specified by the splitting operator is the highest dimension or the dimension whose first element number is not 1, the output tensor of the splitting operator reuses the memory block occupied by the input tensor. In an embodiment of the present application, the computer device can eliminate the data copy filling operation corresponding to the splitting operator when the neural network model is running by making multiple tensors of the splitting operator reuse the memory block occupied by the output tensor of the splitting operator according to the offset. For example, as shown in the schematic diagram of the splitting operator in FIG14 , tensor C = [768, 32, 32], assuming that tensor C passes through the splitting operator with a splitting dimension of 0, and the dimension 0 is split from 768 to 512 and 256, corresponding to the output tensor A = [512, 32, 32], tensor B = [256, 32, 32] (assuming that the data types of tensors A, B and C are all Float 32), if the computer device allocates a memory block C of 768*32*32*4=3MB for the input tensor A of the splitting operator, then the memory block C is divided into two sub-memory blocks A and sub-memory blocks B with sizes of 2MB and 1MB respectively according to the sizes of the input tensors A and B, and the sub-memory blocks A and B are allocated to tensors A and B respectively. Then, when the neural network model is running, the splitting operator in this scenario does not need to perform data transfer operations, thereby avoiding consumption caused by frequent memory accesses.

In summary, the method provided in this embodiment obtains the calculation graph corresponding to the neural network model; based on the calculation graph, determines the memory size to be allocated to the network layer operator; based on the memory size, obtains the allocated memory block that matches the memory size from the free memory blocks in the free memory block list, and allocates the allocated memory block to the network layer operator. This application allocates the allocated memory block that matches the memory size to the network layer operator by reusing the free memory blocks in the free memory block list, thereby reducing the memory allocated by the neural network model during operation and improving memory utilization.

The method provided in this embodiment determines different acquisition methods by judging the size between the size of the input tensor and the tensor size threshold; based on the different acquisition methods, the allocated memory block matching the size of the input tensor is obtained from the free memory block list, thereby reducing the memory allocated during the operation of the neural network model and improving memory utilization.

The method provided in this embodiment reduces the memory allocated during the operation of the neural network model and improves memory utilization by combining a free memory block list and unallocated memory to obtain an allocated memory block that matches the size of the input tensor.

The method provided in this embodiment directly obtains an allocated memory block that matches the size of the input tensor from unallocated memory when the size of the input tensor is less than or equal to the tensor size threshold, thereby avoiding the generation of small memory blocks and improving memory utilization.

The method provided in this embodiment, when releasing a memory block, merges the currently released memory block with the released memory block to obtain a large merged released memory block, and puts the merged released memory block into a free memory block list. By merging scattered free memory blocks through the above method, the free memory blocks in the free memory block list can be applied to a variety of allocation scenarios, thereby improving the allocation efficiency of the memory block.

The method provided in this embodiment enables the input and output of the data processing layer operators in the neural network model to reuse the same memory block, thereby reducing the data transfer overhead when the neural network model is running and improving memory utilization.

Figure 15 is a flow chart of a memory management method for a neural network model provided by an exemplary embodiment of the present application. The method can be executed by a computer device, which can be a terminal or a server. The method includes the following steps.

Step 1501: Get the size of the input tensor to be allocated to the current network layer operator.

Input tensors are used to represent multidimensional arrays that are input to network layer operators.

Taking the ONNX standard as an example, a tensor is represented by a triple of (rank, shape, node type).

The size of the input tensor is used to indicate the memory size that the input data of the network layer operator needs to occupy when the neural network model is running.

Exemplarily, the computer device determines the size of the input tensor to be allocated to the network layer operator based on the computation graph.

Step 1502: Determine whether the size of the input tensor is less than the tensor size threshold.

The tensor size threshold refers to the size of the smallest memory block that can be reused.

The computer device determines whether the size of the input tensor is less than the tensor size threshold. If the size of the input tensor is less than the tensor size threshold, execute step 1508; if the size of the input tensor is greater than or equal to the tensor size threshold, execute step 1503.

Step 1503: Get the largest free memory block in the free memory block list.

The free memory block list is used to store free memory blocks that have been allocated but released.

Exemplarily, when the size of the input tensor is greater than or equal to the tensor size threshold, the computer device obtains the largest free memory block in the free memory block list.

Step 1504: Determine whether the largest free memory block is greater than or equal to the size of the input tensor.

Exemplarily, after obtaining the largest free memory block in the free memory block list, the computer device determines whether the largest free memory block is greater than or equal to the size of the input tensor. If the largest free memory block is greater than or equal to the size of the input tensor, step 1505 is executed; if the largest free memory block is smaller than the size of the input tensor, step 1506 is executed.

Step 1505: Divide the largest free memory block into two memory blocks, one matching the size of the input tensor and allocated to the network layer operator, and the other put back into the free memory block list.

Exemplarily, when the largest free memory block is greater than or equal to the size of the input tensor, the computer device divides the largest free memory block into two memory blocks, one of which matches the size of the input tensor to obtain an allocated memory block that matches the size of the input tensor, and allocates the allocated memory block to the network layer operator; the other is put back into the free memory block list for next use.

Step 1506: Determine whether the largest free memory block is at the end.

The free memory blocks in the free memory block list are arranged in order from small to large.

Exemplarily, when the largest free memory block is smaller than the size of the input tensor, determine whether the largest free memory block is at the end, that is, determine whether there are free memory blocks in the free memory block list; when the largest free memory block is at the end, execute step 1507; when the largest free memory block is not at the end, execute step 1508.

Step 1507: Take out the largest free memory block, divide it from the unallocated memory to obtain a merged memory block, merge the largest free memory block and the merged memory block, and allocate them to the network layer operator.

Exemplarily, when the largest free memory block is at the end, the largest free memory block is taken out and divided from the unallocated memory to obtain a merged memory block, and the largest free memory block and the merged memory block are merged and allocated to the network layer operator.

Step 1508: Determine whether the remaining memory in the first-level unallocated memory is greater than/equal to the size of the input tensor.

Exemplarily, when the largest free memory block is not at the end, it is further determined whether the remaining memory in the first-level unallocated memory is greater than/equal to the size of the input tensor. When the remaining memory in the first-level unallocated memory is greater than/equal to the size of the input tensor, step 1509 is executed; when the remaining memory in the first-level unallocated memory is less than the size of the input tensor, step 1510 is executed.

Step 1509: Divide a memory block that matches the size of the input tensor from the first-level unallocated memory as an allocated memory block, and allocate it to the network layer operator.

Exemplarily, when the remaining memory in the first-level unallocated memory is greater than/equal to the size of the input tensor, a memory block matching the size of the input tensor is directly divided from the first-level unallocated memory as an allocated memory block and allocated to the network layer operator.

Step 1510: Divide a memory block that matches the size of the input tensor from the secondary unallocated memory as an allocated memory block, and allocate it to the network layer operator.

Exemplarily, when the remaining memory in the first-level unallocated memory is smaller than the size of the input tensor, a memory block matching the size of the input tensor is divided from the second-level unallocated memory as an allocated memory block and allocated to the network layer operator.

FIG16 is a structural diagram of an AI chip provided by an exemplary embodiment of the present application. The AI chip includes an AI processor 1601 , a memory 1602 , and a memory management unit 1603 .

The AI processor 1601 is used to run the neural network model. Optionally, the AI processor 1601 may include multiple processor core clusters, each of which includes multiple processor cores.

Optionally, the memory 1602 may be composed of multiple levels of cache. For example, the memory 1602 may be composed of L2 and L3 caches.

The memory management unit 1603 is used to allocate memory blocks to the network layers in the running neural network model. The memory management unit 1603 is used to implement the memory management method of the neural network model described in various embodiments of the present application.

Fig. 17 shows a schematic diagram of the structure of a memory management device for a neural network model provided by an exemplary embodiment of the present application. The device includes the following modules.

The acquisition module 1701 is used to obtain a calculation graph corresponding to the neural network model, wherein the calculation graph includes at least two network layer operators, and the network layer operators are used to represent the network layers in the neural network model.

Determination module 1702 is used to determine the memory size to be allocated to the network layer operator based on the calculation graph, and the memory size is used to represent the memory size that the network layer operator needs to occupy when the neural network model is running.

The allocation module 1703 is used to obtain an allocated memory block matching the memory size from the free memory block list, and allocate the allocated memory block to the network layer operator.

The free memory block list is used to store free memory blocks that have been allocated but released, and the allocated memory blocks refer to memory blocks allocated to the network layer operator for storing data.

In some embodiments, the acquisition module 1701 is used to obtain the arrangement order of the network layer operators, and the arrangement order is used to represent the execution order of the network layer operators when the neural network model is running.

In some embodiments, the allocation module 1703 is used to obtain the allocated memory block matching the size of the input tensor from the free memory block list when the size of the input tensor of the network layer operator is greater than a tensor size threshold, and the tensor size threshold is the minimum memory block size for memory reuse.

In some embodiments, the allocation module 1703 is used to allocate the allocated memory blocks to the network layer operators for storing the input tensors in accordance with the arrangement order.

The input tensor refers to a multidimensional array input into the network layer operator.

In some embodiments, the allocation module 1703 is used to obtain the allocated memory block that matches the size of the input tensor from the free memory block list when the size of the input tensor is greater than the tensor size threshold and there is at least one free memory block in the free memory block list whose size is greater than or equal to the size of the input tensor.

In some embodiments, the allocation module 1703 is used to use the first free memory block in the free memory block list as the allocated memory block when the size of the input tensor is greater than the tensor size threshold.

In some embodiments, the allocation module 1703 is used to divide a third memory block matching the size of the input tensor from the second free memory block in the free memory block list when the size of the input tensor is greater than the tensor size threshold, and use the third memory block as the allocated memory block.

The size of the first free memory block is the same as the size of the input tensor, and the size of the second free memory block is larger than the size of the input tensor.

In some embodiments, the allocation module 1703 is used to obtain the allocated memory block matching the size of the input tensor from the free memory block list and the unallocated memory when the size of the input tensor is greater than the tensor size threshold and the size of the free memory block in the free memory block list is smaller than the size of the input tensor.

The unallocated memory refers to the memory in the storage space that has not been allocated or occupied.

In some embodiments, the allocation module 1703 is used to merge the fourth free memory block in the free memory block list with the merged memory block in the unallocated memory to obtain the allocated memory block when the size of the input tensor is greater than the tensor size threshold.

The size of the fourth free memory block is smaller than the size of the input tensor, the merged memory block is a memory block obtained by dividing the unallocated memory, and the size of the merged memory block is the sum of the size of the input tensor and the size of the fourth free memory block. The difference in the sizes of free memory blocks.

In some embodiments, the allocation module 1703 is used to allocate a memory block matching the size of the input tensor from the unallocated memory as the allocated memory block when the size of the input tensor is greater than the tensor size threshold and there is no free memory block in the free memory block list.

The unallocated memory includes first-level unallocated memory and second-level unallocated memory, and the allocation priority of the first-level unallocated memory is higher than the allocation priority of the second-level unallocated memory.

In some embodiments, the allocation module 1703 is used to allocate a memory block matching the size of the input tensor from the first-level unallocated memory or the second-level unallocated memory as the allocated memory block when the size of the input tensor is greater than the tensor size threshold and there is no free memory block in the free memory block list.

In some embodiments, the allocation module 1703 is used to allocate a memory block matching the size of the input tensor from the first-level unallocated memory as the allocated memory block when the size of the input tensor is greater than the tensor size threshold, there are no free memory blocks in the free memory block list, and the remaining memory in the first-level unallocated memory is greater than or equal to the size of the input tensor.

In some embodiments, the allocation module 1703 is used to allocate a memory block matching the size of the input tensor from the secondary unallocated memory as the allocated memory block when the size of the input tensor is greater than the tensor size threshold, there is no free memory block in the free memory block list, and the remaining memory in the first-level unallocated memory is less than the size of the input tensor.

In some embodiments, the allocation module 1703 is used to obtain the allocated memory block matching the size of the input tensor from the unallocated memory when the size of the input tensor of the network layer operator is less than or equal to the tensor size threshold, wherein the unallocated memory refers to the memory in the storage space that has not been allocated and occupied.

In some embodiments, the determination module 1702 is used to determine the life cycle corresponding to the input tensor and the output tensor of the network layer operator based on the computation graph, wherein the input tensor and the output tensor are no longer used by other network layer operators after reaching the life cycle.

In some embodiments, the apparatus further comprises a release module 1704, and the release module 1704 is used for releasing the memory block in response to the memory block in the allocated memory block list reaching the storage period, and putting the memory block into the free memory block list.

The allocated memory block list is used to store occupied memory blocks.

In some embodiments, the release module 1704 is used to release the memory block in response to the memory block in the allocated memory block list reaching the life cycle.

In some embodiments, the device further includes a merging module 1705, which is used to merge the currently released memory block with the released memory block to obtain a merged released memory block when there is a released memory block adjacent to the currently released memory block.

In some embodiments, the merging module 1705 is used to put the merged released memory block into the free memory block list.

In some embodiments, the acquisition module 1701 is used to obtain the input tensor and output tensor corresponding to the data processing layer operator.

In some embodiments, the apparatus further comprises a multiplexing module 1706, the multiplexing module 1706 being configured to configure the output tensor to reuse the allocated memory block occupied by the input tensor.

In some embodiments, the multiplexing module 1706 is configured to allocate the allocated memory block occupied by the input tensor to the reshaped tensor based on the allocated memory block occupied by the input tensor.

The reshape operator is used to adjust the shape of the input tensor but does not change the data in the input tensor, and the reshape tensor refers to the tensor output by the reshape operator.

In some embodiments, the multiplexing module 1706 is used to divide the allocated memory block occupied by the input tensor to obtain sub-memory blocks corresponding to each of the at least two sub-input tensors; and allocate the sub-memory blocks to the at least two sub-output tensors.

The splitting operator is used to split the input tensor into at least two sub-input tensors, and the sub-output tensors Refers to the tensor output by the operator of the data processing layer.

In some embodiments, the multiplexing module 1706 is used to determine the allocated memory block occupied by the output tensor; configure at least two of the input tensors to offset multiplex the allocated memory block occupied by the output tensor;

The concatenation operator is used to concatenate at least two of the input tensors.

FIG18 shows a block diagram of a computer device 1800 shown in an exemplary embodiment of the present application. The computer device can be implemented as a server in the above-mentioned solution of the present application. The computer device 1800 includes a central processing unit (CPU) 1801, a system memory 1804 including a random access memory (RAM) 1802 and a read-only memory (ROM) 1803, and a system bus 1805 connecting the system memory 1804 and the central processing unit 1801. The computer device 1800 also includes a large-capacity storage device 1806 for storing an operating system 1809, an application program 1810, and other program modules 1811.

The mass storage device 1806 is connected to the central processing unit 1801 through a mass storage controller (not shown) connected to the system bus 1805. The mass storage device 1806 and its associated computer readable medium provide non-volatile storage for the computer device 1800. That is, the mass storage device 1806 may include a computer readable medium (not shown) such as a hard disk or a Compact Disc Read-Only Memory (CD-ROM) drive.

Without loss of generality, the computer-readable medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer storage media include RAM, Erasable Programmable Read Only Memory (EPROM), Electronically-Erasable Programmable Read-Only Memory (EEPROM) flash memory or other solid-state storage technology, CD-ROM, Digital Versatile Disc (DVD) or other optical storage, tape cassettes, magnetic tapes, disk storage or other magnetic storage devices. Of course, those skilled in the art will appreciate that the computer storage media is not limited to the above. The above-mentioned system memory 1804 and mass storage device 1806 can be collectively referred to as memory.

Optionally, the memory includes a first-level unallocated memory (not shown) and a second-level unallocated memory (not shown). In order to speed up memory access, the central processing unit 1801 usually adopts a multi-level storage architecture. The first-level unallocated memory is a storage layer close to the processor and has a larger data transmission bandwidth, but the hardware cost is higher, so the storage space is relatively limited; the second-level unallocated memory is a storage layer far from the processor and has a smaller data transmission bandwidth, but the hardware cost is low and the storage space is larger.

According to various embodiments of the present disclosure, the computer device 1800 can also be connected to a remote computer on the network through a network such as the Internet. That is, the computer device 1800 can be connected to the network 1808 through the network interface unit 1807 connected to the system bus 1805, or the network interface unit 1807 can be used to connect to other types of networks or remote computer systems (not shown).

The memory also includes at least one computer program, which is stored in the memory. The central processing unit 1801 implements all or part of the steps in the memory management method of the neural network model shown in the above-mentioned embodiments by executing the at least one program.

An embodiment of the present application also provides a computer device, which includes a processor and a memory, wherein the memory stores at least one program, and the at least one program is loaded and executed by the processor to implement the memory management method of the neural network model provided by the above-mentioned method embodiments.

An embodiment of the present application also provides a computer-readable storage medium, which stores at least one computer program. The at least one computer program is loaded and executed by a processor to implement the memory management method of the neural network model provided by the above-mentioned method embodiments.

An embodiment of the present application also provides a computer program product, which includes a computer program, and the computer program is stored in a computer-readable storage medium; the computer program is read and executed from the computer-readable storage medium by a processor of a computer device, so that the computer device executes to implement the memory management method of the neural network model provided in the above-mentioned method embodiments.

It is understood that in the specific implementation of this application, the data, historical data, and portraits involved are related to For user data processing related to user identity or characteristics and other related data, when the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

It should be understood that the "plurality" mentioned in this article refers to two or more. "And/or" describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and/or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character "/" generally indicates that the associated objects are in an "or" relationship.

A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent switching, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims (21)

A memory management method for a neural network model, the method being executed by a computer device, the method comprising: Obtaining a computational graph corresponding to a neural network model, wherein the computational graph includes at least two network layer operators, and the network layer operators are used to represent network layers in the neural network model; Based on the computation graph, determining a memory size to be allocated to the network layer operator, wherein the memory size is used to indicate a memory size that the network layer operator needs to occupy when the neural network model is running; Acquire an allocated memory block matching the memory size from a free memory block list, and allocate the allocated memory block to the network layer operator; The free memory block list is used to store free memory blocks that have been allocated but released, and the allocated memory blocks refer to memory blocks allocated to the network layer operator for storing data. The method according to claim 1, wherein the memory size includes the size of the input tensor of the network layer operator; The acquiring an allocated memory block matching the memory size from a free memory block list and allocating the allocated memory block to the network layer operator comprises: Obtaining an arrangement order of the network layer operators, wherein the arrangement order is used to represent an execution order of the network layer operators when the neural network model is running; When the size of the input tensor of the network layer operator is greater than a tensor size threshold, obtaining the allocated memory block matching the size of the input tensor from the free memory block list, wherein the tensor size threshold is a minimum memory block size for memory reuse; Allocating the allocated memory blocks to the network layer operators for storing the input tensors according to the arrangement order; The input tensor refers to a multidimensional array input into the network layer operator. The method according to claim 2, wherein, when the size of the input tensor of the network layer operator is greater than a tensor size threshold, obtaining the allocated memory block matching the size of the input tensor from the free memory block list comprises: When the size of the input tensor is greater than the tensor size threshold and there is at least one free memory block in the free memory block list whose size is greater than or equal to the size of the input tensor, the allocated memory block matching the size of the input tensor is obtained from the free memory block list. The method according to claim 3, wherein, when the size of the input tensor is greater than the tensor size threshold, and there is at least one free memory block in the free memory block list whose size is greater than or equal to the size of the input tensor, obtaining the allocated memory block matching the size of the input tensor from the free memory block list comprises: When the size of the input tensor is greater than the tensor size threshold, using the first free memory block in the free memory block list as the allocated memory block; or, when the size of the input tensor is greater than the tensor size threshold, dividing a third memory block matching the size of the input tensor from a second free memory block in the free memory block list, and using the third memory block as the allocated memory block; The size of the first free memory block is the same as the size of the input tensor, and the size of the second free memory block is larger than the size of the input tensor. The method according to claim 3 or 4, wherein the method further comprises: When the size of the input tensor is greater than the tensor size threshold and the size of the free memory block in the free memory block list is smaller than the size of the input tensor, acquiring the allocated memory block matching the size of the input tensor from the free memory block list and unallocated memory; The unallocated memory refers to the memory in the storage space that has not been allocated or occupied. The method according to claim 5, wherein, when the size of the input tensor is greater than the tensor size threshold, and the size of the free memory block in the free memory block list is smaller than the size of the input tensor, Acquiring the allocated memory block matching the size of the input tensor from the free memory block list and the unallocated memory, comprising: When the size of the input tensor is greater than the tensor size threshold, merging the fourth free memory block in the free memory block list with the merged memory block in the unallocated memory to obtain the allocated memory block; Among them, the size of the fourth free memory block is smaller than the size of the input tensor, the merged memory block is a memory block divided from the unallocated memory, and the size of the merged memory block is the difference between the size of the input tensor and the size of the fourth free memory block. The method according to any one of claims 3 to 6, wherein the method further comprises: When the size of the input tensor is greater than the tensor size threshold and there is no free memory block in the free memory block list, a memory block matching the size of the input tensor is allocated from unallocated memory as the allocated memory block. The method according to claim 7, wherein the unallocated memory comprises a first-level unallocated memory and a second-level unallocated memory, and the allocation priority of the first-level unallocated memory is higher than the allocation priority of the second-level unallocated memory; When the size of the input tensor is greater than the tensor size threshold and there is no free memory block in the free memory block list, dividing a memory block matching the size of the input tensor from the unallocated memory as the allocated memory block includes: When the size of the input tensor is greater than the tensor size threshold and there is no free memory block in the free memory block list, a memory block matching the size of the input tensor is allocated from the first-level unallocated memory or the second-level unallocated memory as the allocated memory block. The method according to claim 8, wherein, when the size of the input tensor is greater than the tensor size threshold and there is no free memory block in the free memory block list, allocating a memory block matching the size of the input tensor from the first-level unallocated memory or the second-level unallocated memory as the allocated memory block, comprises: When the size of the input tensor is greater than the tensor size threshold, there is no free memory block in the free memory block list, and the remaining memory in the first-level unallocated memory is greater than or equal to the size of the input tensor, a memory block matching the size of the input tensor is allocated from the first-level unallocated memory as the allocated memory block. The method according to claim 8 or 9, wherein, when the size of the input tensor is greater than the tensor size threshold and there is no free memory block in the free memory block list, allocating a memory block matching the size of the input tensor from the first-level unallocated memory or the second-level unallocated memory as the allocated memory block, comprises: When the size of the input tensor is greater than the tensor size threshold, there is no free memory block in the free memory block list, and the remaining memory in the first-level unallocated memory is less than the size of the input tensor, a memory block matching the size of the input tensor is allocated from the second-level unallocated memory as the allocated memory block. The method according to any one of claims 2 to 10, wherein the method further comprises: When the size of the input tensor of the network layer operator is less than or equal to a tensor size threshold, the allocated memory block matching the size of the input tensor is obtained from unallocated memory, where the unallocated memory refers to memory in the storage space that has not been allocated or occupied. The method according to any one of claims 1 to 11, wherein the method further comprises: Based on the computation graph, determining a life cycle corresponding to an input tensor and an output tensor of the network layer operator, wherein the input tensor and the output tensor are no longer used by other network layer operators after reaching the life cycle; In response to a memory block in the allocated memory block list reaching the life cycle, releasing the memory block and placing the memory block into the free memory block list; The allocated memory block list is used to store occupied memory blocks. The method according to claim 12, wherein, in response to a memory block in the allocated memory block list reaching the life cycle, releasing the memory block and placing the memory block in the free memory block list comprises: In response to a memory block in the allocated memory block list reaching the life cycle, releasing the memory block; In the case that there is a released memory block at an adjacent position of the currently released memory block, merging the currently released memory block with the released memory block to obtain a merged released memory block; The merged released memory block is put into the free memory block list. The method according to any one of claims 1 to 13, wherein the network layer operator includes a data processing layer operator, and the data processing layer operator is used to adjust the data format in the neural network model; the method further comprises: Obtaining input tensors and output tensors of the data processing layer operator; The output tensor is configured to reuse the allocated memory block occupied by the input tensor. The method of claim 14, wherein the data processing layer operator comprises a reshape operator and the output tensor comprises a reshape tensor; The configuring the output tensor to reuse the allocated memory block occupied by the input tensor comprises: allocating the allocated memory block occupied by the input tensor to the reshaped tensor based on the allocated memory block occupied by the input tensor; The reshape operator is used to adjust the shape of the input tensor but does not change the data in the input tensor, and the reshape tensor refers to the tensor output by the reshape operator. The method of claim 14, wherein the data processing layer operator comprises a split operator, and the output tensor comprises at least two sub-output tensors; The configuring the output tensor to reuse the allocated memory block occupied by the input tensor comprises: Dividing the allocated memory block occupied by the input tensor to obtain sub-memory blocks corresponding to the at least two sub-input tensors; Allocating the sub-memory block to the at least two sub-output tensors; The splitting operator is used to divide the input tensor into at least two sub-input tensors, and the sub-output tensor refers to the tensor output by the data processing layer operator. The method according to claim 14, wherein the data processing layer operator comprises a concatenation operator; the method further comprising: Determining the allocated memory block occupied by the output tensor; Configuring at least two of the input tensors to offset-reuse the allocated memory block occupied by the output tensor; The concatenation operator is used to concatenate at least two of the input tensors. A memory management device for a neural network model, the device comprising: An acquisition module, used to acquire a computational graph corresponding to a neural network model, wherein the computational graph includes at least two network layer operators, and the network layer operators are used to represent network layers in the neural network model; A determination module, used to determine the memory size to be allocated to the network layer operator based on the calculation graph, wherein the memory size is used to indicate the memory size that the network layer operator needs to occupy when the neural network model is running; An allocation module, configured to obtain an allocated memory block matching the memory size from a free memory block list, and allocate the allocated memory block to the network layer operator; The free memory block list is used to store free memory blocks that have been allocated but released, and the allocated memory blocks refer to memory blocks allocated to the network layer operator for storing data. A computer device, comprising: a processor and a memory, wherein at least one computer program is stored in the memory, and at least one computer program is loaded and executed by the processor to implement the memory management method of the neural network model as described in any one of claims 1 to 17. A computer storage medium, wherein at least one computer program is stored in the computer-readable storage medium, and the at least one computer program is loaded and executed by a processor to implement the memory management method of a neural network model as described in any one of claims 1 to 17. A computer program product, comprising a computer program, wherein the computer program is stored in a computer-readable storage medium; the computer program is read and executed from the computer-readable storage medium by a processor of a computer device, so that the computer device executes the memory management method of a neural network model as described in any one of claims 1 to 17.