WO2024247053A1 - Layer integration determination device, method, and program - Google Patents

Abstract

A layer integration determination device (10), wherein: a generation unit (21) generates a computation-integrated model by integrating computations corresponding to each of a convolution layer that can be processed by dedicated hardware that executes inference processing of a deep learning model and a peripheral layer accompanying the convolution layer, among computations corresponding to layers of a deep learning model which includes a plurality of convolution layers and which is input in the form of a calculation graph in which the layers are represented by nodes; an extraction unit (26) extracts a partial graph matching a pattern registered in advance in a pattern description unit (42) as a combination of convolution layers for which layer integration is possible, on the basis of the layer configuration of each of the convolution layers in the computation-integrated model; and a determination unit (28) determines, as a layer integration section, a section corresponding to a partial graph that satisfies a condition registered in advance in a condition description unit (44) as a condition for layer integration based on the specifications of the AI chip.

Description

Layer integration determination device, method, and program

The disclosed technology relates to a layer integration determination device, a layer integration determination method, and a layer integration determination program.

Deep learning is a machine learning method that uses a neural network that reproduces the mechanism of human nerve cells, and is applied to fields such as video processing and natural language processing. For example, in the field of video processing, an image recognition method has been proposed that uses a deep learning model called a convolutional neural network (CNN) to determine the class of an object in an image. In addition, many other methods have been proposed using deep learning models, such as object detection, which determines the position and class of an object in an image, and segmentation, which infers the class of an object for each pixel. In the field of natural language processing, methods using models equipped with an attention mechanism, such as the Transformer, are applied to tasks such as machine translation and summarization. Such deep learning models have achieved performance that exceeds that of conventional machine learning models, and there is a movement to utilize deep learning in various fields such as medicine and industry.

Factors behind the improved performance of these deep learning models include the improvement of computer computing power and the development of cloud technology. For example, large-scale parallel calculations are now possible by repurposing GPUs (Graphical Processing Units). In addition, the emergence of cloud services equipped with many GPUs, such as GCP (Google Cloud Platform) and AWS (Amazon Web Service), has made it easier to train large-scale deep learning models.

When using a trained model to perform inference processing on data acquired by a device such as an in-vehicle camera or a smartphone, there are two methods: performing inference on the cloud, or performing inference on the device that acquired the data. With the former method, real-time performance is compromised due to network delays, and sending data to the cloud via the internet can pose security risks and privacy violations. For this reason, in recent years, attention has been focused on edge AI (Artificial Intelligence), which performs inference processing on the device that acquired the data.

Edge AI enables real-time data processing while protecting privacy, but one issue is the difficulty of securing power sources and computing resources. In particular, when performing inference processing on mobile objects such as drones and smartphones, it is difficult to install devices that consume large amounts of power, such as GPUs, due to weight and other restrictions. Therefore, by installing specialized hardware for inference processing, known as an AI chip, on the terminal, it is possible to reduce power consumption while securing the computing resources required for edge AI.

Also, to efficiently use limited computing resources, a technique called layer integration has been introduced that integrates and processes the calculations of multiple layers included in a deep learning model (Non-Patent Document 1). In layer integration, the convolution calculations of a deep learning model are not processed independently for each layer, but multiple convolution layers are integrated and processed. This makes it possible to process calculations using a small-capacity cache, thereby reducing access to external memory, shortening processing time, and improving power efficiency. A technique for automating this integration of calculations has also been proposed (Non-Patent Document 2).

F. Indirli, A. Erdem and C. Silvano, "A Tile-based Fused-layer CNN Accelerator for FPGAs," 2020 27th IEEE International Conference on Electronics, Circuits and Systems (ICECS), Glasgow, UK, 2020, pp. 1 -4. Zhi Chen and Cody Yu, "How to Bring Your Own Codegen to TVM," [online], Jul 15, 2020, Amazon Web Services, Inc., [Retrieved April 12, 2023], Internet <URL: https://tvm.apache.org/2020/07/15/how-to-bring-your-own-codegen-to-tvm>

　Although conventional methods can automate the integration of primitive operations in deep learning models, they do not take into account layer integration of deep learning models that include multiple convolutional layers. Therefore, when integrating layers of deep learning models that include multiple convolutional layers using conventional methods, it is necessary to analyze the target deep learning model in detail and make a layer integration decision for each deep learning model individually, which poses the issue of time-consuming layer integration decisions.

The disclosed technology has been developed in consideration of the above points, and aims to flexibly support a variety of deep learning models while shortening the time required to determine layer integration for deep learning models that include multiple convolutional layers.

A first aspect of the present disclosure is a layer integration determination device, which includes a generation unit that integrates operations corresponding to each of the convolutional layers and the surrounding layers associated with the convolutional layers that can be processed by dedicated hardware that executes the inference processing of the deep learning model, among the operations corresponding to each layer of a deep learning model including multiple convolutional layers input in the form of a computation graph in which each layer is represented by a node, to generate a computationally integrated model; an extraction unit that extracts a subgraph from the computationally integrated model that matches a pattern registered in advance in a pattern description unit as a combination of convolutional layers that can be layer integrated based on the layer configuration of each of the convolutional layers; and a determination unit that determines, as a layer integration section, a section corresponding to the subgraph that satisfies a condition registered in advance in a condition description unit as a condition for layer integration based on the specifications of the dedicated hardware.

The second aspect of the present disclosure is a layer integration determination method executed by a layer integration determination device including a generation unit, an extraction unit, and a determination unit, in which the generation unit integrates operations corresponding to each layer of a deep learning model including multiple convolution layers, which are input in the form of a computation graph in which each layer is represented by a node, and operations corresponding to each of the convolution layers that can be processed by dedicated hardware that executes the inference processing of the deep learning model and the surrounding layers associated with the convolution layers, to generate a computation-integrated model, the extraction unit extracts a subgraph from the computation-integrated model that matches a pattern registered in advance in a pattern description unit as a combination of convolution layers that can be layer-integrated based on the layer configuration of each of the convolution layers, and the determination unit determines, as a layer integration section, a section corresponding to the subgraph that satisfies a condition registered in advance in a condition description unit as a condition for layer integration based on the specifications of the dedicated hardware.

The third aspect of the present disclosure is a layer integration determination program that causes a computer to function as each part of the layer integration determination device described above.

The disclosed technology can flexibly accommodate a variety of deep learning models while reducing the time required to determine layer integration for deep learning models that include multiple convolutional layers.

FIG. 1 is a block diagram showing a configuration of a conventional arithmetic integration device. FIG. 1 is a diagram for explaining layers included in a deep learning model. 1 is a flowchart showing the flow of a conventional operation integration process. 1 is a block diagram showing a hardware configuration of a layer integration determination device according to the first and second embodiments. FIG. 1 is a block diagram showing an example of a functional configuration of a layer integration determination device according to a first embodiment. FIG. FIG. 13 is a diagram showing an example of a pattern registered in a pattern description section; 13A and 13B are diagrams illustrating other examples of patterns registered in the pattern description section. 13 is a flowchart showing a flow of a layer integration determination process according to the first embodiment. 13 is a flowchart showing the flow of a layer-merged model candidate generation process. FIG. 11 is a block diagram showing an example of a functional configuration of a layer integration determination device according to a second embodiment. 13 is a flowchart showing the flow of a layer integration determination process according to the second embodiment.

Below, an example of an embodiment of the disclosed technology will be described with reference to the drawings. Note that the same reference symbols are used for identical or equivalent components and parts in each drawing. Also, the dimensional ratios in the drawings have been exaggerated for the convenience of explanation and may differ from the actual ratios.

<About conventional methods>
First, before describing each embodiment in detail, a conventional method will be described.

FIG. 1 shows the configuration of a computation integration device 1000 that uses a conventional method to automate the integration of computations in a deep learning model, such as the method described in Non-Patent Document 2.

The deep learning model is input to the computation integration device 1000 in the form of a computation graph in which each layer of the deep learning model is represented by a node. The computation integration device 1000 scans the input computation graph, performs processing to integrate layers corresponding to computations that can be integrated, and outputs a computationally integrated model. As shown in FIG. 1, the computation integration device 1000 functionally includes one description unit called the correspondence description unit 1040, and two processing units, a labeling unit 1022 and a computation integration unit 1024.

Generally, when inference processing of a machine learning model is performed using hardware, the calculations of the convolutional (Conv) layer as shown in Figure 2 are integrated with the calculations of the surrounding layers associated with the convolutional layer to improve calculation efficiency. In the example of Figure 2, the surrounding layers associated with the convolutional layer are the padding (Pad) layer, the batch normalization (BN) layer, and the activation function (ReLU) layer.

Then, in the correspondence description unit 1040, operations that can be processed by dedicated hardware (hereinafter, also referred to as an "AI chip" as an example) that executes the inference processing of the deep learning model, and combinations of operations that can be integrated, are registered. Operations that can be processed by an AI chip are operations for which a dedicated circuit required to execute the processing of that operation is implemented on the AI chip. Combinations of operations that can be integrated are combinations of primitive operations that can be integrated, such as the combination of an operation of a convolution layer and the operations of each of the surrounding layers associated with it, as described above.

The labeling unit 1022 individually determines whether the calculations of each layer of the deep learning model can be processed on an AI chip, and labels the layers of calculations to be processed on the AI chip so that they can be identified. Specifically, the labeling unit 1022 determines whether each calculation matches the calculations that can be processed by the AI chip described in the correspondence description unit 1040. The labeling unit 1022 attaches a label indicating that the calculations will be processed by the AI chip to layers of calculations that can be processed by the AI chip, and attaches a label indicating that the other layers will be processed by general-purpose hardware (e.g., CPU, GPU, etc.) for controlling the AI chip, etc. The labeling unit 1022 passes the labeled model in which each layer of the deep learning model (each node of the computation graph) has been labeled to the calculation integration unit 1024.

The computation integration unit 1024 combines into one layer each layer that corresponds to a combination of computations in a layer that is labeled to be processed by an AI chip and that matches a combination of computations described in the correspondence description unit 1040 in the labeled model. The computation integration unit 1024 thereby reconstructs a computation graph that represents the deep learning model, and generates and outputs a computation-integrated model.

FIG. 3 shows a flowchart illustrating the flow of the computation integration process executed by the computation integration device 1000 using the conventional method.

When a deep learning model represented by a computation graph is input to the computation integration device 1000, the deep learning model is scanned and the loop process of step S1000 is executed with each layer (each node of the computation graph) included in the deep learning model as the processing target. Specifically, in step S1002, the labeling unit 1022 determines whether or not the computation of the processing target layer can be processed on the AI chip based on the computations that can be processed by the AI chip and that are registered in the correspondence description unit 1040. If the computation can be processed, the process proceeds to step S1004, and if the computation cannot be processed, the process proceeds to step S1006.

In step S1004, the labeling unit 1022 attaches a label to the layer to be processed indicating that it will be processed by an AI chip. On the other hand, in step S1006, the labeling unit 1022 attaches a label to the layer to be processed indicating that it will be processed by general-purpose hardware.

Next, in step S1008, the labeling unit 1022 determines whether scanning of the deep learning model is complete, i.e., whether labeling processing is complete for all layers included in the deep learning model. If scanning is complete, the process proceeds to step S1010, and if scanning is not complete, the loop processing of step S1000 is repeated. In step S1010, the computation integration unit 1024 groups together sections in which consecutive layers are labeled to be processed by the AI chip, and reconstructs the computation graph.

Then, while scanning the deep learning model, the loop process of step S1012 is executed with each section as the processing target. Specifically, in step S1014, the operation integration unit 1024 performs pattern matching between the combination of operations of each layer in the processing target section and the combination of primitive operations that can be integrated and are registered in the correspondence description unit 1040. The operation integration unit 1024 determines whether or not each layer in the processing target section can be integrated depending on whether or not the combination of operations matches through pattern matching. If integration is possible, the process proceeds to step S1016, and if integration is not possible, the process proceeds to step S1018.

In step S1016, the layer combinations corresponding to the operation combinations that match through pattern matching are integrated by replacing them with one layer, and the computation graph is reconstructed. Next, in step S1018, the computation integration unit 1024 determines whether scanning of the deep learning model is complete, that is, whether the process of determining whether to integrate layers within all sections included in the deep learning model is complete. If scanning is complete, the process proceeds to step S1020, and if scanning is not complete, the loop process of step S1012 is repeated until there are no more sections that can be integrated. In step S1020, the computation integration unit 1024 outputs the computation graph finally reconstructed in step S1016 as a computationally integrated model, and the computation integration process ends.

As mentioned above, conventional methods can automate the integration of primitive operations in deep learning models, but do not take into account layer integration of deep learning models that include multiple convolutional layers. When multiple convolutional layers are included, the network structure differs for each deep learning model. Therefore, in order to perform layer integration of deep learning models that include multiple convolutional layers using conventional methods, it was necessary to perform a detailed analysis of the deep learning model that performs layer integration and respond individually to each model.

Specifically, in conventional methods, it is necessary to determine the combination of convolutional layers to be integrated based on the constraint of the maximum number of convolutional layers that can be integrated on an AI chip, in accordance with the specifications of the AI chip. In addition, in conventional methods, it is necessary to check whether the kernel size, input/output data size, etc. within the layer integration section for each combination of convolutional layers to be integrated are equal to or smaller than the cache capacity on the AI chip. As a result, conventional methods have the problem of being unable to flexibly respond to layer integration of various deep learning models.

In the following embodiments, we propose a method for automating the extraction of layer integration sections that satisfy constraints based on the AI chip specifications and the determination of the optimal layer integration section for any input deep learning model.

First Embodiment
4 is a block diagram showing the hardware configuration of the layer integration determination device 10 according to the first embodiment. As shown in FIG. 4, the layer integration determination device 10 has a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a storage 14, an input unit 15, a display unit 16, and a communication I/F (Interface) 17. Each component is connected to each other via a bus 19 so as to be able to communicate with each other.

The CPU 11 is a central processing unit that executes various programs and controls each part. That is, the CPU 11 reads out a program from the ROM 12 or storage 14, and executes the program using the RAM 13 as a working area. The CPU 11 controls each of the above components and performs various calculation processes according to the program stored in the ROM 12 or storage 14. In this embodiment, the layer integration determination program described below is stored in the ROM 12 or storage 14.

ROM 12 stores various programs and data. RAM 13 temporarily stores programs or data as a working area. Storage 14 is made up of storage devices such as HDD (Hard Disk Drive) and SSD (Solid State Drive), and stores various programs and data including the operating system.

The input unit 15 includes a pointing device such as a mouse and a keyboard, and is used to perform various input operations. The display unit 16 is, for example, a liquid crystal display, and displays various types of information. The display unit 16 may also function as the input unit 15 by employing a touch panel system.

The communication I/F 17 is an interface for communicating with other devices. For this communication, for example, a wired communication standard such as Ethernet (registered trademark) or FDDI, or a wireless communication standard such as 4G, 5G, or Wi-Fi (registered trademark) is used.

Next, the functional configuration of the layer integration determination device 10 according to the first embodiment will be described. FIG. 5 is a block diagram showing an example of the functional configuration of the layer integration determination device 10. As shown in FIG. 5, the layer integration determination device 10 includes, as its functional configuration, a generation unit 21, an extraction unit 26, a determination unit 28, and an optimization unit 30. In addition, a correspondence description unit 40, a pattern description unit 42, and a condition description unit 44 are provided in a predetermined storage area of the layer integration determination device 10. Each functional configuration is realized by the CPU 11 reading out a layer integration determination program stored in the ROM 12 or storage 14, expanding it in the RAM 13, and executing it.

The deep learning model to be processed by layer integration, which includes multiple convolution layers, is input to the layer integration determination device 10 in the form of a computation graph in which each layer, such as a convolution (Conv) layer and an activation function (Activation) layer, is represented by a node. Each node holds parameter information related to the layer corresponding to the node. The parameters are, for example, the size of the input and output feature maps, the kernel size and number of channels used in the convolution operation, the number of multiplications of the convolution matrix operation and activation function, the number of additions for bias addition, etc.

The correspondence description unit 40 is similar to the correspondence description unit 1040 of the conventional method described in FIG. 1. That is, the correspondence description unit 40 registers operations that can be processed by the AI chip and combinations of primitive operations that can be integrated.

The generation unit 21 integrates the calculations corresponding to each layer of the input deep learning model, including the convolutional layer that can be processed by the AI chip and the surrounding layers associated with the convolutional layer, to generate a calculation-integrated model. Specifically, the generation unit 21 includes a labeling unit 22 and a calculation integration unit 24.

The labeling unit 22 identifiably labels, among the layers of the input deep learning model, layers of operations that match operations that can be processed by the AI chip and that are pre-registered in the correspondence description unit 40. Other specific details of the labeling unit 22 are similar to those of the labeling unit 1022 in FIG. 1, so detailed explanations will be omitted. The labeling unit 22 passes the labeled model (computation graph) to the computation integration unit 24.

Based on the labeled model passed from the labeling unit 22, the computation integration unit 24 identifies a combination of operations that corresponds to a layer labeled as a layer of operations that can be processed by the AI chip. The computation integration unit 24 then groups, as processing blocks, combinations of operations that match combinations of primitive operations that can be integrated and are preregistered in the correspondence description unit 40, from among the identified combinations of operations, and reconstructs a computation graph. In this way, the computation integration unit 24 generates a computation-integrated model. Other specific details of the computation integration unit 24 are similar to those of the computation integration unit 1024 in FIG. 1, so a detailed description will be omitted. The computation integration unit 24 passes the generated computation-integrated model to the extraction unit 26.

The pattern description unit 42 stores patterns of combinations of convolutional layers that can be merged based on the layer structure of each convolutional layer. Specifically, the pattern description unit 42 stores a comprehensive set of partial graph patterns of deep learning models that satisfy constraints that take into account the "number of convolutional layers that can be merged" and the "connections between layers" determined by the AI chip specifications.

FIG. 6 shows an example of a subgraph pattern. FIG. 6(a) is a computation graph showing a combination of layers of integrable operations registered in the correspondence description unit 40 shown in FIG. 2. This is treated as one processing block (Conv Block), and a pattern combining multiple Conv Blocks, for example, as shown in FIG. 6(b), is registered in the pattern description unit 42. As shown in FIG. 6(b), the pattern may also include other operators such as an upsampling layer. As shown in FIG. 7(c), the pattern may also be a more complex pattern including a layer such as a residual layer (Res) that combines a skip connection and an addition layer.

The condition description section 44 registers the layer integration conditions based on the AI chip specifications. For example, the condition equation regarding the cache usage on the AI chip required to read the kernel of the convolution layer is specified based on the parameters of each node as in the following equation (1).

In equation (1), the layer integration section is from the l _s layer to the l _e layer, k _a is the kernel size of the a-th convolutional layer, iCH _a is the number of input channels of the a-th layer, and oCH _a is the number of output channels of the a-th layer. C _cache on the right side represents the cache capacity, which is a value determined according to the specifications of the AI chip.

The extraction unit 26 extracts, as a subgraph, a portion of the computationally integrated model (computation graph) passed from the computation integration unit 24 that matches a pattern registered in advance in the pattern description unit 42. Specifically, the extraction unit 26 scans the computation graph indicating the computationally integrated model, matches it with the pattern registered in the pattern description unit 42, and extracts the section where the pattern matches as a subgraph that is a candidate for a layer integration section. The extraction unit 26 passes the extracted subgraph to the determination unit 28.

The determination unit 28 determines, from among the subgraphs passed from the extraction unit 26, a section corresponding to a subgraph that satisfies the conditions registered in advance in the condition description unit 44 as a layer integration section. Specifically, the determination unit 28 extracts parameters of each node in the subgraph, and reads out the conditional expression registered in the condition description unit 44. The determination unit 28 then uses the extracted parameters to calculate the read out conditional expression, and determines whether or not the subgraph satisfies the corresponding conditional expression. The determination unit 28 generates a layer-integrated model candidate by integrating each layer in the layer integration section that satisfies the condition and reconstructing the computationally integrated model. The determination unit 28 passes the generated layer-integrated model candidate to the optimization unit 30.

The optimization unit 30 selects and outputs the optimal candidate as the layer-integrated model based on an optimization index from among multiple layer-integrated model candidates with different layer-integrated sections generated by executing the processes of the extraction unit 26 and the determination unit 28 multiple times. The optimization index is arbitrary, but it is preferable to use an index that uses at least one of a numerical value representing the processing time of each layer-integrated section, specifically, the amount of calculation of the deep learning model and the amount of data exchanged between the external memory and the AI chip. For example, the optimization unit 30 calculates the total amount of product-sum calculations of each layer-integrated section, the variance of the read/write time of the convolution kernel and the input/output feature, etc. as the optimization index. The optimization unit 30 selects and outputs the layer-integrated model candidate with the smallest variance as the final layer-integrated model. As a result, a candidate with the most uniform processing time for each layer-integrated section included in the layer-integrated model candidate is selected, and when assembling pipeline processing, processing delays due to waiting for the completion of the previous stage can be suppressed, enabling efficient execution of inference processing.

Next, the operation of the layer merging determination device 10 according to the first embodiment will be described. FIG. 8 is a flowchart showing the flow of the layer merging determination process by the layer merging determination device 10. The layer merging determination process is performed by the CPU 11 reading out a layer merging determination program from the ROM 12 or storage 14, expanding it into the RAM 13, and executing it. The layer merging determination process is an example of the layer merging determination method of the present disclosure.

In step S10, the CPU 11, functioning as the generation unit 21, executes a computation integration process. The computation integration process is similar to the computation integration process of the conventional method shown in FIG. 3. Next, as the loop process in step S12, a specified number of repetitions are executed. The specified number of repetitions is specified externally as a hyperparameter.

Specifically, in step S14, the CPU 11, functioning as the extraction unit 26, changes the order of patterns to be matched for the patterns registered in the pattern description unit 42. The order of the patterns can be set arbitrarily; for example, the order may be changed randomly, or the order may be changed regularly by grouping the patterns by the number of convolutional layers in the patterns. This is a process for generating a different layer-integrated model candidate for each iteration in the process of step S16 described below.

Next, in step S16, the CPU 11 executes a layer-integrated model candidate generation process. Next, in step S18, the CPU 11, functioning as the optimization unit 30, determines whether the repetition process has reached a designated number of times. If the repetition process has reached the designated number of times, the process proceeds to step S20, and if the repetition process has not reached the designated number of times, the process repeats the loop process of step S12.

Now, with reference to FIG. 9, we will explain the layer-integrated model candidate generation process executed in step S16.

The CPU 11 sets each of the patterns registered in the pattern description unit 42 as a pattern to be processed in the order changed in step S14, and executes the loop process of step S160. Specifically, the CPU 11 scans the computationally integrated model and executes the loop process of step S162.

In step S164, the CPU 11, functioning as the extraction unit 26, determines whether or not there are any unintegrated sections remaining, i.e., sections other than the layer-integrated sections extracted based on patterns other than the pattern to be processed. If there are any unintegrated sections remaining, the process proceeds to step S166; if there are no unintegrated sections remaining, the process proceeds to step S176.

In step S166, the CPU 11, functioning as the extraction unit 26, searches for a subgraph that matches the pattern to be processed in the unintegrated section of the computationally integrated model (computation graph). The CPU 11, functioning as the extraction unit 26, determines whether or not a subgraph that matches the pattern to be processed exists. If a subgraph exists, the process proceeds to step S168; if not, the loop process of step S162 ends.

In step S168, the CPU 11, functioning as the extraction unit 26, extracts a subgraph that matches the pattern to be processed. Then, the CPU 11, functioning as the judgment unit 28, extracts parameters for each node in the subgraph and reads out a condition expression registered in the condition description unit 44. Furthermore, the CPU 11, functioning as the judgment unit 28, calculates the read out condition expression using the extracted parameters and judges whether or not the subgraph satisfies the corresponding condition expression. If the subgraph satisfies the condition, the process proceeds to step S170; if the subgraph does not satisfy the condition, the loop process of step S162 ends.

In step S170, the CPU 11, functioning as the extraction unit 26, determines whether scanning of the entire computationally integrated model has been completed. If scanning has not been completed, the loop process of step S162 is repeated, and if scanning has been completed, the process proceeds to step S174. In step S174, the CPU 11, functioning as the extraction unit 26, determines whether matching processing with the computationally integrated model has been completed for all patterns registered in the pattern description unit 42. If there are unprocessed patterns, the loop process of step S160 is repeated for the unprocessed patterns, and if all patterns have been processed, the process proceeds to step S176.

In step S176, there are no unintegrated sections remaining in the computationally integrated model, or matching processing has been completed for all patterns. The CPU 11, as the determination unit 28, generates a layer-integrated model candidate by integrating each layer in the layer integration section extracted in step S170 above to reconstruct the computationally integrated model. Note that layers that are labeled to be processed by the AI chip and are not included in any layer integration section are treated as being processed as a single layer, and are output together with each layer integration section as part of the layer-integrated model candidate. Then, the layer-integrated model candidate generation process ends, and the process returns to the layer integration determination process (FIG. 8). The loop process (step S12) including the layer-integrated model candidate generation process in step S16 is repeated a specified number of times, generating a specified number of layer-integrated model candidates.

Next, in step S20, the CPU 11, functioning as the optimization unit 30, executes the loop process of step S20 for each generated layer-integrated model candidate. Specifically, in step S22, the CPU 11, functioning as the optimization unit 30, calculates, as an optimization index, the variance of an index representing the processing time of each layer-integrated section included in the layer-integrated model candidate to be processed.

Next, in step S24, the CPU 11, functioning as the optimization unit 30, determines whether the process of calculating the optimization index for all layer-integrated model candidates has been completed. If there are any unprocessed layer-integrated model candidates, the loop process of step S20 is repeated, and if all have been completed, the process proceeds to step S26.

In step S26, the CPU 11, as the optimization unit 30, selects the optimal layer-merged model from the layer-merged model candidates based on the optimization index. For example, the CPU 11, as the optimization unit 30, selects and outputs, as the final layer-merged model, the layer-merged model candidate that has the smallest variance of the index representing the processing time for each layer-merged section calculated as the optimization index. Then, the layer-merging determination process ends.

As described above, the layer integration determination device according to the first embodiment integrates the operations corresponding to each of the convolution layers and the surrounding layers associated with the convolution layers that can be processed by the dedicated hardware that executes the inference processing of the deep learning model, among the operations corresponding to each layer of the deep learning model including multiple convolution layers input in the form of a computation graph in which each layer is represented by a node, to generate a computation-integrated model, extracts a subgraph from the computation-integrated model that matches a pattern preregistered in the pattern description section as a combination of convolution layers that can be layer-integrated based on the layer configuration of each of the convolution layers, determines a section corresponding to a subgraph that satisfies a condition preregistered in the condition description section as a condition for layer integration based on the specifications of the dedicated hardware, and selects and outputs the optimal layer-integrated model based on a predetermined index from among multiple layer-integrated model candidates with different layer-integration sections. This makes it possible to flexibly respond to various deep learning models while shortening the time required to determine layer integration of a deep learning model including multiple convolution layers using only the deep learning model represented as a computation graph as input.

Second Embodiment
In the second embodiment, a deep learning model is scanned in advance to generate conditions to be used in optimization as additional conditional expressions, and the conditions are passed to the judgment unit, so that the processing of the extraction unit and the judgment unit can be completed in one go. Note that in the layer integration judgment device according to the second embodiment, the same components as those in the layer integration judgment device 10 according to the first embodiment are denoted by the same reference numerals and detailed descriptions thereof are omitted. In addition, the hardware configuration of the layer integration judgment device according to the second embodiment is the same as the hardware configuration of the layer integration judgment device 10 according to the first embodiment shown in FIG. 4, and therefore the description thereof is omitted.

The functional configuration of the layer integration determination device 210 according to the second embodiment will be described. FIG. 10 is a block diagram showing an example of the functional configuration of the layer integration determination device 210. As shown in FIG. 10, the layer integration determination device 210 includes, as its functional configuration, a generation unit 21 including a labeling unit 22 and a calculation integration unit 24, an extraction unit 26, a determination unit 228, and an addition unit 232. In addition, a correspondence description unit 40, a pattern description unit 42, and a condition description unit 44 are provided in a predetermined storage area of the layer integration determination device 210. Each functional configuration is realized by the CPU 11 reading out a layer integration determination program stored in the ROM 12 or storage 14, expanding it in the RAM 13, and executing it.

The adding unit 232 adds an optimization condition based on a predetermined number of layer integration sections to the conditions used by the determining unit 228. For example, the adding unit 232 adds, as a condition, a range of the amount of calculation for each layer integration section that is included in the layer-integrated model and that uniforms the processing time for each layer integration section, using at least one of the amount of calculation for the deep learning model and the amount of data exchanged between the external memory and the AI chip.

More specifically, the adding unit 232 acquires the deep learning model (computation graph) input to the layer integration determination device 210, and scans the computation graph to estimate the computational volume of the entire deep learning model. The adding unit 232 estimates the computation volume from parameters of the deep learning model, including the sizes of the input and output feature maps, the kernel size used in the convolution computation, the number of channels, the convolution matrix computation and the number of multiplications of the activation function, the number of additions for bias addition, etc. These parameters are held by each node of the computation graph, as described above.

The adding unit 232 sets an upper limit and a lower limit of the amount of calculation per layer-integrated section from the estimated amount of calculation for the entire learning model. Specifically, the adding unit 232 calculates the amount of calculation per layer-integrated section by dividing the amount of calculation for the entire deep learning model by the expected number of layer-integrated sections. The expected number of layer-integrated sections may be given manually as a hyperparameter, or may be estimated mechanically. When estimating mechanically, for example, a layer-integrated model is generated in a state where no additional condition equation is given from the adding unit 232, and the number of layer-integrated sections included in the generated layer-integrated model is adopted. Then, the adding unit 232 sets an upper limit and a lower limit based on the amount of calculation per layer-integrated section.

The adding unit 232 passes the formula used to calculate the amount of calculation and an additional condition formula indicating the range of the amount of calculation per one-layer integration section to the determining unit 228. The additional condition formula is written, for example, as in the following formula (2).

In formula (2), the layer integration section is from the l _s layer to the l _e layer, m _a is the number of multiplications of the a-th layer, which is a convolution layer or an activation function layer (type = conv, activation), C _min is the lower limit of the amount of calculation, and C _max is the upper limit of the amount of calculation. Formula (2) is a conditional judgment formula for judging whether the sum of the number of multiplications of the convolution layer and the activation function layer is within the range from the upper limit to the lower limit.

The conditions to be added are not limited to conditions related to the amount of calculations of the deep learning model, but may be conditions using the amount of data exchanged with external memory, or combinations of these. In addition, in formula (2), an example of an additional condition formula in which an upper and lower limits are set is shown, but an additional condition formula in which only an upper limit is set may also be used. For example, an upper limit on the amount of calculations per one-layer integration section may be set based on the computing power of the AI chip and the processing time for one stage of pipeline processing.

In addition, if a conditional expression using the same calculation method as the created additional conditional expression is registered in the condition description section 44, the adding section 232 may overwrite the conditional expression registered in the condition description section 44 with the additional conditional expression, thereby passing the additional conditional expression to the determination section 228.

The determination unit 228, like the determination unit 28 in the first embodiment, determines whether the subgraph extracted by the extraction unit 26 satisfies the conditions. However, the determination unit 228 in the second embodiment determines whether the optimization conditions based on the number of layer integration sections transferred from the addition unit 232 are satisfied, along with the layer integration conditions based on the AI chip specifications registered in the condition description unit 44. The determination unit 228 generates and outputs a layer-integrated model in which the layers in the layer integration sections that satisfy the conditions are integrated to reconstruct the computationally integrated model.

Next, the operation of the layer merging determination device 210 according to the second embodiment will be described. FIG. 11 is a flowchart showing the flow of the layer merging determination process by the layer merging determination device 210. The layer merging determination process is performed by the CPU 11 reading out a layer merging determination program from the ROM 12 or storage 14, expanding it into the RAM 13, and executing it.

In step S210, the CPU 11, as the adding unit 232, determines the expected number of layer integration sections, for example, by acquiring manually assigned hyperparameters or by mechanically estimating. Next, in step S212, the CPU 11, as the adding unit 232, acquires the deep learning model (computation graph) input to the layer integration determination device 210, scans the computation graph, and estimates the amount of calculation for the entire deep learning model from the parameters of the deep learning model.

Next, in step S214, the CPU 11, as the adding unit 232, calculates the amount of calculation per layer integration section by dividing the amount of calculation for the entire deep learning model by the expected number of layer integration sections. Then, the CPU 11, as the adding unit 232, sets a range of the amount of calculation per layer integration section (e.g., upper and lower limits) based on the calculated amount of calculation per layer integration section, and creates an additional condition equation. Next, in step S216, the CPU 11, as the adding unit 232, passes the equation used to calculate the amount of calculation and the created additional condition equation to the determining unit 228.

Next, in step S10, the CPU 11, functioning as the generation unit 21, executes a calculation integration process. The calculation integration process is similar to the calculation integration process of the conventional method shown in FIG. 3. Next, in step S218, the CPU 11, functioning as the extraction unit 26 and the determination unit 228, executes a layer-integrated model generation process. The layer-integrated model generation process is similar to the layer-integrated model candidate generation process shown in FIG. 9. However, when determining in step S168 whether the subgraph satisfies the condition, the CPU 11, functioning as the determination unit 228, determines whether the subgraph satisfies the condition transferred from the addition unit 232, together with the condition registered in the condition description unit 44. Furthermore, the model generated in step S176 is not the layer-integrated model candidate, but the final layer-integrated model.

Next, in step S220, the determination unit 228 outputs the generated layer-integrated model, and the layer-integration determination process ends.

As described above, the layer integration determination device according to the second embodiment integrates layers within a layer integration section that satisfies optimization conditions based on a predetermined number of layer integration sections, such as the range of the amount of calculations per layer integration section, along with layer integration conditions based on the specifications of the AI chip, which are registered in the condition description section, to generate a layer-integrated model. This eliminates the need for the repeated processing of the extraction section and determination section in the first embodiment, and makes it possible to reduce the processing time of the layer integration determination process compared to the layer integration determination device according to the first embodiment.

In addition, the layer integration judgment process executed by the CPU by reading the software (program) in each of the above embodiments may be executed by various processors other than the CPU. Examples of processors in this case include PLDs (Programmable Logic Devices) such as FPGAs (Field-Programmable Gate Arrays) whose circuit configuration can be changed after manufacture, and dedicated electric circuits such as ASICs (Application Specific Integrated Circuits), which are processors having a circuit configuration designed exclusively to execute specific processes. In addition, the layer integration judgment process may be executed by one of these various processors, or may be executed by a combination of two or more processors of the same or different types (for example, multiple FPGAs, and a combination of a CPU and an FPGA, etc.). In addition, the hardware structure of these various processors is, more specifically, an electric circuit that combines circuit elements such as semiconductor elements.

In addition, in each of the above embodiments, the layer integration judgment program is described as being pre-stored (installed) in storage, but this is not limiting. The program may be provided in a form stored in a non-transitory storage medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), or a USB (Universal Serial Bus) memory. The program may also be downloaded from an external device via a network.

The following notes are further provided with respect to each of the above embodiments.

(Additional Note 1)
Memory,
at least one processor coupled to the memory;
Including,
The processor,
Among the operations corresponding to each layer of a deep learning model including a plurality of convolution layers, which are input in the form of a computation graph in which each layer is represented by a node, operations corresponding to each of the convolution layers that can be processed by dedicated hardware that executes the inference processing of the deep learning model and the surrounding layers associated with the convolution layers are integrated to generate a computation-integrated model;
extracting a subgraph that matches a pattern registered in advance in a pattern description section as a combination of convolutional layers that can be integrated based on the layer configuration of each of the convolutional layers from the computationally integrated model;
a layer merging determination device configured to determine, as a layer merging interval, an interval corresponding to the subgraph that satisfies a condition registered in advance in a condition description section as a condition for layer merging based on a specification of the dedicated hardware.

(Additional Note 2)
A non-transitory storage medium storing a program executable by a computer to execute a layer integration determination process,
The layer integration determination process includes:
Among the operations corresponding to each layer of a deep learning model including a plurality of convolution layers, which are input in the form of a computation graph in which each layer is represented by a node, operations corresponding to each of the convolution layers that can be processed by dedicated hardware that executes the inference processing of the deep learning model and the surrounding layers associated with the convolution layers are integrated to generate a computation-integrated model;
extracting a subgraph that matches a pattern registered in advance in a pattern description section as a combination of convolutional layers that can be integrated based on the layer configuration of each of the convolutional layers from the computationally integrated model;
A non-transitory storage medium that determines, as a layer integration interval, an interval corresponding to the subgraph that satisfies a condition registered in advance in a condition description section as a condition for layer integration based on a specification of the dedicated hardware.

10, 210 layer integration determination device 11 CPU
12 ROM
13 RAM
14 Storage 15 Input unit 16 Display unit 17 Communication I/F
19 Bus 21 Generation unit 22 Labeling unit 24 Operation integration unit 26 Extraction unit 28, 228 Determination unit 232 Addition unit 30 Optimization unit 40 Correspondence relationship description unit 42 Pattern description unit 44 Condition description unit

Claims (8)

A generation unit that integrates operations corresponding to each of the convolution layers and the surrounding layers associated with the convolution layers that can be processed by dedicated hardware that executes the inference processing of the deep learning model, among operations corresponding to each layer of the deep learning model including multiple convolution layers, input in the form of a computation graph in which each layer is represented by a node, to generate a computation-integrated model;
an extraction unit that extracts a subgraph that matches a pattern registered in advance in a pattern description unit as a combination of convolutional layers that can be integrated based on the layer configuration of each of the convolutional layers from the computationally integrated model;
a determination unit that determines, as a layer integration section, an interval corresponding to the subgraph that satisfies a condition registered in advance in a condition description unit as a layer integration condition based on a specification of the dedicated hardware;
A layer integration determination device comprising: The layer integration determination device according to claim 1, further comprising an optimization unit that selects an optimal layer integration model based on a predetermined index from among a plurality of layer integration model candidates each having a different layer integration section, which are generated by executing the processes of the extraction unit and the determination unit multiple times. The layer integration determination device according to claim 2, wherein the optimization unit uses at least one of the amount of calculation of the deep learning model and the amount of data exchanged between an external memory and the dedicated hardware as the predetermined index, and selects the layer integration model in which the processing time of each layer integration section included in the layer integration model is most uniform as the optimal layer integration model. The layer integration determination device according to claim 1, further comprising an addition unit that adds an optimization condition based on a predetermined number of layer integration sections to the conditions used by the determination unit. The layer integration determination device according to claim 4, wherein the adding unit uses at least one of the amount of calculation of the deep learning model and the amount of data exchanged between an external memory and the dedicated hardware to set a condition for adding a range of the amount of calculation of the layer integration section such that the processing time of each layer integration section included in the computationally integrated model is uniform. The generation unit is
A labeling unit that identifiably labels operations of the input deep learning model that match operations that are pre-registered in a correspondence description unit and can be processed by the dedicated hardware;
a computation integration unit that generates the computation-integrated model by integrating, as a processing block, a combination of operations that is processable by the dedicated hardware and matches a combination of primitive operations that can be integrated and is registered in advance in the correspondence description unit, and reconstructing a computation graph;
The layer integration determination device according to any one of claims 1 to 5, comprising: A layer integration determination method executed by a layer integration determination device including a generation unit, an extraction unit, and a determination unit,
The generation unit integrates operations corresponding to each of the convolution layers that can be processed by dedicated hardware that executes the inference processing of the deep learning model and the surrounding layers associated with the convolution layers, among the operations corresponding to each layer of the deep learning model including a plurality of convolution layers, input in the form of a computation graph in which each layer is represented by a node, to generate a computation-integrated model;
the extraction unit extracts, from the computationally integrated model, a subgraph that matches a pattern registered in advance in a pattern description unit as a combination of convolutional layers that can be integrated based on a layer configuration of each of the convolutional layers;
the determination unit determines, as a layer-merging interval, an interval corresponding to the subgraph that satisfies a condition registered in advance in a condition description unit as a condition for layer-merging based on a specification of the dedicated hardware. A layer integration determination program for causing a computer to function as each part of a layer integration determination device according to any one of claims 1 to 5.