WO2025108230A1 - Multi-model reasoning method and system, network processing unit, extended reality display chip and apparatus, and image processing method - Google Patents

Abstract

The present invention provides a multi-model reasoning method and system, a network processing unit, an extended reality display chip and apparatus, and an image processing method. The multi-model reasoning method comprises the following steps: acquiring reasoning data of a plurality of models; determining at least one switching node from a plurality of processing nodes of each model on the basis of a target having a time delay priority and/or a storage space priority, and determining a storage space corresponding to each model; writing the reasoning data of each model into a corresponding storage space of a network processing unit; and on the basis of the switching node, using full computing resources of the network processing unit to sequentially perform time-division multiplexing reasoning of the models so as to implement time delay optimization reasoning and/or storage space optimization reasoning of the plurality of models. By using the described configurations, the multi-model reasoning method can flexibly switch time-division multiplexing among a plurality of neural network models according to actual requirements, thereby implementing time delay optimization reasoning and/or storage space optimization reasoning of the plurality of models.

Description

Multi-model reasoning method and system, network processing unit, extended reality display chip and device, image processing method

This application claims priority to a patent application with a filing date of November 24, 2023, Chinese application number 202311587154.0, entitled “A multi-model reasoning method, system and storage medium”, and a patent application with a filing date of November 24, 2023, Chinese application number 202311582338.8, entitled “Network processing unit, extended reality display chip and device, image processing method”. The entire contents of these two applications are incorporated herein by reference.

Technical Field

The present invention relates to neural network reasoning technology and extended reality display technology, and in particular to a multi-model reasoning method, a multi-model reasoning system, a network processing unit, an extended reality display chip, an extended reality display device, a parallel processing method for image data, and a computer-readable storage medium.

Background Art

A network processing unit (NPU) is a processor that uses circuits to simulate human neurons and synaptic structures. It is widely used in the field of artificial intelligence (AI) technology to simulate humans' processing of various complex information such as images, sounds, and languages.

With the continuous development of AI technology, the data dimensions, data volume and model complexity involved have shown an explosive growth trend, which has put forward a huge test on the computing power of NPU. The existing technology usually uses multiple independently packaged NPU chips or multi-core NPU chips to perform multi-model reasoning of multiple neural network models. However, the independently packaged NPU chips have fixed computing and storage resources. On the one hand, there are problems such as high chip area, power consumption and cost, which cannot be integrated on a large scale. On the other hand, there is insufficient flexibility and the problem of being unable to adapt to the increase or decrease in the number of models to allocate computing and storage resources. Therefore, it cannot adapt to the multi-scenario and multi-functional application requirements of extended reality (XR) display devices. Relatively speaking, although the multiple NPU cores (Core) of the multi-core NPU chip can share part of the chip's storage resources, the number of its NPU cores and the computing resources of each NPU core are independently fixed, so there are also the above-mentioned problems of high area, power consumption, cost and insufficient flexibility, which cannot adapt to the multi-scenario and multi-functional application requirements of XR display devices.

In order to overcome the above-mentioned defects of the prior art, the field is in urgent need of a multi-model reasoning technology, which can be used to flexibly switch time-division multiplexing between multiple neural network models according to actual needs, thereby realizing latency optimized reasoning and/or storage space optimized reasoning of multiple models.

In addition, Extended Reality (XR) display technology is an immersive display technology that uses modern high-tech means with computers as the core to create a digital environment that combines real and virtual things, providing users with seamless transitions between the virtual world and the real world. It mainly includes Virtual Reality (VR) display, Augmented Reality (AR) display, Mixed Reality (MR) display and many other implementation methods.

In the existing technologies in the fields of mobile phones, cameras, and surveillance, a separate network processing unit (NPU) is usually configured for each image signal processor (ISP) to perform network inference. As a result, the overall computing power of the system cannot be flexibly and fully utilized, resulting in a waste of the overall computing power of the system and an increase in network inference latency.

In addition, after obtaining the entire image data with a resolution of H×W from the camera module, the existing technology usually needs to tile the entire image through the image signal processor (ISP), divide it into multiple N×M (N<H, M<W) tile images according to the data processing capability of the network processing unit (NPU), and store them in the ISP buffer. The network processing unit (NPU) then reads the image data of each tile image from the ISP buffer one by one for network reasoning. On the one hand, this will introduce additional operations and intermediate data such as slicing, zero-filling and windowing, and trimming of overlapping images to increase the requirements for the overall computing power of the system. On the other hand, it also requires the system to have a larger data cache space to greatly increase the area, power consumption and cost of the image signal processor (ISP), thereby severely limiting the development and application of the existing network processing unit (NPU) to parallel process multiple image data.

In order to overcome the above-mentioned defects of the prior art, there is an urgent need in the art for a parallel processing technology for multi-channel image data, which improves the accuracy of image processing by eliminating the need for tile processing of the entire image and reduces the requirements for the overall system computing power and data cache space. Thus, under the same hardware processing technology and cost conditions, the ability of a single network processing unit (NPU) to process multi-channel image data in parallel is improved, and the overall computing power of the system is flexibly and fully utilized to reduce network inference latency, thereby meeting the needs of parallel processing of binocular image data in XR devices.

Summary of the invention

A brief summary of one or more aspects is given below to provide a basic understanding of these aspects. This summary is not an exhaustive overview of all conceived aspects, and is neither intended to identify the key or decisive elements of all aspects nor to define the scope of any or all aspects. Its only purpose is to give some concepts of one or more aspects in a simplified form as a prelude to a more detailed description that will be given later.

In order to overcome the above-mentioned defects of the prior art, the present invention provides a multi-model reasoning method, a multi-model reasoning system, and a computer-readable storage medium, which can determine at least one switching node from multiple processing nodes of each model according to the goals of latency priority and/or storage space priority, and determine the storage space corresponding to each model, and then use the full computing resources of the network processing unit to perform time-division multiplexing reasoning of each model in turn according to the switching node, so as to achieve latency optimized reasoning and/or storage space optimized reasoning of multiple models.

In addition, the present invention also provides a network processing unit, an extended reality display chip, an extended reality display device, a parallel processing method for image data, and a computer-readable storage medium, which can acquire multiple channels of image data line by line in parallel, write each channel of image data into a buffer at a preset speed, and then use the full computing resources of the network processing unit through the i model to read and process the written i-th channel of image data from the buffer. By adopting these configurations, the present invention can improve the accuracy of image processing by eliminating the need to tile the entire image, and reduce the requirements for the overall computing power of the system and data cache space, thereby improving the ability of a single network processing unit (NPU) to process multiple channels of image data in parallel under the same hardware processing technology and cost conditions, and thereby flexibly and fully utilizing the overall computing power of the system to reduce network inference delay, so as to meet the needs of parallel processing of binocular image data in XR devices.

BRIEF DESCRIPTION OF THE DRAWINGS

The above features and advantages of the present invention can be better understood after reading the detailed description of the embodiments of the present disclosure in conjunction with the following drawings. In the drawings, the components are not necessarily drawn to scale, and components with similar related properties or features may have the same or similar reference numerals.

FIG. 1 shows an architecture diagram of a network processing unit provided according to some embodiments of the present invention.

FIG2 shows a flowchart of a multi-model reasoning method provided according to some embodiments of the present invention.

FIG3 shows a flowchart of determining switching nodes and storage space distribution for latency optimization reasoning according to some embodiments of the present invention.

FIG. 4 shows a schematic diagram of latency optimization reasoning provided according to some embodiments of the present invention.

FIG. 5 shows a flowchart of determining a switching node and storage space distribution for storage space optimization reasoning according to some embodiments of the present invention.

FIG6 shows a schematic diagram of storage space optimization reasoning provided according to some embodiments of the present invention.

FIG. 7 shows an architecture diagram of an extended reality display chip provided according to some embodiments of the present invention.

FIG. 8 shows a flowchart of parallel processing of multiple channels of image data according to some embodiments of the present invention.

FIG. 9 is a schematic diagram showing a line-by-line acquisition of image data according to some embodiments of the present invention.

FIG. 10 shows a structural diagram of an XR display chip provided according to some embodiments of the present invention.

FIG. 11 is a schematic diagram showing a noise reduction process according to some embodiments of the present invention.

DETAILED DESCRIPTION

The following specific embodiments illustrate the implementation of the present invention, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Although the description of the present invention will be introduced in conjunction with the preferred embodiment, this does not mean that the features of this invention are limited to this implementation. On the contrary, the purpose of introducing the invention in conjunction with the implementation is to cover other options or modifications that may be extended based on the claims of the present invention. In order to provide a deep understanding of the present invention, the following description will include many specific details. The present invention can also be implemented without using these details. In addition, in order to avoid confusion or blurring the focus of the present invention, some specific details will be omitted in the description.

In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

In addition, the terms "upper", "lower", "left", "right", "top", "bottom", "horizontal" and "vertical" used in the following description should be understood as the directions shown in the paragraph and the related drawings. Such relative terms are only used for the convenience of description and do not mean that the device described therein must be manufactured or operated in a specific direction, and therefore should not be understood as limiting the present invention.

It is understood that although the terms "first", "second", "third", etc. may be used herein to describe various components, regions, layers and/or parts, these components, regions, layers and/or parts should not be limited by these terms, and these terms are only used to distinguish different components, regions, layers and/or parts. Therefore, the first component, region, layer and/or part discussed below may be referred to as a second component, region, layer and/or part without departing from some embodiments of the present invention.

As mentioned above, the prior art usually uses multiple independently packaged Network Processing Unit (NPU) chips or multi-core NPU chips to perform multi-model reasoning for multiple neural network models. However, independently packaged NPU chips have fixed computing and storage resources. On the one hand, there are problems such as high chip area, power consumption, and cost, which cannot be integrated on a large scale. On the other hand, there is insufficient flexibility and the problem of being unable to adapt to the increase or decrease in the number of models to allocate computing and storage resources. Therefore, it cannot adapt to the multi-scenario and multi-functional application requirements of extended reality (XR) display devices. Relatively speaking, although the multiple NPU cores (Cores) of a multi-core NPU chip can share part of the chip's storage resources, the number of its NPU cores and the computing resources of each NPU core are independently fixed, so there are also the above-mentioned problems of area, power consumption, high cost and insufficient flexibility, which cannot adapt to the multi-scenario and multi-functional application requirements of XR display devices.

In order to overcome the above-mentioned defects existing in the prior art, the present invention provides a multi-model reasoning method, a multi-model reasoning system and a computer-readable storage medium, which can determine at least one switching node from multiple processing nodes of each model according to the goals of latency priority and/or storage space priority, and determine the storage space corresponding to each model, and then use the full computing resources of the network processing unit to perform time-division multiplexing reasoning of each model in turn according to the switching node, so as to achieve latency optimized reasoning and/or storage space optimized reasoning of multiple models.

In some non-limiting embodiments, the multi-model reasoning method provided in the first aspect of the present invention can be implemented via the multi-model reasoning system provided in the second aspect of the present invention. Specifically, the multi-model reasoning system can be configured in a network processing unit (NPU) chip, which is configured with a memory and a processor. The memory includes but is not limited to the computer-readable storage medium provided in the third aspect of the present invention, on which computer instructions are stored. The processor is connected to the memory and is configured to execute the computer instructions stored on the memory to implement the multi-model reasoning method provided in the first aspect of the present invention.

The following will describe the working principles of the above-mentioned NPU chip and multi-model reasoning system in conjunction with some embodiments of the multi-model reasoning method. Those skilled in the art will understand that the embodiments of these multi-model reasoning methods are only some non-restrictive implementation methods provided by the present invention, which are intended to clearly demonstrate the main concept of the present invention and provide some specific solutions that are convenient for the public to implement, rather than to limit all functions or all working modes of the NPU chip and multi-model reasoning system. Similarly, the NPU chip and multi-model reasoning system are only a non-restrictive implementation method provided by the present invention, and do not constitute any limitation on the execution subject and execution order of each step in these multi-model reasoning methods.

Please refer to Figure 1 and Figure 2. Figure 1 shows an architecture diagram of a network processing unit provided according to some embodiments of the present invention. Figure 2 shows a flow chart of a multi-model reasoning method provided according to some embodiments of the present invention.

As shown in FIG1 , in some embodiments of the present invention, a multiplication and accumulation (MAC) array computing unit 11, a vector processing unit (VPU) 12, a time division multiplexing (TDM) unit 13, and a static random access memory (SRAM) for storing neural network reasoning data are integrated in the NPU chip 10. Here, the SRAM can meet the needs of neural network reasoning and is specifically divided into a parameter storage space SRAM_W 141 for storing neural network model parameters and an input and intermediate data storage space SRAM_F 142 for storing input data and intermediate data of neural network reasoning.

As shown in FIG2 , in the process of multi-model reasoning, the NPU chip 10 can first obtain the reasoning data of multiple models M1 to Mm from a signal source such as an image signal processor (ISP) of the XR display device. Here, the reasoning data can be image data of multiple complete images synchronously obtained by multiple image signal processors, or multiple sliced image data obtained by one image signal processor after slicing (Tile) a complete image according to the cache space size of SRAM_F. The multiple sliced images correspond to a part of the complete image respectively, and the amount of data is not greater than the cache space of SRAM_F. The multiple models M1 to Mm can be the same neural network model with the same structure, parameters and functions, or different neural network models with different structures, parameters and functions.

Afterwards, the TDM unit 13 of the NPU chip 10 can search and determine at least one switching node from the multiple processing nodes of each model M1~Mm according to the goals of delay priority and/or storage space priority, and determine the static storage space corresponding to each model M1~Mm, and then the NPU chip 10 writes the reasoning data of each model M1~Mm into the input and intermediate data storage space SRAM_F in the corresponding static storage space for each model M1~Mm to read. Here, the neural network model can include multiple inherent processing nodes respectively, and complete a cycle of neural network reasoning by sequentially executing the relevant operations of each processing node. The switching node is obtained by screening from each processing node according to the goals of delay priority and/or storage space priority, and is correspondingly distributed according to the delay requirements of each neural network model and/or the total cache data volume of the intermediate data generated by each neural network model to indicate the opening and closing time of the processing window of each neural network model.

For details, please refer to Figures 3 and 4. Figure 3 shows a flow chart of determining a switching node and storage space distribution for latency optimization reasoning according to some embodiments of the present invention. Figure 4 shows a schematic diagram of latency optimization reasoning according to some embodiments of the present invention.

In the embodiment shown in FIG3 , the process of determining the storage space distribution of the latency optimization reasoning can be implemented in an offline manner based on a large number of pre-prepared reasoning data samples. Specifically, the present invention can first assume that there are m neural network models M1 to Mm that need to be processed in parallel, and respectively determine the reasoning cycle P1 to Pm (in ms) for each model M1 to Mm to complete a reasoning. Afterwards, the TDM unit 13 in the NPU chip 10 can statically divide the full storage space of the NPU chip 10 into parameter storage spaces W1 to Wm for storing parameters of each model, and input and intermediate data storage spaces F1 to Fm for storing input data and intermediate data of each model according to the number of models m that need to be processed in parallel, so as to determine the first storage space distribution for storing the reasoning data of each model. Here, the parameter storage space W1 to Wm of each model M1 to Mm can be determined by compiling each model M1 to Mm respectively, and setting the target performance FPS (Frames per second) and total bandwidth, and its total size should not exceed SRAM_W 141, that is, SUM (W1: Wm) ≤ SRAM_W. Correspondingly, the input and intermediate data storage space F1~Fm of each model M1~Mm can also be determined by compiling each model M1~Mm separately and setting the target performance FPS (Frames per second) and total bandwidth, and the total size should not exceed SRAM_F 142, that is, SUM(F1:Fm)≤SRAM_F.

Afterwards, the TDM unit 13 can determine the total cycle N of multi-model reasoning according to the reasoning cycles P1-Pm of each model M1-Mm, and thereby determine the number of inferences of each model M1-Mm within the total cycle N. Specifically, the total cycle N can be determined by the least common multiple (LCM) of the reasoning cycles P1-Pm of each model M1-Mm, that is, N=LCM(P1,P2,…,Pm)=LCM(P1,LCM(P2,…,LCM(Pm-1,Pm))). The TDM unit 13 can obtain the number of inferences of each model i within a total cycle N by calculating Num(Mi)=N/Pi, and thereby determine the timing distribution of each model M1-Mm within a total cycle N (for example, multi-model parallel reasoning of model M1 once, model M2 twice, and model M3 four times within a total cycle N).

After that, the TDM unit 13 can determine at least one switching node of each model M1-Mm to be time-division multiplexed from multiple processing nodes inherent to each model M1-Mm according to the total cycle N and the number of inferences that each model M1-Mm needs to complete within a total cycle N, so as to give priority to meeting the delay requirements of each model M1-Mm (i.e., the interval between completing two neural network inferences). Here, the present invention uses two adjacent switching nodes i and i+1 to form a rotation time slice T for performing neural network inference on the corresponding model Mi. The number of rotation time slices T of the model Mi within a total cycle N is Num_T(i)=N/T*Num(Mi).

Afterwards, the TDM unit 13 can perform multi-model reasoning on the inference data samples of each model M1~Mm based on the above-mentioned first storage space distribution and the switching node i of each model M1~Mm, using the full computing resources of the NPU chip 10 to determine the storage space missing of each model M1~Mm.

Specifically, the above-mentioned inference data samples can have the same form and size as the inference data for subsequent multi-model inference, and have similar content (for example: both are image data of the same target size and with noise signals and/or mosaics). The technician can connect the NPU chip 10 to an external buffer, and statically divide the cache space of the external buffer into m blocks according to the compilation requirements of each model M1~Mm, and then load the model parameters of each model M1~Mm into each parameter storage space W1~Wm inside the NPU chip 10 to complete the initialization of each model M1~Mm.

Afterwards, the TDM unit 13 may start the MAC array computing unit 11 and the VPU 12 in the NPU chip 10 to run the NPU chip 10 as shown in FIG4 . In response to starting the first switching node of the first model M1, the MAC array computing unit 11 and the VPU 12 may read the model parameters of the first model M1 from the first storage space W1 corresponding to the first model M1 in the above-mentioned first storage space distribution, and read the inference data samples of the first model M1 from the first storage space F1 corresponding to the first model M1 in the above-mentioned first storage space distribution, so as to perform the first model inference using the full computing resources of the NPU chip 10. In response to starting the second switching node of the subsequent second model M2, the NPU chip 10 may interrupt the first model inference, first write the first intermediate data generated by the first model inference into the first storage space F1 of the SRAM_F 142, then write the overflowed first intermediate data into the corresponding first external storage space, and count the data amount of the overflowed first intermediate data to determine the storage space S_M1_j missing from the first model M1, where j represents the rotation of the jth cycle.

Furthermore, the read and write operations and calculation operations of the NPU chip 10 can be performed independently and simultaneously. The NPU chip 10 can also preferably write the inference data samples of the second model M2 into the second storage space F2 corresponding to the second model M2 in the first storage space distribution while performing the first model inference. In this way, in response to starting the subsequent second switching node of the second model M2, the MAC array computing unit 11 and the VPU 12 in the NPU chip 10 can read the inference data samples of the second model M2 from the second storage space F2 in real time via on-chip transmission, and switch the full computing resources of the NPU chip 10 to the second model M2 to perform the second model inference, so as to determine the second external storage space S_M2_j used for the second model inference as described above. By synchronously processing the separated data read and write and calculation operations, the present invention can write the inference data of the subsequent model i+1 into the corresponding storage space Fi in advance while performing the neural network inference calculation of the previous model i, thereby further improving the efficiency and real-time performance of multi-model inference.

Similarly, the NPU chip 10 can perform round-robin calculations according to the fixed time slice T of each model M1~Mm as described above, so as to complete multiple round-robin calculations of all models M1~Mm when the total cycle N/T time slices are consumed. At this time, each model Mi has completed Num(Mi) inference calculations, and the external storage requirement size S_Mi_j required by the NPU chip 10 for each model i within a total cycle N is obtained. Afterwards, the TDM unit 13 can take the maximum value of the external storage space required for each model i in multiple rounds of round-robin calculations, that is, S_Mi=MAX(S_Mi_j), to respectively determine the storage space S_Mi missing for each model M1~Mm in the first storage space distribution.

Please continue to refer to Figure 3. After determining the storage space S_Mi that is missing for each model in the first storage space distribution, the TDM unit 13 can optimize the first storage space distribution accordingly to determine a second storage space distribution that meets the multi-model reasoning requirements.

Specifically, in the process of determining the second storage space distribution, the TDM unit 13 can preferably expand the corresponding input data and intermediate data storage space Fi (i.e., Fi'=Fi+S_Mi) according to the storage space S_Mi that is missing in the model Mi in the above-mentioned first storage space distribution. In addition, in order to ensure that the total size of the input and intermediate data storage space F1'~Fm' of each model M1~Mm in the second storage space distribution does not exceed SRAM_F 142, that is, SUM(F1':Fm')≤SRAM_F, the TDM unit 13 can also correspondingly reduce the input data and intermediate data storage space Fj of at least one remaining model Mj (i.e., Fj'=Fj-S_Mi) to determine the second storage space distribution in which each model M1~Mm does not lack storage space. If the distribution scheme of SUM(F1':Fm')≤SRAM_F cannot be obtained, the TDM unit 13 can determine that the on-chip storage space of the NPU chip 10 cannot support the time division multiplexing of m models, and the number of models needs to be reduced.

Further, after optimizing the above-mentioned first storage space distribution to determine the second storage space distribution, the TDM unit 13 can also reinitialize the state of each model M1~Mm based on the second storage space distribution, and re-rotate and calculate whether each model M1~Mm in the second storage space distribution still lacks storage space as described above. If each model M1~Mm in the second storage space distribution still lacks storage space, the TDM unit 13 can optimize the on-chip storage space of the NPU chip 10 again as described above until each model Mi has no lack of storage space, that is, S_Mi＝0.

Please continue to refer to Figures 2 and 4. After determining the switching nodes for latency optimization reasoning and the storage space corresponding to each model M1~Mm, the NPU chip 10 can synchronously obtain reasoning data about multiple images through a multi-channel image signal processor (Image Signal Processor, ISP), and/or obtain multi-channel reasoning data by slicing at least one acquired reasoning data. Then, its MAC array computing unit 11 and VPU 12 use the full computing resources of the NPU chip 10 to perform time-division multiplexing reasoning of each model M1~Mm in turn according to the switching nodes determined by the TDM unit 13, so as to achieve latency optimization reasoning of multiple models M1~Mm.

Specifically, in the process of performing latency optimization reasoning, in response to starting the first switching node of the first model M1, the MAC array computing unit 11 and the VPU 12 can read the reasoning data of the first model M1 from the third storage space F1' corresponding to the first model M1 in the predetermined second storage space distribution, so as to use the full computing resources of the NPU chip 10 to perform the first model reasoning. Afterwards, in response to starting the second switching node of the subsequent second model M2, the MAC array computing unit 11 and the VPU 12 can interrupt the first model reasoning, write the third intermediate data generated by the first model reasoning into the third storage space F1', and read the reasoning data of the second model from the fourth storage space F2' corresponding to the second model M2 in the second storage space distribution, so as to use the full computing resources of the NPU chip 10 to perform the second model reasoning. Similarly, in response to starting the subsequent mth switching node of model Mm, the MAC array computing unit 11 and the VPU 12 may interrupt the inference of the m-1th model as described above, write the intermediate data generated by the inference of the m-1th model into the storage space F(m-1)’, and read the inference data of model Mm from the storage space Fm’ corresponding to model Mm in the second storage space distribution, so as to utilize the full computing resources of the NPU chip 10 to perform the inference of the mth model, thereby realizing the delay-optimized inference of multiple models M1~Mm, and effectively controlling the delay of each model M1~Mm (i.e., the interval between completing two neural network inferences).

In addition, please refer to Figure 5 and Figure 6. Figure 5 shows a flow chart of determining a switching node and storage space distribution for storage space optimization reasoning according to some embodiments of the present invention. Figure 6 shows a schematic diagram of storage space optimization reasoning according to some embodiments of the present invention.

As described above, in the embodiment where each model M1-Mm performs time-division multiplexing round-robin reasoning according to a fixed time slice T0 or a time slice T composed of delay-priority switching nodes, it is very likely that the on-chip storage space of the NPU chip 10 cannot support the time-division multiplexing of m models, thereby causing the failure of multi-model reasoning. To this end, in the embodiment shown in FIG5 , the present invention can also determine the switching node for storage space optimization reasoning in an offline manner based on a large number of pre-prepared reasoning data samples.

Specifically, the present invention can still assume that there are m neural network models M1~Mm) that need to be processed in parallel, and determine the inference cycle P1~Pm (in ms) for each model M1~Mm to complete an inference. Afterwards, the TDM unit 13 in the NPU chip 10 can statically divide the full storage space of the NPU chip 10 into parameter storage spaces W1~Wm for storing parameters of each model, and input and intermediate data storage spaces F1~Fm for storing input data and intermediate data of each model according to the number of models m that need to be processed in parallel, so as to determine the third storage space distribution for storing the inference data of each model M1~Mm. Here, the parameter storage space W1~Wm of each model M1~Mm can be determined by compiling each model M1~Mm respectively and setting the target performance FPS (Frames per second) and total bandwidth, and its total size should not exceed SRAM_W 141, that is, SUM(W1:Wm)≤SRAM_W. Correspondingly, the input and intermediate data storage space F1~Fm of each model M1~Mm can also be determined by compiling each model M1~Mm separately and setting the target performance FPS (Frames per second) and total bandwidth, and the total size should not exceed SRAM_F 142, that is, SUM(F1:Fm)≤SRAM_F.

Afterwards, the TDM unit 13 can determine the total cycle N of multi-model reasoning according to the reasoning cycles P1-Pm of each model M1-Mm, and thereby determine the number of reasonings of each model M1-Mm within the total cycle N, that is, N=LCM(P1, P2,…, Pm)=LCM(P1, LCM(P2,…, LCM(Pm-1, Pm))). The TDM unit 13 can obtain the number of reasonings of each model i within a total cycle N by calculating Num(Mi)=N/Pi, and thereby determine the timing distribution of each model M1-Mm within a total cycle N (for example: multi-model parallel reasoning of model M1 once, model M2 twice, and model M3 four times within a total cycle N).

Afterwards, the TDM unit 13 can use the full computing resources of the NPU chip 10 to perform model reasoning on the inference data samples of each model M1~Mm based on the above-mentioned third storage space distribution and the processing nodes inherent in each model M1~Mm, so as to respectively determine the storage space lacking in each processing node of each model M1~Mm.

Specifically, the inference data sample can have the same form and size as the inference data for subsequent multi-model inference, and have similar content (for example: both are of the same target size and have image data with noise signals and/or mosaics). The technician can connect the NPU chip 10 to an external buffer, and statically divide the cache space of the external buffer into m blocks according to the compilation requirements of each model M1~Mm, and then load the model parameters of each model M1~Mm into each parameter storage space W1~Wm inside the NPU chip 10 to complete the initialization of each model M1~Mm.

Afterwards, the TDM unit 13 can start the MAC array computing unit 11 and the VPU 12 in the NPU chip 10 to use the full computing resources of the NPU chip 10 to perform model reasoning on the reasoning data samples of each model M1~Mm in turn. Specifically, in the process of traversing the model reasoning of any model Mi through its processing nodes, the NPU chip 10 can interrupt the current model reasoning in response to any processing node in the model Mi, first write the fifth intermediate data generated by the model reasoning into the fifth storage space Fi corresponding to the model Mi in the third storage space distribution, and then write the overflowed fifth intermediate data into the external storage space, so as to count the amount of data S_Mi_j of the model Mi overflowing to the external buffer at each processing node j according to the amount of data of the overflowed fifth intermediate data, and thereby determine the storage space lacking in each model M1~Mm at its processing node. Here, j≤Ni, j is the jth node of the model Mi, and Ni is the total number of nodes of the model Mi.

Please continue to refer to Figure 5. After determining the storage space missing in each processing node of each model M1~Mm, the TDM unit 13 can sort the size of the storage space missing in each processing node of each model M1~Mm, recorded as S_Mi[N]=sort(S_Mi_j), and divide each model into subgraphs of different numbers n in order of the missing storage space from small to large. For example, when dividing into two subgraphs, the TDM unit 13 can determine the cutting position of each subgraph according to the processing node S_Mi[0]. For another example, when dividing into three subgraphs, the TDM unit 13 can determine the cutting position of each subgraph according to the processing nodes S_Mi[0] and S_Mi[1]. By analogy, when divided into n subgraphs, the TDM unit 13 can determine the segmentation position of each subgraph according to the processing nodes S_Mi[0]~S_Mi[n-2], and finally obtain the subgraph model Mi=set{S_i(1), S_i(2),..., S_i(n-1)} of different numbers n divided by each model M1~Mm, where S_i(n-1) indicates that the model Mi is divided into a set of n-1 subgraphs.

Afterwards, the TDM unit 13 can perform inference tests on the subgraph set of the model Mi to determine the required calculation time Mi_Tn, and determine the switching node position C_i = set{Node1, Node2,…, Node_j} of the model Mi based on the position of the processing node corresponding to the maximum number of subgraphs n _max that meets the total performance requirements of multi-model reasoning according to the number of inferences Num(Mi) and the calculation time Mi_Tn.

Specifically, during the inference test, the TDM unit 13 may first start the MAC array computing unit 11 and the VPU 12 in the NPU chip 10, and use the full computing resources of the NPU chip 10 to perform model inference on the inference data samples of each model M1 to Mm in turn. In response to the first processing node of any model Mi, the MAC array computing unit 11 and the VPU 12 may read the model parameters of the model Mi from the sixth storage space Wi corresponding to the model Mi in the third storage space distribution, and read the inference data samples of the model Mi from the sixth storage space Fi corresponding to the model Mi in the third storage space distribution, so as to use the full computing resources of the NPU chip 10 to perform the sixth model inference. Afterwards, in response to the subsequent second processing node of the model Mi, the NPU chip 10 can interrupt the model reasoning of the model Mi by time-division multiplexing multiple models, first write the sixth intermediate data generated by the model reasoning into the sixth storage space Fi, and then re-read the sixth intermediate data from the sixth storage space to continue the model reasoning between the first processing node and the second processing node, and so on, until the complete reasoning of the model Mi is completed to determine the calculation time Mi_Tn required for the model Mi.

Afterwards, the TDM unit 13 can determine the total performance requirement of multi-model reasoning, i.e., m*N, according to the number of models m and the total cycle N of multi-model reasoning, and then determine the maximum number of subgraphs n _max that meets the total performance requirement of multi-model reasoning (i.e., SUM(Num(Mi)*Mi_Tn)≤m*N), thereby determining the switching node position Ci=set{Node1,Node2,…,Node_j} of the model Mi according to the position of the processing node corresponding to the maximum number of subgraphs n _max . Here, the larger the n value is, the more subgraphs are divided, so that each model M1~Mm can be switched more frequently for time-division multiplexing multi-model reasoning, thereby further reducing the delay of each model M1~Mm (i.e., the interval between completing two neural network reasoning).

Please continue to refer to Figures 2 and 5. After determining the switching nodes for storage space optimization reasoning and the storage space corresponding to each model M1~Mm, the NPU chip 10 can synchronously obtain reasoning data about multiple images through a multi-channel image signal processor (ISP), and/or obtain multi-channel reasoning data by slicing at least one acquired reasoning data. Then, its MAC array computing unit 11 and VPU 12 use the full computing resources of the NPU chip 10 to perform time-division multiplexing reasoning of each model M1~Mm in turn according to the switching nodes determined by the TDM unit 13, so as to realize storage space optimization reasoning of multiple models M1~Mm.

Specifically, in the process of performing storage space optimization reasoning, in response to starting the third switching node Ci of the third model Mi, the MAC array computing unit 11 and the VPU 12 can read the reasoning data of the third model Mi from the seventh storage space F1 corresponding to the third model Mi in the predetermined third storage space distribution, so as to perform the third model reasoning using the full computing resources of the NPU chip 10. Afterwards, in response to starting the fourth switching node Ci+1 of the subsequent fourth model Mi+1, the MAC array computing unit 11 and the VPU 12 can interrupt the third model reasoning, write the seventh intermediate data generated by the third model reasoning into the seventh storage space F1, and read the reasoning data of the fourth model Mi+1 from the eighth storage space corresponding to the fourth model Mi+1 in the third storage space distribution, so as to perform the fourth model reasoning using the full computing resources of the NPU chip 10. By analogy, in response to starting the mth switching node Cm of the subsequent model Mm, the MAC array computing unit 11 and the VPU 12 can interrupt the inference of the m-1th model as described above, write the intermediate data generated by the inference of the m-1th model into the storage space F(m-1), and read the inference data of the model Mm from the storage space Fm corresponding to the model Mm in the third storage space distribution, so as to utilize the full computing resources of the NPU chip 10 to perform the inference of the mth model, thereby realizing the storage space optimized inference of multiple models M1~Mm, and effectively controlling the intermediate data generated by the time-division multiplexing inference of each model M1~Mm.

In summary, by adopting the switching nodes determined according to the goals of delay priority and/or storage space priority, the calculation time of each model M1~Mm is divided into multiple small time slots. The above-mentioned multi-model reasoning method, multi-model reasoning system and computer-readable storage medium provided by the present invention can rotate the neural network reasoning calculation of each model M1~Mm by time-division multiplexing reasoning, and immediately save the intermediate data such as the state information calculated by the current model Mi after the corresponding time slot ends, so as to rotate the neural network reasoning calculation of the next model M(i+1), thereby effectively controlling the delay of each model M1~Mm and reducing the cache space required for the intermediate data generated by multi-model reasoning. In addition, since each model M1~Mm caches the intermediate data by on-chip storage and shares the full computing resources of the NPU chip 10 for neural network reasoning, the present invention can effectively shorten the switching time between each model M1~Mm and avoid the idle waste of computing resources in the entire multi-model reasoning system, thereby improving the computing efficiency of each model M1~Mm as a whole to support concurrent reasoning of multiple models M1~Mm.

In addition, as mentioned above, after obtaining the entire image data with a resolution of H×W from the camera module, the existing technology usually needs to tile the entire image through the image signal processor (Image Signal Processor, ISP), divide it into multiple N×M (N<H, M<W) tile images according to the data processing capability of the network processing unit (NPU), and store them in the ISP buffer. Then the network processing unit (NPU) reads the image data of each tile image from the ISP buffer one by one for network reasoning. On the one hand, this will introduce additional operations and intermediate data such as slicing, zero-filling and windowing, and trimming of overlapping images to increase the requirements for the overall computing power of the system. On the other hand, it also requires the system to have a larger data cache space to greatly increase the area, power consumption and cost of the image signal processor (ISP), thereby severely limiting the development and application of the existing network processing unit (Network Processing Unit, NPU) to parallel process multiple image data.

In order to overcome the above-mentioned defects existing in the prior art, the present invention provides a network processing unit, an extended reality display chip, an extended reality display device, a parallel processing method for image data, and a computer-readable storage medium, which can acquire multiple channels of image data line by line in parallel, write each channel of image data into a buffer at a preset speed, and then read and process the written i-th channel of image data from the buffer by using the full computing resources of the network processing unit through the i model. By adopting these configurations, the present invention can improve the accuracy of image processing by eliminating the need for tile processing of the entire image, and reduce the requirements for the overall computing power of the system and data cache space, thereby improving the ability of a single network processing unit (NPU) to process multiple channels of image data in parallel under the same hardware processing technology and cost conditions, and thereby flexibly and fully utilizing the overall computing power of the system to reduce network inference delay, so as to meet the needs of parallel processing of binocular image data in extended reality (Extended Reality, XR) display devices.

In some non-limiting embodiments, the parallel processing method of the image data provided in the seventh aspect of the present invention can be implemented via the extended reality (XR) display chip provided in the fifth aspect of the present invention. Please refer to Figure 7 for details, which shows an architecture diagram of an extended reality display chip provided according to some embodiments of the present invention.

In the embodiment shown in FIG. 7 , the above-mentioned extended reality (XR) display chip provided in the fifth aspect of the present invention can be configured in the above-mentioned extended reality (XR) display device provided in the sixth aspect of the present invention, which is configured with a memory (not shown), at least two image signal processors 71-72, and a network processing unit 80 provided in the fourth aspect of the present invention. The memory includes but is not limited to the above-mentioned computer-readable storage medium provided in the fifth aspect of the present invention, on which computer instructions are stored. The at least two image signal processors 71-72 are respectively connected to the left eye camera and the right eye camera of the extended reality display device to obtain the real scene image of the real world collected by them. The network processing unit 80 is respectively connected to the memory and each image signal processor 71-72, and is suitable for reading and executing the computer instructions stored on the memory to implement the above-mentioned parallel processing method of image data provided in the fourth aspect of the present invention, so as to alternately obtain the image data outputted by each image signal processor 71-72 line by line, and perform parallel processing on the obtained image data.

The following will describe the working principles of the above-mentioned network processing unit 80, extended reality display chip and extended reality display device in conjunction with some embodiments of the parallel processing method of image data. Those skilled in the art will understand that these embodiments of the parallel processing method are only some non-restrictive implementation methods provided by the present invention, which are intended to clearly demonstrate the main concept of the present invention and provide some specific solutions that are convenient for the public to implement, rather than to limit all functions or all working modes of the network processing unit 80, the extended reality display chip and the extended reality display device. Similarly, the network processing unit 80, the extended reality display chip and the extended reality display device are also only a non-restrictive implementation method provided by the present invention, and do not constitute a limitation on the execution subject and execution order of each step in the parallel processing method of the image data.

Please refer to FIG. 7 and FIG. 8 . FIG. 8 shows a flowchart of parallel processing of image data according to some embodiments of the present invention.

As shown in FIG7 , the network processing unit 80 provided by the present invention is configured with hardware devices such as caches 811 to 812, a multiplication and accumulation (MAC) array computing unit 82, a vector processing unit (VPU) 83, and a register 84, and is also configured with a software program containing a plurality of pre-trained image processing models, wherein computing operations and storage operations can be performed separately and independently, thereby enabling full computing resource sharing of the network processing unit 80 through the MAC array computing unit 82 and the vector processing unit (VPU) 83, so as to flexibly support the network processing unit 80 to perform operations such as data reading, computing, and caching.

In some embodiments, technicians can train and compile the input parameters of the neural network model for image processing such as denoising (Denoise) and/or de-mosaicing (Demosic) in an offline manner in advance according to the size of the entire real-scene image, and adapt to the specific usage scenario of the binocular camera in the XR display device, initialize the trained model N (for example: N=2) times, and obtain N neural network models with the same functions to correspond to each image signal processor 71~72 respectively.

Those skilled in the art will understand that the above-mentioned scheme of configuring N neural network models with the same functions is only a non-limiting implementation method provided by the present invention, which is intended to clearly demonstrate the main concept of the present invention and provide a specific scheme that is easy for the public to implement, rather than to limit the scope of protection of the present invention.

Optionally, in other embodiments, those skilled in the art may also configure a variety of neural network models with different parameters, structures and/or functions to adapt to various image processing requirements such as denoising and demosic removal, so as to correspondingly meet the requirements of multifunctional parallel processing of binocular image data in XR devices.

In addition, in some embodiments, the buffers 811-812 may use a static random access memory (SRAM), which is configured at the input end of the network processing unit 80 and is statically divided into N (for example, N=2) independent cache spaces to adapt to the specific usage scenario of the binocular camera in the XR display device. Further, each cache space can be preferably divided into an input buffer space, a parameter buffer space, and a feature buffer space to cache the image data currently to be processed by the corresponding image processing model, the model parameter data, and the intermediate data generated by the neural network inference performed by the corresponding image processing model.

The first buffer 811 and the second buffer 812 are used below to refer to the two independent buffer spaces. The first buffer 811 is connected to the image signal processor 71 via the ISP buffer 711, and the second buffer 812 is connected to the image signal processor 72 via the ISP buffer 721. The two buffers read two channels of image data line by line from the corresponding ISP buffers 711-712 at a preset first speed _v1 , respectively, so that each image processing model can read the currently written image data from the corresponding buffers 811-812 according to the corresponding switching nodes, and use the full computing resources of the network processing unit 80 to process the image data, so as to perform parallel processing of the two channels of image data of the binocular camera. Here, the first speed _v1 is not less than N times the second speed _v2 of the image data output by each image signal processor 71-72. The image processing model can include multiple inherent processing nodes, and complete a cycle of neural network reasoning by sequentially executing the relevant operations of each processing node. The switching node can be obtained by screening from the processing nodes of each image processing model according to the goals of delay priority and/or storage space priority, and correspondingly distributed according to the delay requirements of each neural network model and/or the total cache data volume of intermediate data generated by each neural network model to indicate the opening and closing time of the processing window of each image processing model.

Specifically, each of the above-mentioned image processing models can respectively include multiple inherent processing nodes, and complete a cycle of neural network reasoning by sequentially executing relevant operations of each processing node.

For the above-mentioned embodiment of distribution according to the delay requirements of each image processing model, the technician can divide the full storage space of the network processing unit 80 in advance according to the number of models N that need to be processed in parallel in an offline manner to determine the first storage space distribution for storing the input image data, model parameter data, intermediate data and other reasoning data of each model. In addition, the technician can also determine the total cycle of the N model reasoning to complete one reasoning according to the reasoning cycle of each model to complete one reasoning, and thereby determine the number of reasoning times of each model in the total cycle. Afterwards, the technician can select and determine at least one switching node for time-division multiplexing each model from multiple processing nodes of each model according to the total cycle and the number of reasoning times of each model, and then based on the first storage space distribution and the switching nodes of each model, use the full computing resources of the network processing unit 80 to perform multi-model reasoning on the reasoning data samples of multiple models to determine the storage space lacking for each model. Afterwards, the technicians can optimize the first storage space distribution according to the total storage space of the network processing unit 80 and the storage space lacking in each model to determine the second storage space distribution that meets the multi-model reasoning requirements, and thereby determine the static partitioning scheme of the buffers 811 to 812 to ensure that the parallel processing of the binocular image data of the XR device can be completed within the specified delay range.

In addition, for the above-mentioned embodiment of the total cache data volume distribution of the intermediate data generated by each image processing model, the technician can divide the full storage space of the network processing unit 80 according to the number of models N that need to be processed in parallel to determine the third storage space distribution for storing the reasoning data of each model, and determine the total cycle of multi-model reasoning according to the reasoning cycle of each model as described above, and then determine the number of reasoning times of each model in the total cycle. Afterwards, the technician can use the full computing resources of the network processing unit 80 to perform model reasoning on the reasoning data samples of each model based on the third storage space distribution and each processing node of each model, so as to respectively determine the storage space lacking in each processing node of each model. Afterwards, the technician can divide different numbers of subgraphs for each model according to the processing node in the order of the lack of storage space from small to large, and perform reasoning tests to respectively determine the calculation time required for each model. Afterwards, the technician can determine the maximum number of subgraphs whose reasoning times and calculation time meet the total performance requirements of multi-model reasoning, and select the switching node that needs to cache the minimum amount of intermediate data from the processing nodes of each model according to the position of the processing node corresponding to the maximum number of subgraphs. Here, the total performance requirement of the multi-model reasoning can be represented by the cumulative sum of the number of reasonings and the calculation time of each image processing model. By selecting the position of the processing node corresponding to the maximum number of sub-graphs to determine the switching node of each model, the present invention can further reduce the amount of intermediate data generated by time-division multiplexing of each image processing model, thereby further reducing the area, power consumption and cost of the network processing unit 80 and the image signal processors 71-72.

Therefore, the present invention can determine at least one switching node from multiple processing nodes of each image processing model according to the specific goals of latency priority and/or storage space priority, and determine the storage space corresponding to each image processing model, so as to utilize the full computing resources of the network processing unit 80 to perform time-division multiplexing reasoning of each image processing model in sequence, so as to realize parallel reasoning of latency optimization and/or storage space optimization of multiple models.

As shown in FIG8 , in the process of processing two channels of image data of the binocular camera in parallel, the two-channel image signal processors 71-72 can be connected to the left camera and the right camera of the XR display device respectively, so as to obtain the collected left-eye image data and right-eye image data from the image sensors of the left camera and the right camera respectively, and pre-process them. Afterwards, the two-channel image signal processors 71-72 can continuously transmit the pre-processed left-eye image data and right-eye image data to the corresponding ISP buffers 711-712 line by line at _{the second} speed v ₂ in a ping-pong buffer read-write manner within the preset exposure time t 1. Here, for the image processing function of eliminating noise and/or eliminating mosaics, the left-eye image data and the right-eye image data collected by the binocular camera of the XR display device can be original image data with noise signals and/or mosaics.

Specifically, in the above-mentioned ping-pong buffer read and write mode, the image signal processors 71-72 can synchronously write the left-eye image data collected by the left-eye camera and the right-eye image data collected by the right-eye camera to the corresponding ISP buffers 711-721 line by line. In response to the end of the preset exposure time _t1 , the input buffer space of the ISP buffers 711-721 will be filled with the 1st to Mth lines of image data of the left-eye image and the right-eye image at the same time. At this time, the image signal processors 71-72 can send an interrupt instruction to the network processing unit 80 to notify it to use the above-mentioned exposure time _t1 as a fixed time slice and write the 1st to Mth lines of image data cached on the ISP buffers 711-721 _line by line to the corresponding buffers 811-812 on the network processing unit 80 at the above-mentioned first speed v1 ( _v1 ≥N· _v2 ).

Afterwards, in response to reaching the predetermined first switching node _T11 that triggers the first model The network processing unit 80 can first determine that the first processing window of the first model is open, thereby reading the 1st to Mth rows of image data currently written into the left-eye image from the first buffer 811, and driving the above-mentioned MAC array calculation unit 82 and the vector processing unit (VPU) 83, so as to support the first model to process the currently written left-eye image data with the full computing resources of the network processing unit 80, and generate corresponding first intermediate data.

Afterwards, in response to reaching the predetermined second switching node _T21 that triggers the second model The network processing unit 80 can determine that the first processing window of the above-mentioned first model is closed, and the second processing window of the second model is opened, so as to first write the first intermediate data generated by the first model between the first switching node _T11 and the second switching node _T21 back to the feature buffer space of the first buffer 811 to free up the computing resources of the network processing unit 80, and then read the 1st to Mth rows of image data currently written in the right-eye image from the second buffer 812, and drive the above-mentioned MAC array computing unit 82 and vector processing unit (VPU) 83 to switch all computing resources of the network processing unit 80 to the second model to support the second model to process the currently written right-eye image data to generate corresponding second intermediate data.

Furthermore, in the above-mentioned ping-pong buffer read and write mode, the image signal processors 71-72 can continue to write the left eye image data collected by the left eye camera and the right eye image data collected by the right eye camera to the corresponding ISP buffers 711-721 line by line while the network processing unit 80 reads the image data cached in the ISP buffers 711-721, so as to realize the dynamic synchronization of reading and writing data. In response to the end of the preset exposure time 2t ₁ , the input buffer space of the ISP buffers 711-721 will be filled with the image data of the M+1 to 2Mth lines of the left eye image and the right eye image again. As described above, the image signal processors 71-72 may again send an interrupt instruction to the network processing unit 30, notifying it to continue to use the above-mentioned exposure time t ₁ as a fixed time slice and write out the M+ _1-2Mth lines of image data cached in _the ISP buffers 711-721 to the corresponding buffers 811-812 on the network processing unit 80 line by line at the _above -mentioned first speed v 1 (v 1 ≥N·v 2 ), thereby preventing the 2M+1-3Mth lines of left-eye image data written into the ISP buffers 711-721 within the next exposure time 2t ₁ -3t ₁ from being accumulated and overflowing.

Afterwards, in response to reaching the first switching node _T12 that triggers the first model again The network processing unit 80 can determine that the second processing window of the above-mentioned second model is closed, and the first processing window of the first model is opened again, so as to first write the second intermediate data generated by the second model between the second switching node T ₂₁ and the first switching node T ₁₂ back to the feature buffer space of the second buffer 812 to free up the computing resources of the network processing unit 80, and then read the M+1~2Mth rows of image data currently written in the left-eye image from the first buffer 811, as well as the first intermediate data generated by the previous round of the first neural network reasoning, and drive the above-mentioned MAC array computing unit 82 and the vector processing unit (VPU) 83 to switch all computing resources of the network processing unit 80 back to the first model to support the first model to continue processing the currently written left-eye image data to regenerate the corresponding first intermediate data.

By analogy, the two image processing models in the network processing unit 80 can perform neural network inference on the left-eye image and the right-eye image alternately window by window according to the predetermined switching nodes and the reading and writing method of the ping-pong cache, thereby meeting the demand for parallel processing of binocular image data in the XR device by configuring less ISP cache space (for example: 2M lines of image data).

Please further refer to FIG. 9 , which shows a schematic diagram of acquiring image data line by line according to some embodiments of the present invention.

As shown in FIG9 , the XR display chip provided by the present invention can use the sliding window shown in the figure to perform convolution calculation from left to right and from top to bottom in the direction of row priority. This way of reading and calculating image data is consistent with the direction in which the image signal processor 71-72 writes data to the ISP buffer 711-721. The real-time flow of image data from the image signal processor 71-72 to the network processing unit 80 can be achieved by quantitatively configuring the read and write speeds of the image signal processor 71-72 and the network processing unit 80, thereby reducing the waiting time of the image data in the ISP buffer 711-721 cache, so as to reduce the latency of image processing.

On the other hand, compared with the traditional column-wise calculation method, which requires writing and caching all 22 rows of data of the entire image before starting the neural network inference calculation, and therefore the entire image must be divided into multiple tile images to reduce the cache requirements, the row-priority image data reading and calculation method adopted by the present invention only needs to write and cache a few rows (for example: 3 rows corresponding to the sliding window size) of image data to perform neural network inference of the corresponding image in real time, thereby greatly reducing the cache space requirements of the ISP buffers 711 to 721, thereby greatly reducing the area, power consumption and cost of the image signal processor, and eliminating the need for tile processing of the entire image to reduce the requirements for the overall system computing power and data cache space, thereby reducing the requirements for the overall system computing power.

Please refer to FIG. 10 for details, which shows a structural diagram of an XR display chip provided according to some embodiments of the present invention.

As shown in FIG. 10 , by adopting the above-mentioned row-priority image data reading and calculation method provided by the present invention, in addition to the internal buffer 91 of the network processing unit 90, the XR display chip only needs to configure a small-area, small-capacity (for example: 0.4MB) ISP buffer 92 outside the network processing unit 80 to meet the needs of parallel processing of binocular image data in the XR device, thereby facilitating the further development and application of the network processing unit 80 to parallel processing of multi-channel image data.

Furthermore, in some embodiments of the invention, each image processing model may include a multi-layer neural network structure. In response to reading multiple lines (e.g., 3 lines) of image data satisfying the above sliding window size from the corresponding buffer 811 or 812, the image processing model may perform convolution calculations from left to right and from top to bottom in a row-first direction to synchronously complete the neural network reasoning of the corresponding number of layers.

Please refer to Figure 11 for details, which shows a schematic diagram of a noise reduction process provided according to some embodiments of the present invention. Taking the AI noise reduction (AIDenoise) process shown in Figure 11 as an example, it is divided into two parts, an encoder and a decoder, at the algorithm module level. AIDenoise reasoning based on machine learning is a process of estimating a potential clean image from an actually observed noisy image. The image processing model can feature map the image through an encoder, and then integrate and restore the image features through a decoder to finally output a clean image with noise eliminated. Specifically, in the process of performing neural network reasoning, the image processing model can use convolution to form a jump connection part between the encoder and the decoder, so that the features extracted by image downsampling are integrated into the upsampling part to promote the fusion of feature information. Afterwards, by learning the noise in the training image, the potential mapping of the corresponding reference image is obtained, and the image processing model can perform neural network reasoning on the acquired noisy image according to the potential mapping to finally obtain a clean image with noise eliminated. Compared with traditional denoising methods, this AI denoising not only better preserves the edge texture details of the image, but also can use the architecture of the network processing unit (NPU) for parallel computing, thereby making full use of hardware performance to speed up the computing operation rate.

Afterwards, in response to completing the neural network reasoning of the preset number of layers L according to the currently written multiple lines of image data, the image processing model begins to generate result data about the initial multiple lines of images of the corresponding left/right eye images. Here, for the above-mentioned image processing function of eliminating noise, the result data generated by the image processing model can be denoised image data after eliminating noise. Correspondingly, for the above-mentioned image processing function of eliminating mosaics, the result data generated by the image processing model can also be restored image data after eliminating mosaics.

Afterwards, the network processing unit 80 can output the generated result data line by line to the corresponding ISP buffers 712~722 at the first speed _v1 as shown in Figure ₇ according to the reading and writing method of the Ping-pong buffer, and the ISP buffers 712~722 return the received result data of the preset number of lines to the corresponding image signal processors 71~72 at the second speed v2, so that the image data in each image signal processor 71~72 can flow through the network processing unit 80, thereby reducing the size requirements of the ISP buffers 712~722 and the internal buffers 811~812 of the network processing unit 80, and reducing the latency of image processing.

Specifically, continuing to take the original image data shown in FIG. 9 as an example, assuming that its total resolution is above 1000 lines, the network processing unit 80 can complete the neural network reasoning of the preset number of layers L when processing the Kth line of image data, and output the 1st to Mth lines of processing result data, and output the M+1st to 2Mth lines, 2M+1st to 3Mth lines, and other subsequent processing result data window by window as the corresponding model processing window is opened and closed. Here, K can be determined by the receptive field of the image processing model. For the same model that processes binocular image data, it can have the same K value of about 60 to 100 lines. The value of M can correspond to the number of image data lines (for example: M=8) written by the ISP buffer 711 to 721 to the corresponding buffer 811 to 812 on the network processing unit 80 at the fixed time slice _t1 , so as to prevent the backlog and overflow of image data in the buffer 811 to 812.

In this way, by outputting the generated result data line by line to the corresponding ISP buffers 712~722 in a quantitative and constant speed manner according to the reading and writing method of the ping-pong cache, the network processing unit 80 can alternately obtain the multiple original image data provided by the multiple image signal processors 71~72 while returning the processing result data to each image signal processor 71~72 in equal amounts, thereby realizing the flow and dynamic balance of image data.

Furthermore, in some embodiments, in response to completing the neural network inference of a preset number of layers L and outputting the generated result data to the corresponding ISP buffers 712~722, the network processing unit 80 may also preferably delete multiple lines of image data that only involve the first L-1 layers of neural network inference to save feature buffer space of the buffers 811~812.

In summary, the network processing unit 80, XR display chip, XR display device, parallel processing method of image data, and computer-readable storage medium provided by the present invention can acquire multiple channels of image data line by line in parallel, write each channel of image data into the buffer at a preset speed, and then use the full computing resources of the network processing unit through the i model to read and process the written i-th channel of image data from the buffer. By adopting these configurations, the present invention can improve the accuracy of image processing by eliminating the need for tile processing of the entire image, and reduce the requirements for the overall computing power of the system and data cache space, thereby improving the ability of a single network processing unit (NPU) to process multiple channels of image data in parallel under the same hardware processing technology and cost conditions, and thereby flexibly and fully utilizing the overall computing power of the system to reduce network inference delay, so as to meet the needs of parallel processing of binocular image data in XR devices.

Although the above methods are illustrated and described as a series of actions for simplicity of explanation, it should be understood and appreciated that these methods are not limited by the order of the actions, because according to one or more embodiments, some actions may occur in a different order and/or concurrently with other actions from those illustrated and described herein or not illustrated and described herein but understandable to those skilled in the art.

Those skilled in the art will appreciate that information, signals, and data may be represented using any of a variety of different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips cited throughout the above description may be represented by voltage, current, electromagnetic waves, magnetic fields or magnetic particles, optical fields or optical particles, or any combination thereof.

Those skilled in the art will further appreciate that the various illustrative logic blocks, modules, circuits, and algorithm steps described in conjunction with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or a combination of the two. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps are generally described above in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. The technician may implement the described functionality in different ways for each specific application, but such implementation decisions should not be interpreted as resulting in a departure from the scope of the present invention.

The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. The software module may reside in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read and write information from/to the storage medium. In an alternative, a storage medium may be integrated into a processor. The processor and the storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In an alternative, the processor and the storage medium may reside in a user terminal as discrete components.

In one or more exemplary embodiments, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented as a computer program product in software, each function may be stored on or transmitted by a computer-readable medium as one or more instructions or codes. Computer-readable media include both computer storage media and communication media, including any medium that facilitates the transfer of a computer program from one place to another. Storage media may be any available medium that can be accessed by a computer. As an example and not limitation, such a computer-readable medium may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, disk storage or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of an instruction or data structure and can be accessed by a computer. Any connection is also properly referred to as a computer-readable medium. For example, if the software is transmitted from a website, a server, or other remote source using a coaxial cable, a fiber optic cable, a twisted pair, a digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwaves, the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwaves are included in the definition of the medium. Disk and disc as used herein include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc, wherein disk often reproduces data magnetically, while disc reproduces data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.

The previous description of the disclosure is provided to enable any person skilled in the art to make or use the disclosure. Various modifications to the disclosure will be apparent to those skilled in the art, and the general principles defined herein may be applied to other variations without departing from the spirit or scope of the disclosure. Thus, the disclosure is not intended to be limited to the examples and designs described herein, but should be granted the widest scope consistent with the principles and novel features disclosed herein.

Claims (28)

A multi-model reasoning method, characterized by comprising the following steps: Get inference data for multiple models; According to the goals of delay priority and/or storage space priority, at least one switching node is determined from a plurality of processing nodes of each of the models, and a storage space corresponding to each of the models is determined; Writing the inference data of each model into the corresponding storage space of the network processing unit respectively; and According to the switching node, the full computing resources of the network processing unit are utilized to perform time-division multiplexing reasoning of each of the models in turn, so as to realize delay optimization reasoning and/or storage space optimization reasoning of the multiple models. The multi-model reasoning method according to claim 1 is characterized in that, according to the latency priority goal, the steps of determining at least one switching node from a plurality of processing nodes of each of the models, and determining the storage space corresponding to each of the models include: Dividing the full storage space of the network processing unit according to the number of models that need to be processed in parallel to determine a first storage space distribution for storing the inference data of each of the models; Determine the total cycle of multi-model reasoning according to the reasoning cycle of each model, and determine the number of reasoning times of each model in the total cycle; According to the total cycle and the number of inferences of each model, respectively determine at least one switching node for time-division multiplexing each model from a plurality of processing nodes of each model; Based on the first storage space distribution and the switching nodes of each of the models, using the full computing resources of the network processing unit, multi-model reasoning is performed on the reasoning data samples of the multiple models to determine the storage space missing from each of the models; and According to the total storage space of the network processing unit and the storage space lacking in each of the models, the first storage space distribution is optimized to determine a second storage space distribution that meets the multi-model reasoning requirements. The multi-model reasoning method according to claim 2 is characterized in that the step of performing multi-model reasoning on the reasoning data samples of the multiple models based on the first storage space distribution and the switching nodes between the models using the full computing resources of the network processing unit to determine the storage space missing from each of the models comprises: connecting the network processing unit to an external buffer; In response to starting a first switching node of the first model, reading an inference data sample of the first model from a first storage space corresponding to the first model in the first storage space distribution, performing first model inference using full computing resources of the network processing unit, and counting a first external storage space used for the first model inference; and A storage space lacking in the first model is determined according to the first external storage space. The multi-model reasoning method according to claim 3 is characterized in that, in response to starting the first switching node of the first model, reading the reasoning data sample of the first model from the first storage space corresponding to the first model in the first storage space distribution, performing the first model reasoning using the full computing resources of the network processing unit, and counting the first external storage space used for the first model reasoning comprises: In response to starting a second switch node of a subsequent second model, interrupting the first model reasoning, first writing the first intermediate data generated by the first model reasoning into the first storage space, and then writing the overflowed first intermediate data into the external storage space; and The first external storage space is determined according to the data amount of the overflowed first intermediate data. The multi-model reasoning method according to claim 4 is characterized in that the step of performing multi-model reasoning on the reasoning data samples of the multiple models based on the first storage space distribution and the switching nodes between the models using the full computing resources of the network processing unit to determine the storage space missing from each of the models further includes: While performing the first model inference, writing the inference data samples of the second model into a second storage space corresponding to the second model in the first storage space distribution; and In response to the second switching node, the inference data sample of the second model is read from the second storage space, the full computing resources of the network processing unit are switched to the second model, the second model inference is performed using the full computing resources, and the second external storage space used for the second model inference is determined. The multi-model reasoning method according to claim 3 is characterized in that the reasoning data includes model parameter data, input data and intermediate data, the storage space distribution involves the parameter storage space, input data storage space and intermediate data storage space of each of the models, and the step of optimizing the first storage space distribution according to the total storage space of the network processing unit and the storage space lacking of each of the models to determine the second storage space distribution that meets the multi-model reasoning requirements comprises: According to the storage space lacking in each of the models in the first storage space distribution, the corresponding input data storage space and intermediate data storage space are expanded, and/or the input data storage space and intermediate data storage space of the remaining models are reduced to determine a second storage space distribution in which each of the models has sufficient storage space. The multi-model reasoning method according to claim 1 is characterized in that the step of performing time-division multiplexing reasoning of each of the models in sequence using the full computing resources of the network processing unit according to the switching node to achieve delay optimization reasoning and/or storage space optimization reasoning of the multiple models includes: In response to starting the first switch node of the first model, reading the inference data of the first model from a third storage space corresponding to the first model in a predetermined second storage space distribution, and performing the first model inference using the full computing resources of the network processing unit; and In response to starting a subsequent second switching node of a second model, the first model reasoning is interrupted, the third intermediate data generated by the first model reasoning is written into the third storage space, and the reasoning data of the second model is read from the fourth storage space corresponding to the second model in the second storage space distribution, and the second model reasoning is performed using the full computing resources of the network processing unit to achieve delay optimized reasoning of the multiple models. The multi-model reasoning method according to claim 1 is characterized in that, according to the storage space priority goal, the steps of determining at least one switching node from a plurality of processing nodes of each of the models, and determining the storage space corresponding to each of the models include: Dividing the full storage space of the network processing unit according to the number of models that need to be processed in parallel to determine the distribution of the third storage space for storing the inference data of each of the models; Determine the total cycle of multi-model reasoning according to the reasoning cycle of each model, and determine the number of reasoning times of each model in the total cycle; Based on the third storage space distribution and the processing nodes of the models, using the full computing resources of the network processing unit, respectively performing model reasoning on the reasoning data samples of the models to determine the storage space lacking in the processing nodes of the models; In the order of the lack of storage space from small to large, dividing each model into different numbers of subgraphs according to the processing nodes, and performing reasoning tests to determine the calculation time required for each model; Determine the maximum number of subgraphs whose inference times and computation time meet the total performance requirements of the multi-model inference; and According to the positions of the processing nodes corresponding to the maximum number of subgraphs, the switching nodes of each of the models are determined respectively. The multi-model reasoning method according to claim 8, characterized in that the step of performing model reasoning on the reasoning data samples of each model based on the third storage space distribution and each processing node of each model using the full computing resources of the network processing unit to determine the storage space lacking of each model at each processing node comprises: connecting the network processing unit to an external buffer; Using the full computing resources of the network processing unit, perform model reasoning on the reasoning data samples of each model in turn; In response to any of the processing nodes of any of the models, interrupt the current model reasoning of the corresponding model, first write the fifth intermediate data generated by the model reasoning into the fifth storage space corresponding to the model in the third storage space distribution, and then write the overflowed fifth intermediate data into the external storage space; and According to the data amount of the overflowed fifth intermediate data, a fifth external storage space lacking of the model at the processing node is determined. The multi-model reasoning method according to claim 8, wherein the step of performing reasoning testing to determine the calculation time required for each of the models comprises: Using the full computing resources of the network processing unit, perform model reasoning on the reasoning data samples of each model in turn; In response to a first processing node of any of the models, reading inference data samples of the model from a sixth storage space corresponding to the model in the third storage space distribution, and performing sixth model inference using full computing resources of the network processing unit; In response to a subsequent second processing node of the model, interrupting the model reasoning of the model, first writing the sixth intermediate data generated by the model reasoning into the sixth storage space, and then reading the sixth intermediate data from the sixth storage space, so as to continue the model reasoning between the first processing node and the second processing node, and so on, until the complete reasoning of the model is completed; and The computation time required for the model is determined based on the time required to complete the complete inference of the model. The multi-model reasoning method according to claim 8, characterized in that before determining the maximum number of subgraphs whose number of inferences and calculation time meet the total performance requirements of the multi-model reasoning, the multi-model reasoning method further includes the following steps: The total performance requirement of the multi-model reasoning is determined according to the number of the models and the total cycle of the multi-model reasoning. The multi-model reasoning method according to claim 1 is characterized in that the step of performing time-division multiplexing reasoning of each of the models in sequence using the full computing resources of the network processing unit according to the switching node to achieve delay optimization reasoning and/or storage space optimization reasoning of the multiple models includes: In response to starting a third switch node of a third model, reading inference data of the third model from a seventh storage space corresponding to the third model in a predetermined third storage space distribution, and performing inference of the third model using full computing resources of the network processing unit; and In response to starting the fourth switching node of the subsequent fourth model, the third model reasoning is interrupted, the seventh intermediate data generated by the third model reasoning is written into the seventh storage space, and the reasoning data of the fourth model is read from the eighth storage space corresponding to the fourth model in the third storage space distribution, and the fourth model reasoning is performed using the full computing resources of the network processing unit to achieve storage space optimized reasoning of the multiple models. The multi-model reasoning method according to claim 1, wherein the step of obtaining reasoning data of multiple models comprises: Synchronously acquiring inference data about multiple channels of images via multiple channel image signal processors; and/or Data slicing is performed on the at least one path of acquired reasoning data to obtain multiple paths of reasoning data. A multi-model reasoning system, characterized by comprising: a memory having computer instructions stored thereon; and A processor is connected to the memory and configured to execute computer instructions stored in the memory to implement the multi-model reasoning method according to any one of claims 1 to 13. The multi-model reasoning system as described in claim 14 is characterized in that the multi-model reasoning system is configured in a network processing unit. A network processing unit is connected to multiple image signal processors and is equipped with a buffer and multiple pre-trained image processing models, characterized in that: The network processing unit alternately obtains N channels of image data line by line through N channels of image signal processors, and writes each channel of the image data into the buffer at a first speed, wherein N is an integer greater than 1, and the first speed is not less than N times a second speed at which each channel of the image signal processor outputs the image data. In response to reaching at least one preset i-switching node, the image processing model i uses the full computing resources of the network processing unit to read and process the i-th channel of image data currently written from the buffer, where i is an integer not greater than N, In response to reaching the subsequent i+1 switching node, the network processing unit caches the i-th path of intermediate data generated by the image processing model i, waiting for the next i switching nodes to continue reading and processing the i-th path of image data that is subsequently written. The network processing unit according to claim 16, characterized in that the network processing unit obtains the entire i-th channel of image data line by line via an image signal processor i, The image processing model i reads the previously cached intermediate data i according to multiple corresponding i switching nodes, and processes the multiple rows of the i-th image data that have been written currently, window by window, to update the intermediate data i about the i-th image data, and/or generate result data i about the i-th image data. The network processing unit according to claim 17, characterized in that the image processing model i comprises a multi-layer neural network structure and is configured as follows: In response to completing the preset number L of neural network inferences based on the multiple lines of image data currently written, the result data is output line by line to the corresponding image signal processor i at the first speed, and the multiple lines of image data that only involve the first L-1 layers of neural network inference are deleted. The network processing unit as described in any one of claims 16 to 18 is characterized in that each of the switching nodes is distributed according to a preset time interval, or is distributed according to the latency requirement of each of the image processing models, or is distributed according to the total cached data volume of each of the intermediate data. The network processing unit according to claim 19, wherein the steps of determining the switching nodes distributed according to the latency requirements and determining the storage space corresponding to each of the image processing models comprises: Dividing the entire storage space of the buffer according to the number of models that need to be processed in parallel to determine a first storage space distribution for storing the inference data of each of the image processing models; Determine the total cycle of multi-model reasoning according to the reasoning cycle of each of the image processing models, and determine the number of reasoning times of each of the image processing models within the total cycle; Determine at least one switching node for time-division multiplexing each of the image processing models from a plurality of processing nodes of each of the image processing models according to the total cycle and the number of inferences of each of the image processing models; Based on the first storage space distribution and the switching nodes of each of the image processing models, using the full computing resources of the network processing unit, multi-model reasoning is performed on the inference data samples of the multiple image processing models to determine the storage space missing from each of the image processing models; and Based on the full storage space and the storage space lacking in each of the image processing models, the first storage space distribution is optimized to determine a second storage space distribution that meets the multi-model reasoning requirements. The network processing unit according to claim 19, wherein the steps of determining the switching nodes distributed according to the total cache data volume and determining the storage space corresponding to each of the image processing models include: Dividing the entire storage space of the buffer according to the number of models that need to be processed in parallel to determine the distribution of the third storage space for storing the inference data of each of the image processing models; Determine the total cycle of multi-model reasoning according to the reasoning cycle of each of the image processing models, and determine the number of reasoning times of each of the image processing models within the total cycle; Based on the third storage space distribution and the processing nodes of the image processing models, the full computing resources of the buffer are used to perform model reasoning on the inference data samples of the image processing models to determine the storage space lacking in the processing nodes of the image processing models; In the order of the lack of storage space from small to large, dividing each of the image processing models into different numbers of subgraphs according to the processing nodes, and performing reasoning tests to determine the calculation time required for each of the image processing models; Determine the maximum number of subgraphs whose inference times and computation time meet the total performance requirements of the multi-model inference; and According to the positions of the processing nodes corresponding to the maximum number of sub-graphs, the switching nodes of the image processing models are determined respectively. The network processing unit according to any one of claims 16 to 18, characterized in that the buffer is statically divided into N buffer spaces to cache the image data currently to be processed and the intermediate data generated by previous processing. The network processing unit according to claim 16, characterized in that the network processing unit is connected to the binocular camera via two image signal processors and is configured as follows: Alternately acquiring left-eye image data and right-eye image data of the binocular camera line by line through the two image signal processors, and writing the left-eye image data and the right-eye image data into the buffer respectively; In response to reaching the first switching node, using the full computing resources of the network processing unit via the first image processing model, reading and processing the currently written left-eye image data from the buffer to generate corresponding first intermediate data; In response to reaching a second switching node, the first intermediate data is cached, and the full computing resources of the network processing unit are switched to a second image processing model, and the currently written right-eye image data is read and processed from the cache via the second image processing model to generate corresponding second intermediate data; and In response to the next first switching node, the second intermediate data is cached, the first intermediate data is read, and the full computing resources of the network processing unit are switched back to the first image processing model, and the further written left-eye image data is read and processed from the cache via the first image processing model to update the first intermediate data, and/or generate result data about the left-eye image data. The network processing unit according to claim 16 or 23, characterized in that the left-eye image data and the right-eye image data are original image data with noise, and the result data about the left-eye image data and the right-eye image data are denoised image data with noise eliminated, or The left-eye image data and the right-eye image data are original image data with mosaics, and the result data regarding the left-eye image data and the right-eye image data are restored image data with the mosaics removed. An extended reality display chip, characterized by comprising: At least two image signal processors, connected to the left camera and the right camera of the augmented reality display device respectively; and The network processing unit according to any one of claims 16 to 24 is connected to each of the image signal processors, respectively, to synchronously acquire image data output by each of the image signal processors, and process the image data in parallel. An extended reality display device, comprising: Binocular camera; and The extended reality display chip as described in claim 25, wherein the extended reality display chip is connected to the binocular camera to synchronously acquire multiple channels of image data outputted by the binocular camera and perform parallel processing on each channel of the image data. A parallel processing method for image data, characterized in that it comprises the following steps: Alternately acquiring N channels of image data line by line through N channels of image signal processors, and writing each channel of the image data into a buffer of a network processing unit at a first speed, wherein N is an integer greater than 1, and the first speed is not less than N times a second speed at which each channel of the image signal processor outputs the image data; In response to reaching at least one preset i-switching node, reading and processing the currently written i-th channel of image data from the buffer by using the image processing model i pre-trained and configured in the network processing unit and utilizing the full computing resources of the network processing unit, wherein i is an integer not greater than N; and In response to reaching the subsequent i+1 switching node, the i-th channel of intermediate data generated by the image processing model i is cached, waiting for the next i switching nodes to continue reading and processing the i-th channel of image data that is subsequently written. A computer-readable storage medium having computer instructions stored thereon, characterized in that when the computer instructions are executed by a processor, the multi-model reasoning method as described in any one of claims 1 to 13, or the parallel processing method for image data as described in claim 27 is implemented.