TWI876278B - Image processing method, neural network training method, device and medium - Google Patents

Abstract

An embodiment of the present invention provides an image processing method, a neural network training method, an apparatus and a medium, the method comprising: performing channel segmentation on a first image to obtain a second image; performing compression sampling and image reconstruction on the second image through a target neural network to obtain at least two reconstructed images corresponding to the second image; fusing the second image and the at least two reconstructed images to obtain a target image, wherein the resolution of the target image is higher than the resolution of the first image.

Description

Image processing method, neural network training method, device and medium

The present invention belongs to the field of electronic equipment technology, and specifically relates to an image processing method, a neural network training method, a device, an electronic device and a readable storage medium.

With the development of electronic technology, more and more users use electronic devices to take pictures, and the display effect of images has attracted more and more attention. However, in the process of using electronic devices to take pictures, the clarity of the images taken by the electronic devices may be low, thereby reducing the image quality and affecting the display effect of the image.

The purpose of the embodiments of the present invention is to provide an image processing method, a neural network training method, a device and a medium, which can solve the problem of low clarity and poor image quality of images taken by electronic equipment.

In a first aspect, an embodiment of the present invention provides an image processing method, the method comprising: Performing channel segmentation on a first image to obtain a second image; Performing compression sampling and image reconstruction on the second image through a target neural network to obtain at least two reconstructed images corresponding to the second image; Performing fusion processing on the second image and the at least two reconstructed images to obtain a target image, the resolution of the target image being higher than the resolution of the first image.

In a second aspect, an embodiment of the present invention provides a neural network training method, the method comprising: Obtaining at least two third images; Compressing and sampling each of the third images through a neural network to obtain at least two sensing values corresponding to the third image; Reconstructing an image based on the at least two sensing values through the neural network to obtain at least two reconstructed images, the sensing values corresponding to the reconstructed images one by one; Determining a loss function value based on the image loss between each third image and each corresponding reconstructed image; Repeatedly calculating and training the neural network according to the loss function value to obtain a target neural network.

In a third aspect, an embodiment of the present invention provides an image processing device, which includes: A segmentation module, used to perform channel segmentation on a first image to obtain a second image; A first processing module, used to perform compression sampling and image reconstruction on the second image through a target neural network to obtain at least two reconstructed images corresponding to the second image; A second processing module, used to perform fusion processing on the second image and the at least two reconstructed images to obtain a target image, wherein the resolution of the target image is higher than the resolution of the first image.

In a fourth aspect, an embodiment of the present invention provides a neural network training device, the device comprising: An acquisition module, used to acquire at least two third images; A first training module, used to compress and sample each third image through a neural network to obtain at least two sensing values corresponding to the third image; A second training module, used to reconstruct an image based on the at least two sensing values through the neural network to obtain at least two reconstructed images, wherein the sensing values correspond to the reconstructed images one by one; A third training module, used to determine a loss function value based on the image loss between each third image and each corresponding reconstructed image; The fourth training module is used to repeatedly train the neural network according to the loss function value to obtain the target neural network.

In a fifth aspect, an embodiment of the present invention provides an electronic device, which includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor. When the program or instruction is executed by the processor, the steps of the method described in the first aspect are implemented, or the steps of the method described in the second aspect are implemented.

In a sixth aspect, an embodiment of the present invention provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented, or the steps of the method described in the second aspect are implemented.

In the seventh aspect, an embodiment of the present invention provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the method described in the first aspect, or to implement the method described in the second aspect.

In an eighth aspect, an embodiment of the present invention provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the first aspect, or to implement the method described in the second aspect.

In the embodiment of the present invention, the first image is channel-divided to obtain the second image; the second image is compressed and sampled and reconstructed by the target neural network to obtain at least two reconstructed images corresponding to the second image. The second image and the at least two reconstructed images are then fused to improve the image details, achieve image reconstruction compensation, and obtain a target image with a higher resolution than the first image. In the above manner, a target image with higher image quality is output during the shooting process, and the display effect of the shot image is improved.

The following will be combined with the drawings in the embodiments of the present invention to clearly describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field are within the scope of protection of the present invention.

The terms "first", "second", etc. in the specification and patent application of the present invention are used to distinguish similar objects, rather than to describe a specific order or precedence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of the same type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and/or" in the specification and patent application indicates at least one of the connected objects, and the character "/" generally indicates that the objects associated before and after are in an "or" relationship.

The image processing method provided by the embodiment of the present invention is described in detail below through specific embodiments and application scenarios in conjunction with the accompanying drawings.

The embodiment of the present invention provides an image processing method. Please refer to Figure 1, which is a flow chart of the image processing method provided by the embodiment of the present invention. The image processing method provided by the embodiment of the present invention includes the following steps: S101, channel-divide the first image to obtain the second image.

The image processing method provided by the embodiment of the present invention can be applied to an electronic device, and the first image can be an image taken by the electronic device. It should be understood that in other embodiments, the first image can also be an image obtained by the electronic device through other channels, for example, an image downloaded from the Internet.

The first image includes three color channels, namely, R channel, G channel and B channel. In this step, the first image is channel-divided to obtain a second image, wherein the second image includes three color channels corresponding to the first image, namely, R channel, G channel and B channel. For example, if the first image is a two-dimensional image and the image dimension is 10*8, then the second image is a one-dimensional image and the image dimension is 80*1.

S102: compressing and sampling the second image and reconstructing the image through a target neural network to obtain at least two reconstructed images corresponding to the second image.

In this step, the second image can be compressed and sampled by the trained neural network, i.e., the target neural network, to obtain the sensing value corresponding to the second image, and the trained neural network can reconstruct the image based on the sensing value to obtain at least two reconstructed images. The reconstructed image carries the display content feature information of the second image, and the display content feature information represents the display details of the second image.

Optionally, the above neural network may be a neural network based on compressed sensing (CS), or a neural network based on other functions, which is not specifically limited here.

For the specific implementation of how to perform compression sampling and image reconstruction on the second image through the target neural network, please refer to the subsequent embodiments.

S103: fusing the second image and the at least two reconstructed images to obtain a target image.

In this step, after obtaining at least two reconstructed images corresponding to the second image, the second image and the at least two reconstructed images are fused to obtain a target image, wherein the resolution of the target image is higher than the resolution of the first image.

Optionally, the second image and at least two reconstructed images can be fused based on maximum pixel value and minimum pixel value, or the second image and at least two reconstructed images can be fused based on pixel weighted averaging, or the images can be fused in other ways, which are not specifically limited here.

To facilitate understanding of the technical solution provided by this embodiment, please refer to Figure 2. As shown in Figure 2, first execute step 201 to perform channel division on the first image to obtain a second image; step 202 compresses and samples the second image and reconstructs the image through the trained neural network to obtain multiple reconstructed images; step 203 fuses the second image and the multiple reconstructed images to obtain a target image with a higher resolution than the first image.

In the embodiment of the present invention, the first image is channel-divided to obtain the second image; the second image is compressed and sampled and reconstructed by the target neural network to obtain at least two reconstructed images corresponding to the second image. The second image and the at least two reconstructed images are then fused to improve the image details, achieve image reconstruction compensation, and obtain a target image with a higher resolution than the first image. In the above manner, a target image with higher image quality is output during the shooting process, and the display effect of the shot image is improved.

Optionally, the method of compressing and sampling the second image and reconstructing the image through the target neural network to obtain at least two reconstructed images corresponding to the second image includes: Compressing and sampling the second image through the target neural network to obtain at least two sensing values corresponding to the second image; Reconstructing the image based on the at least two sensing values through the target neural network to obtain at least two reconstructed images corresponding to the second image.

In this embodiment, the second image is first compressed and sampled through the target neural network to obtain at least two sensing values corresponding to the second image, wherein the above sensing values can be understood as samples after the image is compressed and sampled, and the dimension of the sensing value is lower than the image dimension of the corresponding second image.

Then, the target neural network performs image reconstruction based on at least two sensing values to obtain at least two reconstructed images corresponding to the second image, wherein the sensing values correspond to the reconstructed images one by one.

It should be understood that the specific implementation method of how to compress and sample images through a neural network is described in the subsequent embodiments. The image reconstruction process can be understood as the inverse process of compressing and sampling images through a neural network. In this embodiment, a reconstruction algorithm can be used to reconstruct the image based on the sensed value. For example, the above-mentioned reconstruction algorithm is an L0-norm algorithm or an L1-norm algorithm.

The following specifically describes how to compress and sample the second image through the target neural network: Optionally, compressing and sampling the second image through the target neural network to obtain at least two sensing values corresponding to the second image includes: In the process of compressing and sampling the second image through the target neural network, randomly selecting a compressed sensing array in the parameter table that has a mapping relationship with the image dimension of the second image; Compressing and sampling the second image through the compressed sensing array to obtain at least two sensing values corresponding to the second image.

It should be noted that the target neural network includes a parameter table, and the parameter table is used to characterize the mapping relationship between the compressed perception array and the image dimension of the second image, and the image dimension of a second image has a mapping relationship with at least one compressed perception array. Optionally, the parameter table can also characterize the dimension of the sensing value corresponding to the second image, the mapping relationship between the compressed perception array and the image dimension of the second image.

In this embodiment, during a compression sampling process, a compression perception array having a mapping relationship with the image dimension of the second image can be randomly selected from the parameter table, and the second image is compressed and sampled through the compression perception array to obtain the sensing value corresponding to the second image.

Specifically, the second image can be understood as a column of N-dimensional vectors, and the compressed perception array in the parameter table that has a mapping relationship with the image dimension of the second image can be understood as an M*N-dimensional vector, where M is less than N. The sensing value obtained by compressing and sampling the second image based on the compressed perception array is a column of M-dimensional vectors, thereby achieving compression of the second image.

It should be understood that the second image can be compressed and sampled multiple times. In each compression sampling process, different compression sensing arrays are randomly selected from the parameter table to compress and sample the second image, and different sensing values are obtained. Then, in the subsequent step of reconstructing the image based on the sensing values, multiple reconstructed images can be obtained, and the display details represented by the display content feature information included in each reconstructed image are different, thereby ensuring that the target image obtained by fusing the reconstructed images has rich display details, thereby improving the display effect of the target image.

Optionally, the second image and the at least two reconstructed images are fused to obtain a target image, including: Extracting feature points from at least two reconstructed images corresponding to the second image to obtain at least two feature point images corresponding to the second image; Performing image fusion processing on the at least two feature point images and the second image to obtain a third image.

As mentioned above, the reconstructed image includes display content feature information, and the display content feature information can represent the display details of the second image.

In this embodiment, feature point extraction can be performed on at least two reconstructed images corresponding to the second image to obtain at least two feature point images corresponding to the second image, wherein the feature point images correspond to the reconstructed images one by one, and the feature point images include image feature points of the corresponding reconstructed images. Optionally, feature point extraction can be performed on the image based on texture information of the image, or based on grayscale information of the image, or feature points of the image can be extracted by other methods, which are not specifically limited here.

After obtaining at least two feature point images, image fusion processing is performed on the at least two feature point images and the second image to obtain a target image.

In this embodiment, feature points are extracted from the reconstructed image to obtain a feature point image containing image display details, and then the second image and at least two feature point images are fused to obtain a target image with rich display details, thereby improving the resolution and display effect of the target image.

In other embodiments, after obtaining at least two reconstructed images corresponding to the second image, the second image and the at least two reconstructed images corresponding to the second image are fused to obtain a target image.

In order to facilitate understanding of the image processing method provided by the embodiment of the present invention, please refer to Figure 3. As shown in Figure 3, channel splitting is performed on the first image to obtain a second image; the second image is input into the compressed sensing neural network to obtain multiple reconstructed images corresponding to the second image; feature points of the reconstructed image are extracted, and then the multiple feature point images obtained by extracting the feature points are image-fused with the second image to obtain a target image with higher definition.

The present invention provides a neural network training method. Please refer to Figure 4, which is a flow chart of the neural network training method provided by the present invention. The neural network training method provided by the present invention includes the following steps: S401, obtaining at least two third images.

The third image may be a preset image.

S402: compress and sample each of the third images through a neural network to obtain at least two sensing values corresponding to the third image.

The above neural network can be a compression-aware neural network.

In this step, each third image is input into the compression perception neural network, and the third image is compressed and sampled to obtain at least two sensing values corresponding to the third image. For the specific implementation method, please refer to the subsequent embodiments.

S403, reconstructing an image based on the at least two sensing values through the neural network to obtain at least two reconstructed images.

In this step, after obtaining at least two sensing values, image reconstruction can be performed based on the at least two sensing values through a neural network to obtain at least two reconstructed images, wherein the sensing values correspond to the reconstructed images one by one.

Optionally, a reconstruction algorithm may be used to reconstruct the image based on the sensing values. For example, the reconstruction algorithm is an L0-norm algorithm or an L1-norm algorithm.

S404: Determine a loss function value based on the image loss between each third image and each corresponding reconstructed image.

S405, repeatedly performing calculation training on the neural network according to the loss function value to obtain a target neural network.

In this step, the loss function value can be determined according to the image loss between the third image and the corresponding reconstructed image, and the neural network can be adjusted based on the loss function value to achieve repeated calculation training of the neural network.

Optionally, when the above loss function value is less than or equal to a preset threshold, it means that the image loss between the third image and the corresponding reconstructed image is within an allowable range, and the neural network training is completed, and a trained target neural network is obtained.

Optionally, the method of compressing and sampling each third image through a neural network to obtain at least two sensing values corresponding to the third image includes: randomly generating a compressed sensing array during a process of compressing and sampling the third image through a neural network; compressing and sampling the third image through the compressed sensing array to obtain sensing values corresponding to the third image.

In this embodiment, during a compression sampling process, a compression sensing matrix can be randomly generated, and the third image is compressed and sampled by the compression sensing array to obtain the sensing value corresponding to the third image.

Exemplarily, the dimension of the third image is N, the number of randomly generated compressed perception matrices is M, and the dimension of the compressed perception matrix is M*N. Then, the fourth image is compressed and sampled through the compressed perception array, and the dimension of the generated sensing value is M.

It should be understood that in the process of training the neural network, a parameter table can also be generated based on the dimension of the third image in each compression sampling process, the dimension of the randomly generated compressed perception matrix, and the dimension of the sensing value, wherein the above parameter table is used to characterize the mapping relationship between the dimension of the sensing value corresponding to the third image, the compressed perception array, and the image dimension of the third image.

To facilitate understanding of the overall scheme of the neural network training method provided in this embodiment, please refer to Figure 5. As shown in Figure 5, step 501 compresses and samples each third image through the neural network to obtain at least two sensing values corresponding to each third image; step 502 generates a parameter table based on the dimension of the third image in each compression sampling process, the dimension of the randomly generated compressed perception matrix, and the dimension of the sensing value; step 503 reconstructs the image based on at least two sensing values through the neural network to obtain at least two reconstructed images; step 504 repeatedly calculates and trains the neural network according to the image loss between the third image and each corresponding reconstructed image.

The image processing device provided by the embodiment of the present invention is described in detail below through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

As shown in FIG6 , the image processing device 600 includes: A segmentation module 601, used to perform channel segmentation on the first image to obtain a second image; A first processing module 602, used to perform compression sampling and image reconstruction on the second image through a target neural network to obtain at least two reconstructed images corresponding to the second image; A second processing module 603, used to perform fusion processing on the second image and the at least two reconstructed images to obtain a target image, the resolution of which is higher than the resolution of the first image.

Optionally, the first processing module 602 is specifically used to: compress and sample the second image through the target neural network to obtain at least two sensing values corresponding to the second image; reconstruct the image based on the at least two sensing values through the target neural network to obtain at least two reconstructed images corresponding to the second image, and the sensing values correspond to the reconstructed images one by one.

Optionally, the neural network includes a parameter table, which is used to characterize the mapping relationship between the compressed perception array and the image dimension of the second image; The first processing module 602 is also specifically used to: In the process of performing a compression sampling on the second image through the target neural network, randomly select a compressed perception array in the parameter table that has a mapping relationship with the image dimension of the second image; Compress the second image through the compressed perception array to obtain at least two sensing values corresponding to the second image.

Optionally, the second processing module 603 is specifically used to: Extract feature points from at least two reconstructed images corresponding to the second image to obtain at least two feature point images corresponding to the second image, and the feature point images correspond one-to-one to the reconstructed image; Perform image fusion processing on the at least two feature point images and the second image to obtain a target image.

In the embodiment of the present invention, the first image is channel-divided to obtain the second image, and the second image is compressed and sampled and reconstructed by the target neural network to obtain at least two reconstructed images corresponding to the second image. The second image and the at least two reconstructed images are then fused to improve the image details, achieve image reconstruction compensation, and obtain a target image with a higher resolution than the first image. In the above manner, a target image with higher image quality is output during the shooting process, and the display effect of the shot image is improved.

The neural network training device provided by the embodiment of the present invention is described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.

As shown in FIG7 , the neural network training device 700 includes: An acquisition module 701, used to acquire at least two third images; A first training module 702, used to compress and sample each third image through a neural network to obtain at least two sensing values corresponding to the third image; A second training module 703, used to reconstruct an image based on the at least two sensing values through the neural network to obtain at least two reconstructed images, wherein the sensing values correspond to the reconstructed images one by one; A third training module 704, used to determine a loss function value based on the image loss between each third image and each corresponding reconstructed image; The fourth training module 705 is used to repeatedly train the neural network according to the loss function value to obtain the target neural network.

Optionally, the first training module 702 is specifically used to: Randomly generate a compressed perception array during a process of compressing and sampling the third image through a neural network; Compress and sample the third image through the compressed perception array to obtain a sensing value corresponding to the third image.

The image processing device or neural network training device in the embodiment of the present invention can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices except the terminal. Exemplarily, the electronic device may be a mobile phone, a tablet computer, a laptop computer, a handheld computer, a vehicle-mounted electronic device, a mobile Internet device (MID), an augmented reality (AR)/virtual reality (VR) device, a robot, a wearable device, an ultra-mobile personal computer (UMPC), a netbook or a personal digital assistant (PDA), etc. It may also be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine or a self-service machine, etc., and the embodiments of the present invention are not specifically limited.

The image processing device or neural network training device in the embodiment of the present invention may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present invention.

The image processing device provided in the embodiment of the present invention can implement each process implemented in the method embodiment of FIG. 1 , and will not be described again here to avoid repetition.

The neural network training device provided in the embodiment of the present invention can implement each process implemented in the method embodiment of FIG. 4 , and will not be described again here to avoid repetition.

Optionally, as shown in FIG8 , the embodiment of the present invention further provides an electronic device 800, including a processor 801, a memory 802, and a program or instruction stored in the memory 802 and executable on the processor 801. When the program or instruction is executed by the processor 801, the various processes of the above-mentioned image processing method embodiment are implemented, or the various processes of the above-mentioned neural network training method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

It should be noted that the electronic devices in the embodiments of the present invention include the mobile electronic devices and non-mobile electronic devices mentioned above.

FIG9 is a schematic diagram of the hardware structure of an electronic device implementing an embodiment of the present invention.

The electronic device 900 includes but is not limited to: a radio frequency unit 901, a network module 902, an audio output unit 903, an input unit 904, a sensor 905, a display unit 906, a user input unit 907, an interface unit 908, a memory 909, and a processor 910 and other components.

Those skilled in the art can understand that the electronic device 900 can also include a power source (such as a battery) for supplying power to various components. The power source can be logically connected to the processor 910 through a power management system, so that the power management system can manage charging, discharging, and power consumption. The electronic device structure shown in FIG9 does not constitute a limitation on the electronic device. The electronic device can include more or fewer components than shown, or combine certain components, or arrange components differently, which will not be elaborated here.

The processor 910 is also used to perform channel segmentation on the first image to obtain a second image; perform compression sampling and image reconstruction on the second image through the target neural network to obtain at least two reconstructed images corresponding to the second image; perform fusion processing on the second image and at least two reconstructed images corresponding to the second image to obtain a target image.

Optionally, the processor 910 is further used to compress and sample the second image through the target neural network to obtain at least two sensing values corresponding to the second image; and reconstruct the image based on the at least two sensing values through the target neural network to obtain at least two reconstructed images corresponding to the second image.

Optionally, the processor 910 is further used to randomly select a compressed perception array in the parameter table that has a mapping relationship with the image dimension of the second image during a process of performing a compression sampling on the second image through the target neural network; The second image is compressed and sampled through the compressed perception array to obtain at least two sensing values corresponding to the second image.

Optionally, the processor 910 is further used to extract feature points from at least two reconstructed images corresponding to the second image to obtain at least two feature point images corresponding to the second image; Perform image fusion processing on the at least two feature point images and the second image to obtain a third image.

Optionally, the input unit 904 is also used to obtain at least two third images; The processor 910 is also used to compress and sample each of the third images through a neural network to obtain at least two sensing values corresponding to the third image; Reconstruct the image based on the at least two sensing values through the neural network to obtain at least two reconstructed images, and the sensing values correspond to the reconstructed images one by one; Determine the loss function value based on the image loss between each third image and each corresponding reconstructed image; Repeatedly train the neural network according to the loss function value to obtain a target neural network.

Optionally, the processor 910 is also used to randomly generate a compressed perception array in the process of performing a compression sampling on the third image through the neural network; The third image is compressed and sampled through the compressed perception array to obtain the sensing value corresponding to the third image. It should be understood that in the embodiment of the present invention, the input unit 904 may include a graphics processor (Graphics Processing Unit, GPU) 9041 and a microphone 9042, and the graphics processor 9041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 906 may include a display panel 9061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, etc. The user input unit 907 includes a touch panel 9071 and at least one of other input devices 9072. The touch panel 9071 is also called a touch screen. The touch panel 9071 may include two parts: a touch detection device and a touch controller. Other input devices 9072 may include but are not limited to a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

The memory 909 can be used to store software programs and various data. The memory 909 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instruction required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 909 may include a volatile memory or a non-volatile memory, or the memory 909 may include both a volatile memory and a non-volatile memory. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable ROM (PROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM) or a flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (Direct Rambus RAM, DRRAM). The memory 909 in embodiments of the present invention includes but is not limited to these and any other suitable types of memory.

The processor 910 may include one or more processing units; optionally, the processor 910 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to the operating system, user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the above-mentioned modem processor may not be integrated into the processor 910.

The embodiment of the present invention also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned image processing method embodiment are implemented, or the various processes of the above-mentioned neural network training method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

The embodiment of the present invention further provides a chip, which includes a processor and a communication interface, the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned image processing method embodiment, or to implement the various processes of the above-mentioned neural network training method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

It should be understood that the chip mentioned in the embodiments of the present invention can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

An embodiment of the present invention provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned image processing method embodiment, or to implement the various processes of the above-mentioned neural network training method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present invention is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in a reverse order according to the functions to which they belong. For example, the described method may be performed in an order different from that described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

Through the description of the above implementation, the technical personnel in this field can clearly understand that the above implementation method can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation. Based on this understanding, the technical solution of the present invention, or the part that contributes to the existing technology, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM/RAM, disk, optical disk), including a number of instructions to enable a terminal (which can be a mobile phone, computer, server, or network equipment, etc.) to execute the methods described in each embodiment of the present invention.

The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are only illustrative and not restrictive. Under the inspiration of the present invention, ordinary technical personnel in this field can make many forms without departing from the scope protected by the purpose of the present invention and the scope of the patent application, all of which are within the protection of the present invention.

S101~S103, 201~203, S401~S405, 501~504: Steps 600: Image processing device 601: Segmentation module 602: First processing module 603: Second processing module 700: Neural network training device 701: Acquisition module 702: First training module 703: Second training module 704: Third training module 705: Fourth training module 800: Electronic equipment 801: Processor 802: Memory 900: Electronic equipment 901: RF unit 902: Network module 903: Audio output unit 904: Input unit 9041: Graphics processor 9042: Microphone 905: Sensor 906: Display unit 9061: Display panel 907: User input unit 9071: Touch panel 9072: Input device 908: Interface unit 909: Memory 910: Processor

FIG. 1 is a flow chart of an image processing method provided by an embodiment of the present invention; FIG. 2 is one of the application flow charts of the image processing method provided by an embodiment of the present invention; FIG. 3 is one of the application flow charts of the image processing method provided by an embodiment of the present invention; FIG. 4 is a flow chart of a neural network training method provided by an embodiment of the present invention; FIG. 5 is an application flow chart of a neural network training method provided by an embodiment of the present invention; FIG. 6 is a structural diagram of an image processing device provided by an embodiment of the present invention; FIG. 7 is a structural diagram of a neural network training device provided by an embodiment of the present invention; FIG. 8 is a structural diagram of an electronic device provided by an embodiment of the present invention; FIG. 9 is a hardware structural diagram of an electronic device provided by an embodiment of the present invention.

S101~S103: Steps

Claims (8)

An image processing method, comprising: Performing channel division on a first image to obtain a second image, wherein the second image includes three color channels corresponding to the first image, and the three color channels include an R channel, a G channel, and a B channel; Performing compression sampling and image reconstruction on the second image through a target neural network to obtain at least two reconstructed images corresponding to the second image; Performing fusion processing on the second image and the at least two reconstructed images to obtain a target image, and the resolution of the target image is higher than the resolution of the first image; Wherein, performing compression sampling and image reconstruction on the second image through a target neural network to obtain at least two reconstructed images corresponding to the second image includes: performing compression sampling on the second image through a target neural network to obtain at least two sensing values corresponding to the second image; The target neural network performs image reconstruction based on the at least two sensing values to obtain at least two reconstructed images corresponding to the second image, and the sensing values correspond to the reconstructed images one by one. The image processing method as described in claim 1, wherein the neural network includes a parameter table, and the parameter table is used to characterize the mapping relationship between the compressed perception array and the image dimension of the second image; The second image is compressed and sampled by the target neural network to obtain at least two sensing values corresponding to the second image, including: In the process of performing a compression sampling on the second image by the target neural network, a compressed perception array having a mapping relationship with the image dimension of the second image is randomly selected from the parameter table; The second image is compressed and sampled by the compressed perception array to obtain at least two sensing values corresponding to the second image. The image processing method as described in claim 1, wherein the second image and the at least two reconstructed images are fused to obtain a target image, comprising: Extracting feature points from at least two reconstructed images corresponding to the second image to obtain at least two feature point images corresponding to the second image, wherein the feature point images correspond one-to-one to the reconstructed image; Performing image fusion processing on the at least two feature point images and the second image to obtain a target image. A neural network training method, applied to the method as described in any one of claim items 1-3, comprising: Obtaining at least two third images; Compressing and sampling each of the third images through a neural network to obtain at least two sensing values corresponding to the third image; Reconstructing images based on the at least two sensing values through the neural network to obtain at least two reconstructed images, wherein the sensing values correspond to the reconstructed images one by one; Determining a loss function value based on the image loss between each third image and each corresponding reconstructed image; Repeatedly calculating and training the neural network according to the loss function value to obtain a target neural network; The method of compressing and sampling each third image through a neural network to obtain at least two sensing values corresponding to the third image includes: randomly generating a compressed sensing array in the process of compressing and sampling the third image through a neural network; and compressing and sampling the third image through the compressed sensing array to obtain the sensing value corresponding to the third image. An image processing device includes: A segmentation module for performing channel segmentation on a first image to obtain a second image, wherein the second image includes three color channels corresponding to the first image, and the three color channels include an R channel, a G channel, and a B channel; A first processing module for compressing and sampling the second image and reconstructing the image through a target neural network to obtain at least two reconstructed images corresponding to the second image; A second processing module for fusing the second image and the at least two reconstructed images to obtain a target image, wherein the resolution of the target image is higher than that of the first image; The first processing module is specifically used for: Compressing and sampling the second image through a target neural network to obtain at least two sensing values corresponding to the second image; The target neural network performs image reconstruction based on the at least two sensing values to obtain at least two reconstructed images corresponding to the second image, and the sensing values correspond to the reconstructed images one by one. A neural network training device includes: An acquisition module for acquiring at least two third images; A first training module for compressing and sampling each third image through a neural network to obtain at least two sensing values corresponding to the third image; A second training module for reconstructing an image based on the at least two sensing values through the neural network to obtain at least two reconstructed images, wherein the sensing values correspond to the reconstructed images one by one; A third training module for determining a loss function value based on the image loss between each third image and each corresponding reconstructed image; A fourth training module for repeatedly calculating and training the neural network according to the loss function value to obtain a target neural network; The first training module is specifically used to: Randomly generate a compressed perception array during a process of compressing and sampling the third image through a neural network; Compress and sample the third image through the compressed perception array to obtain a sensing value corresponding to the third image. An electronic device comprises a processor, a memory and a program or instruction stored in the memory and executable on the processor, wherein when the program or instruction is executed by the processor, the steps of the image processing method described in any one of claim items 1 to 3 are implemented, or the steps of the neural network training method described in claim item 4 are implemented. A readable storage medium storing a program or instruction, wherein the program or instruction, when executed by a processor, implements the steps of the image processing method as described in any one of claim 1 to 3, or implements the steps of the neural network training method as described in claim 4.