CN113284055A - Image processing method and device - Google Patents

Abstract

This application provides an image processing method and device in the field of artificial intelligence. By extracting features from multiple channels in an input image, up-sampling guidance is performed on bilateral grid data, so that each channel of the image can be implemented. Better image enhancement effects, such as dehazing effects, can achieve lightweight image dehazing and improve user experience. The method includes: acquiring an input image, where the input image includes information of multiple channels; extracting features from the information of multiple channels of the input image to obtain multiple guide images; acquiring bilateral grid data corresponding to the input image, the bilateral grid The data includes the data formed by the information of the luminance dimension arranged in the spatial dimension, and the resolution of the bilateral grid data is lower than that of the input image; each guide image in the multiple guide images is used as a guide condition, and the bilateral grid data is The grid data is up-sampled to obtain multiple feature maps; multiple feature maps are fused to obtain the output image.

Description

Method and device for image processing

technical field

The present application relates to the field of artificial intelligence, and in particular, to an image processing method and apparatus.

Background technique

The dehazing method has undergone roughly three stages of development: the early mainstream methods are traditional methods, which model and estimate light propagation, but the artificially designed models and image prior knowledge cannot be accurately applied to complex and different types of real images. . After a major breakthrough in the application of convolutional neural networks to a large number of vision tasks and achieved results, learning-based methods have become mainstream, replacing some modules in traditional methods with learnable network layers. The most recent method discards artificially designed modules such as light estimation, and uses an end-to-end network to solve the task of dehazing, that is, the entire process of image transformation is learned by the network, and the algorithm is completely driven by data. However, the end-to-end network requires a lot of computation and cannot process images in real time. Therefore, how to achieve a lightweight and better image enhancement effect has become an urgent problem to be solved.

SUMMARY OF THE INVENTION

The present application provides an image processing method and device. By extracting features from multiple channels in an input image, up-sampling guidance is performed on bilateral grid data, so that better images can be achieved in each channel of the image. Enhanced effects, such as dehazing, can achieve lightweight image dehazing to improve user experience.

In view of this, in a first aspect, the present application provides an image processing method, including: acquiring an input image, the input image including information of multiple channels; extracting features from the information of multiple channels of the input image, respectively, to obtain multiple Guide map, the multiple guide maps are in one-to-one correspondence with multiple channels, that is, each channel has a corresponding guide map; obtain the bilateral grid data corresponding to the input image, and the bilateral grid data includes the brightness arranged in the spatial dimension The data formed by the information of the dimension, the information of the brightness dimension is obtained according to the features extracted from the input image, the resolution of the bilateral grid data is lower than that of the input image, and the spatial dimension is the preset space or determined according to the input image space; using each guide map in the multiple guide maps as the guide condition, up-sampling the bilateral grid data to obtain multiple feature maps, wherein each guide map can be used to guide the data from the bilateral grid data. The brightness dimension of , selects the information corresponding to the corresponding channel for up-sampling; fuses multiple feature maps to obtain the output image.

Therefore, in the embodiment of the present application, features can be extracted from the dimension of each channel of the input image as a guide map, and the bilateral grid data of the input image can be upsampled, thereby smoothing the noise in the input image and improving the quality of the image. It can avoid the loss of information in each channel, and further improve the clarity of the image. In addition, the present application implements image enhancement by upsampling the low-resolution bilateral grid data. Since the resolution of the bilateral grid data is lower, the amount of computation consumed is less, thereby realizing lightweight image enhancement. . For example, when the image needs to be dehazed, the method provided by the embodiment of the present application can be used to extract features from each channel of the image, so that more detailed information of each channel in the input image can be retained in the final output image. Lightweight to achieve dehazing effect and improve user experience.

In a possible implementation manner, the aforementioned up-sampling of bilateral grid data using each guide map of the plurality of guide maps as a guide condition may include: using the first guide map as a guide condition, The grid data is up-sampled to obtain up-sampling features, and the first guide map is any one of multiple guide maps; the up-sampling features and the input image are fused to obtain a first feature map, and the first feature map is included in the multiple feature maps. .

In the embodiment of the present application, the guide map can be used as a guide condition to upsample the bilateral grid data, so that when performing upsampling, the characteristics of each channel in the input image can be referred to, and under the guidance of the characteristics of each channel, the realization of Better upsampling effect, so that the features obtained by upsampling can more accurately describe the details in the input image, and smooth the noise in the input image to achieve the denoising effect.

In a possible implementation manner, the above-mentioned fusion of the up-sampling feature and the input image to obtain the first feature map may include: compressing the up-sampling feature to obtain the compressed feature, and the number of channels of the compressed feature is less than that of the up-sampling feature. Number of channels; perform an element-wise product on the compressed feature and the input image, that is, multiply the value of each pixel in the compressed feature with the value of the corresponding pixel in the input image to obtain the first feature map.

In the embodiments of the present application, compressed features with fewer channels can be obtained by compressing the up-sampling features, thereby reducing the amount of calculation in the subsequent fusion of input images and features, helping to achieve lightweight image enhancement, and facilitating application In a variety of devices, improve the generalization ability.

In a possible implementation manner, the above-mentioned obtaining the bilateral grid data corresponding to the input image to obtain the output image may include: down-sampling the input image to obtain the down-sampled image; extracting features from the down-sampled image to obtain the down-sampled image Sampling features, and then obtain bilateral grid data according to the down-sampling features.

Therefore, in the embodiments of the present application, the input image can be down-sampled to obtain a low-resolution image, and feature extraction is performed on the low-resolution image. Upsampling is performed under the guidance of , so as to achieve the effect of smoothing noise.

In a possible implementation manner, the above-mentioned fusion of multiple feature maps to obtain a spliced image includes: splicing multiple feature maps to obtain a spliced image; performing feature extraction on the spliced image at least once to obtain at least one first feature; merging At least one first feature and an input image to obtain an output image.

Therefore, in the embodiments of the present application, the splicing of the channel dimensions can be realized by splicing, so as to obtain output images of multiple channels.

In a possible implementation manner, the above-mentioned splicing of multiple feature maps may include: splicing multiple feature maps and an input image to obtain a spliced image.

In the embodiment of the present application, when splicing feature maps, the input image can be integrated, so that the detailed information of the spliced image can be supplemented by the information included in the input image, so as to avoid the loss of information in the input image and improve the clarity of the image.

In a second aspect, the present application provides a method for training a neural network, including: acquiring a training set, where the training set includes multiple image samples and a ground-truth image corresponding to each image sample, and each image sample includes information of multiple channels; using The training set performs at least one iterative training on the neural network to obtain a trained neural network; wherein, in any iterative training process, the neural network extracts features from the information of multiple channels of the input image to obtain multiple guide maps, The multiple guide maps are in one-to-one correspondence with the multiple channels, that is, each channel corresponds to one guide map, and the bilateral grid data corresponding to the input image is obtained. Each guide map in the multiple guide maps is used as a guide condition. Upsampling the grid data to obtain multiple feature maps, fuse multiple feature maps to obtain the output image, update the neural network according to the true value image corresponding to the output image and the input image, and obtain the current updated neural network, bilateral grid data It includes the data formed by the information of the brightness dimension arranged in the preset space, the information of the brightness dimension is obtained according to the features extracted from the input image, and the resolution of the bilateral grid data is lower than that of the input image.

Therefore, in the method provided in this application, when training the neural network, features can be extracted from the dimension of each channel of the input image as a guide map, and the bilateral grid data of the input image can be upsampled, thereby smoothing the input The noise in the image can improve the clarity of the image, and can avoid the loss of information in each channel, and further improve the clarity of the image output by the neural network. In addition, the present application implements image enhancement by upsampling the low-resolution bilateral grid data. Since the resolution of the bilateral grid data is lower, the amount of computation consumed is less, thereby realizing lightweight image enhancement. . For example, when the image needs to be dehazed, the neural network obtained by training the method provided in the embodiment of the present application can be used to extract features from each channel of the image, so that the final output image can retain more of the various channels in the input image. The detailed information of the channel, and realize the dehazing effect, improve the user experience.

In a possible implementation manner, the aforementioned up-sampling of bilateral grid data using each guide map of the plurality of guide maps as a guide condition may include: using the first guide map as a guide condition, The grid data is up-sampled to obtain up-sampling features, and the first guide map is any one of multiple guide maps; the up-sampling features and the input image are fused to obtain a first feature map, and the first feature map is included in the multiple feature maps. .

In the embodiment of the present application, the guide map can be used as a guide condition to upsample the bilateral grid data, so that when performing upsampling, the characteristics of each channel in the input image can be referred to, and under the guidance of the characteristics of each channel, the realization of Better upsampling effect, so that the features obtained by upsampling can more accurately describe the details in the input image, and smooth the noise in the input image to achieve the denoising effect.

In a possible implementation manner, the above-mentioned fusion of the upsampling feature and the input image may include: compressing the upsampling feature to obtain a compressed feature, and the number of channels of the compressed feature is less than the number of channels of the upsampling feature; Item-wise product with the input image to get the first feature map.

In the embodiments of the present application, compressed features with fewer channels can be obtained by compressing the up-sampling features, thereby reducing the amount of calculation in the subsequent fusion of input images and features, helping to achieve lightweight image enhancement, and facilitating application In a variety of devices, improve the generalization ability.

In a possible implementation manner, obtaining the bilateral grid data corresponding to the input image above may include: down-sampling the input image to obtain a down-sampled image; extracting features from the down-sampled image to obtain down-sampling features, bilateral The grid data includes downsampled features.

Therefore, in the embodiments of the present application, the input image can be down-sampled to obtain a low-resolution image, and feature extraction is performed on the low-resolution image. Upsampling is performed under the guidance of , so as to achieve the effect of smoothing noise.

In a possible implementation manner, the above-mentioned fusing of multiple feature maps includes: splicing multiple feature maps to obtain a spliced image; performing feature extraction on the spliced image at least once to obtain at least one first feature; fusing at least one first feature Features and input image, get output image.

Therefore, in the embodiments of the present application, the splicing of the channel dimensions can be realized by splicing, so as to obtain output images of multiple channels.

In a possible implementation manner, the above-mentioned splicing of multiple feature maps may include: splicing multiple feature maps and an input image to obtain a spliced image.

In the embodiment of the present application, when splicing feature maps, the input image can be integrated, so that the detailed information of the spliced image can be supplemented by the information included in the input image, so as to avoid the loss of information in the input image and improve the clarity of the image.

In a third aspect, the present application provides a neural network, which may include: a bilateral grid generation network, a guide map generation network, a feature reconstruction network, an image reconstruction network, and the like.

Among them, the bilateral grid generation network can be used to: downsample the full-resolution input image to obtain a low-resolution image, and then generate bilateral grid data based on the down-sampled image, where the bilateral grid data includes spatial dimensions and The luminance-related information forms data resulting in at least three dimensions.

The guide map generation network is used to extract features based on each channel of the full-resolution input image, and obtain a guide map corresponding to each channel, that is, each channel corresponds to a guide map.

The feature reconstruction network is used to: for each channel, under the guidance of the corresponding guide map, upsample the bilateral grid to obtain the feature map corresponding to each channel.

The image reconstruction network is used to fuse the feature maps corresponding to each channel, and fuse the fused features with the input image to obtain the output image.

Furthermore, it should be understood that the neural network may be used to perform the method steps of the aforementioned first aspect or any of the alternative embodiments of the first aspect.

In a fourth aspect, an embodiment of the present application provides an image processing apparatus, and the image processing apparatus has the function of implementing the image processing method of the first aspect. This function can be implemented by hardware or by executing corresponding software by hardware. The hardware or software includes one or more modules corresponding to the above functions.

In a fifth aspect, an embodiment of the present application provides a training device, and the training device has the function of implementing the neural network training method of the second aspect. This function can be implemented by hardware or by executing corresponding software by hardware. The hardware or software includes one or more modules corresponding to the above functions.

In a sixth aspect, an embodiment of the present application provides an image processing apparatus, including: a processor and a memory, wherein the processor and the memory are interconnected through a line, and the processor invokes program codes in the memory to execute any one of the first aspects above Processing-related functions in the shown method for image processing. Optionally, the image processing device may be a chip.

In a seventh aspect, an embodiment of the present application provides a training device, including: a processor and a memory, wherein the processor and the memory are interconnected through a line, and the processor invokes program codes in the memory to execute any of the above-mentioned third aspects. processing-related functions in the neural network training method shown. Alternatively, the training device may be a chip.

In an eighth aspect, the embodiments of the present application provide an image processing device, which may also be referred to as a digital processing chip or a chip. The chip includes a processing unit and a communication interface. The processing unit obtains program instructions through the communication interface, and the program instructions are The processing unit executes, and the processing unit is configured to execute the processing-related functions in the first aspect or any optional implementation manner of the first aspect.

In a ninth aspect, an embodiment of the present application provides a training device. The training device may also be called a digital processing chip or a chip. The chip includes a processing unit and a communication interface. The processing unit obtains program instructions through the communication interface, and the program instructions are processed by the processing unit. For execution, the processing unit is configured to perform the processing-related functions in the second aspect or any optional implementation manner of the second aspect.

In a tenth aspect, an embodiment of the present application provides a computer-readable storage medium, including instructions, which, when executed on a computer, cause the computer to execute the method in any optional implementation manner of the first aspect or the second aspect. .

In an eleventh aspect, the embodiments of the present application provide a computer program product including instructions, which, when run on a computer, cause the computer to execute the method in any optional implementation manner of the first aspect or the second aspect.

Description of drawings

Fig. 1 is a schematic diagram of a main frame of artificial intelligence applied by the application;

2 is a schematic structural diagram of a convolutional neural network provided by an embodiment of the present application;

3 is a schematic diagram of a system architecture provided by an embodiment of the present application;

FIG. 4 is a schematic diagram of another system architecture provided by an embodiment of the present application;

FIG. 5 is a schematic diagram of an application scenario provided by an embodiment of the present application;

FIG. 6 is a schematic diagram of another application scenario provided by an embodiment of the present application;

7 is a schematic flowchart of an image processing method provided by an embodiment of the present application;

FIG. 8 is a schematic flowchart of another image processing method provided by an embodiment of the present application;

9 is a schematic flowchart of generating bilateral grid data according to an embodiment of the present application;

FIG. 10 is a schematic flowchart of generating a guide map according to an embodiment of the present application;

FIG. 11 is another schematic flowchart of generating a guide map according to an embodiment of the present application;

12 is a schematic flowchart of feature reconstruction of one of the channels provided by an embodiment of the present application;

13 is another schematic flowchart of feature reconstruction of one of the channels provided by an embodiment of the present application;

14 is a schematic flowchart of an image reconstruction provided by an embodiment of the present application;

15 is a schematic structural diagram of a neural network provided by an embodiment of the application;

16 is a schematic flowchart of a neural network training method provided by the application;

17 is a schematic diagram of an image enhancement effect provided by the application;

18A is a schematic diagram of an image enhancement effect provided by the application;

18B is a schematic diagram of another image enhancement effect provided by the application;

18C is a schematic diagram of another image enhancement effect provided by the application;

19 is a schematic structural diagram of an image processing apparatus provided by the application;

20 is a schematic structural diagram of a neural network training provided by the application;

21 is a schematic structural diagram of another image processing apparatus provided by the application;

22 is a schematic structural diagram of another neural network training provided by the application;

FIG. 23 is a schematic structural diagram of a chip provided by this application.

Detailed ways

The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

First, the overall workflow of the artificial intelligence system will be described. Please refer to Figure 1. Figure 1 shows a schematic structural diagram of the main frame of artificial intelligence. The above-mentioned artificial intelligence theme framework is explained in two dimensions (vertical axis). Among them, the "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it can be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, intelligent execution and output. In this process, data has gone through the process of "data-information-knowledge-wisdom". The "IT value chain" reflects the value brought by artificial intelligence to the information technology industry from the underlying infrastructure of human intelligence, information (providing and processing technology implementation) to the industrial ecological process of the system.

(1) Infrastructure

The infrastructure provides computing power support for artificial intelligence systems, realizes communication with the outside world, and supports through the basic platform. Communicate with the outside through sensors; computing power is provided by smart chips, such as central processing unit (CPU), network processing unit (NPU), graphics processing unit (English: graphics processing unit, GPU), dedicated Integrated circuit (application specific integrated circuit, ASIC) or field programmable gate array (field programmable gate array, FPGA) and other hardware acceleration chips) provided; the basic platform includes distributed computing framework and network and other related platform guarantees and supports, which may include Cloud storage and computing, interconnection networks, etc. For example, sensors communicate with external parties to obtain data, and these data are provided to the intelligent chips in the distributed computing system provided by the basic platform for calculation.

(2) Data

The data on the upper layer of the infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, and text, as well as IoT data from traditional devices, including business data from existing systems and sensory data such as force, displacement, liquid level, temperature, and humidity.

(3) Data processing

Data processing usually includes data training, machine learning, deep learning, search, reasoning, decision-making, etc.

Among them, machine learning and deep learning can perform symbolic and formalized intelligent information modeling, extraction, preprocessing, training, etc. on data.

Reasoning refers to the process of simulating human's intelligent reasoning method in a computer or intelligent system, using formalized information to carry out machine thinking and solving problems according to the reasoning control strategy, and the typical function is search and matching.

Decision-making refers to the process of making decisions after intelligent information is reasoned, usually providing functions such as classification, sorting, and prediction.

(4) General ability

After the above-mentioned data processing, some general capabilities can be formed based on the results of data processing, such as algorithms or a general system, such as translation, text analysis, computer vision processing, speech recognition, image identification, etc.

(5) Smart products and industry applications

Intelligent products and industry applications refer to the products and applications of artificial intelligence systems in various fields. They are the encapsulation of the overall solution of artificial intelligence, and the productization of intelligent information decision-making to achieve landing applications. Its application areas mainly include: intelligent terminals, intelligent transportation, Smart healthcare, autonomous driving, safe city, etc.

The embodiments of the present application involve a large number of related applications of neural networks. In order to better understand the solutions of the embodiments of the present application, the related terms and concepts of the neural networks that may be involved in the embodiments of the present application are first introduced below.

(1) Neural network

A neural network can be composed of neural units, and a neural unit can refer to an operation unit that takes xs and intercept 1 as inputs, and the output of the operation unit can be shown in formula (1-1):

Among them, s=1, 2,...n, n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is an activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into an output signal. The output signal of the activation function can be used as the input of the next convolutional layer, and the activation function can be a sigmoid function. A neural network is a network formed by connecting a plurality of the above single neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected with the local receptive field of the previous layer to extract the features of the local receptive field, and the local receptive field can be an area composed of several neural units.

(2) Deep neural network

A deep neural network (DNN), also known as a multi-layer neural network, can be understood as a neural network with multiple intermediate layers. The DNN is divided according to the position of different layers. The neural network inside the DNN can be divided into three categories: input layer, intermediate layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the middle layers are all intermediate layers, or hidden layers. The layers are fully connected, that is, any neuron in the i-th layer must be connected to any neuron in the i+1-th layer.

其中，

是输入向量，

是输出向量，

是偏移向量或者称为偏置参数，w是权重矩阵(也称系数)，α()是激活函数。每一层仅仅是对输入向量

经过如此简单的操作得到输出向量

由于DNN层数多，系数W和偏移向量

的数量也比较多。这些参数在DNN中的定义如下所述：以系数w为例：假设在一个三层的DNN中，第二层的第4个神经元到第三层的第2个神经元的线性系数定义为

上标3代表系数W所在的层数，而下标对应的是输出的第三层索引2和输入的第二层索引4。Although DNN looks complicated, each layer can be expressed as a linear relational expression:

in,

is the input vector,

is the output vector,

is the offset vector or bias parameter, w is the weight matrix (also known as the coefficient), and α() is the activation function. Each layer is just an input vector

After such a simple operation to get the output vector

Due to the large number of DNN layers, the coefficient W and offset vector

The number is also higher. These parameters are defined in the DNN as follows: Take the coefficient w as an example: Suppose that in a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as

The superscript 3 represents the number of layers where the coefficient W is located, and the subscript corresponds to the output third layer index 2 and the input second layer index 4.

To sum up, the coefficient from the kth neuron in the L-1 layer to the jth neuron in the Lth layer is defined as

It should be noted that the input layer does not have a W parameter. In a deep neural network, more intermediate layers allow the network to better capture the complexities of the real world. In theory, a model with more parameters is more complex and has a larger "capacity", which means that it can complete more complex learning tasks. Training the deep neural network is the process of learning the weight matrix, and its ultimate goal is to obtain the weight matrix of all layers of the trained deep neural network (the weight matrix formed by the vectors W of many layers).

(3) Convolutional Neural Network

Convolutional neural network (CNN) is a deep neural network with a convolutional structure. A convolutional neural network consists of a feature extractor consisting of convolutional layers and subsampling layers, which can be viewed as a filter. The convolutional layer refers to the neuron layer in the convolutional neural network that convolves the input signal. In a convolutional layer of a convolutional neural network, a neuron can only be connected to some of its neighbors. A convolutional layer usually contains several feature planes, and each feature plane can be composed of some neural units arranged in a rectangle. Neural units in the same feature plane share weights, and the shared weights here are convolution kernels. Shared weights can be understood as the way to extract image information is independent of location. The convolution kernel can be initialized in the form of a matrix of random size, and the convolution kernel can obtain reasonable weights by learning during the training process of the convolutional neural network. In addition, the immediate benefit of sharing weights is to reduce the connections between the layers of the convolutional neural network, while reducing the risk of overfitting.

The network for extracting features mentioned below in this application may include one or more layers of convolutional layers. Exemplarily, the network for extracting features may be implemented by using CNN.

(4) Loss function

具体可以根据实际应用场景选择具体的损失函数。In the process of training a deep neural network, because it is hoped that the output of the deep neural network is as close as possible to the value you really want to predict, you can compare the predicted value of the current network with the target value you really want, and then based on the difference between the two to update the weight vector of each layer of neural network (of course, there is usually an initialization process before the first update, that is, to pre-configure parameters for each layer in the deep neural network), for example, if the predicted value of the network If it is high, adjust the weight vector to make the prediction lower, and keep adjusting until the deep neural network can predict the real desired target value or a value very close to the real desired target value. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value", which is the loss function or objective function, which is used to measure the difference between the predicted value and the target value. important equation. Among them, taking the loss function as an example, the higher the output value of the loss function (loss), the greater the difference, then the training of the deep neural network becomes the process of reducing the loss as much as possible. The loss function can generally include loss functions such as mean square error, cross entropy, logarithm, and exponential. For example, the mean squared error can be used as a loss function, defined as

Specifically, a specific loss function can be selected according to the actual application scenario.

(5) Back propagation algorithm

The neural network can use the error back propagation (BP) algorithm to correct the size of the parameters in the initial neural network model during the training process, so that the reconstruction error loss of the neural network model becomes smaller and smaller. Specifically, the input signal is passed forward until the output will generate error loss, and the parameters in the initial neural network model are updated by back-propagating the error loss information, so that the error loss converges. The back-propagation algorithm is a back-propagation movement dominated by error loss, aiming to obtain the parameters of the optimal neural network model, such as the weight matrix.

(6) Receptive Field

A term in the field of deep neural networks in the field of computer vision, used to indicate the size of the sensory range of the original image of neurons in different positions within the neural network. The larger the value of the neuron's receptive field, the larger the range of the original image that it can contact, which also means that the neuron may contain more global and semantic-level features; while the smaller the value, it means that it contains The features tend to be more local and detailed. The value of the receptive field can be roughly used to judge the abstraction level of each layer.

(7) Image (image quality) enhancement

Refers to the technology of processing the brightness, color, contrast, saturation, dynamic range, etc. of the image to meet certain specific indicators, or called image quality enhancement, which is equivalent to improving the quality of the image and making the image clearer.

(8) De-fog

It refers to a kind of image enhancement, which makes the image with certain blurring clearer. For example, an image may be captured in a foggy environment, and the captured image may not be clear at this time. The method provided in this application can be used to perform dehazing processing to improve the clarity or contrast of the image and make the image clearer.

(9) RGB image

An RGB image is an image with at least three channels, namely red (red), green (green) and blue (blue) channels. These three colors make up all the colors that vision can perceive and are one of the most widely used color systems. For example, a frame of RGB image is an M*N*3 color pixel array, where M*N is the size of the image, and each color pixel is a three-value group, and the three values correspond to the red, green and blue components respectively. .

Generally, CNN is a commonly used neural network. As mentioned below in this application, the network used for feature extraction can be CNN or other networks including convolutional layers. The structure of the product neural network is introduced.

The structure of the CNN will be described in detail below by way of example in conjunction with FIG. 2 . As mentioned in the introduction to the basic concepts above, a convolutional neural network is a deep neural network with a convolutional structure and a deep learning architecture. learning at multiple levels of abstraction. As a deep learning architecture, CNN is a feed-forward artificial neural network in which individual neurons can respond to images fed into it.

As shown in FIG. 2 , a convolutional neural network (CNN) 200 may include an input layer 210 , a convolutional/pooling layer 220 (where the pooling layer is optional), and a fully connected layer 230 . In the following embodiments of the present application, for ease of understanding, each layer is referred to as a stage. The relevant contents of these layers are described in detail below.

Convolutional layer/pooling layer 220:

As shown in FIG. 2 , the convolutional layer/pooling layer 220 may include layers 221-226 as examples. For example, in one implementation, layer 221 is a convolutional layer, layer 222 is a pooling layer, and layer 223 is a Convolution layer, 224 layers are pooling layers, 225 are convolution layers, and 226 are pooling layers; in another implementation, 221 and 222 are convolution layers, 223 are pooling layers, and 224 and 225 are volumes Layer, 226 is the pooling layer. That is, the output of a convolutional layer can be used as the input of a subsequent pooling layer, or it can be used as the input of another convolutional layer to continue the convolution operation.

The following will take the convolutional layer 221 as an example to introduce the inner working principle of a convolutional layer.

The convolution layer 221 may include many convolution operators. The convolution operator is also called a kernel. Its role in image processing is equivalent to a filter that extracts specific information from the input image matrix. The convolution operator is essentially Can be a weight matrix, which is usually pre-defined, usually one pixel by one pixel (or two pixels by two pixels) along the horizontal direction on the input image during the convolution operation on the image. ...It depends on the value of the stride step) to process, so as to complete the work of extracting specific features from the image. The size of the weight matrix should be related to the size of the image. It should be noted that the depth dimension of the weight matrix is the same as the depth dimension of the input image. During the convolution operation, the weight matrix will be extended to Enter the entire depth of the image. Therefore, convolution with a single weight matrix will result in a single depth dimension of the convolutional output, but in most cases a single weight matrix is not used, but multiple weight matrices of the same size (row × column) are applied, That is, multiple isotype matrices. The output of each weight matrix is stacked to form the depth dimension of the convolutional image, where the dimension can be understood as determined by the "multiple" described above. Different weight matrices can be used to extract different features in the image. For example, one weight matrix is used to extract image edge information, another weight matrix is used to extract specific colors of the image, and another weight matrix is used to extract unwanted noise in the image. Blur, etc. The multiple weight matrices have the same size (row×column), and the size of the feature maps extracted from the multiple weight matrices with the same size is also the same, and then the multiple extracted feature maps with the same size are combined to form a convolution operation. output.

The weight values in these weight matrices need to be obtained through a lot of training in practical applications, and each weight matrix formed by the weight values obtained by training can be used to extract information from the input image, so that the convolutional neural network 200 can make correct predictions .

When the convolutional neural network 200 has multiple convolutional layers, the initial convolutional layer (eg, 221 ) often extracts more general features, which can also be called low-level features; with the convolutional neural network As the depth of the network 200 deepens, the features extracted by the later convolutional layers (eg, 226) become more and more complex, such as features such as high-level semantics. Features with higher semantics are more suitable for the problem to be solved.

In the following embodiments of the present application, the feature extraction process mentioned may be a process of extracting features through convolution. For example, the convolution layer can include a separate convolution with a kernel size of 3×3 (sep_conv_3x3), a separate convolution with a kernel size of 5×5 (sep_conv_5x5), a kernel size of 3×3 and a dilation rate A dilated convolution of 2 (dil_conv_3x3), a dilated convolution with a kernel size of 5×5 and a dilation ratio of 2 (dil_conv_5x5), etc.

Pooling layer:

Since the number of training parameters often needs to be reduced, a pooling layer is often introduced periodically after the convolutional layer. The pooling layer can also be called a downsampling layer, which can be used to downsample images. In each layer 221-226 exemplified by 220 in Figure 2, it can be a convolutional layer followed by a pooling layer, or a multi-layer convolutional layer followed by one or more pooling layers. During image processing, the purpose of the pooling layer is to reduce the spatial size of the image. The pooling layer may include an average pooling operator and/or a max pooling operator for sampling the input image to obtain a smaller size image. The average pooling operator can calculate the pixel values in the image within a certain range to produce an average value as the result of average pooling. The max pooling operator can take the pixel with the largest value within a specific range as the result of max pooling. Also, just as the size of the weight matrix used in the convolutional layer should be related to the size of the image, the operators in the pooling layer should also be related to the size of the image. The size of the output image after processing by the pooling layer can be smaller than the size of the image input to the pooling layer, and each pixel in the image output by the pooling layer represents the average or maximum value of the corresponding sub-region of the image input to the pooling layer.

Fully connected layer 230:

After being processed by the convolutional layer/pooling layer 220, the convolutional neural network 200 is not sufficient to output the required output information. Because as mentioned before, the convolutional layer/pooling layer 220 only extracts features and reduces the parameters brought by the input image. However, in order to generate the final output information (required class information or other relevant information), the convolutional neural network 200 needs to utilize the fully connected layer 230 to generate one or a set of outputs of the required number of classes. Therefore, the fully connected layer 230 may include multiple hidden layers (231, 232 to 23n as shown in FIG. 2), and the parameters contained in the multiple hidden layers may be based on the relevant training data of specific task types Pre-trained, for example, the task type can include image enhancement, image recognition, image classification, image super-resolution reconstruction, etc.

After the multi-layer hidden layers in the fully connected layer 230, that is, the last layer of the entire convolutional neural network 200 is the output layer 240, the output layer 240 has a loss function similar to the classification cross entropy, and is specifically used to calculate the prediction error, Once the forward propagation of the entire convolutional neural network 200 (as shown in Figure 2, the propagation from the direction 210 to 240 is forward propagation) is completed, the back propagation (as shown in Figure 2, the propagation from the 240 to 210 direction is the back propagation) will Start to update the weight values and biases of the aforementioned layers to reduce the loss of the convolutional neural network 200 and the error between the result output by the convolutional neural network 200 through the output layer and the ideal result.

It should be noted that the convolutional neural network 200 shown in FIG. 2 is only used as an example of a convolutional neural network, and in a specific application, the convolutional neural network may also exist in the form of other network models. For example, only a part of the network structure shown in FIG. 2 is included, for example, the convolutional neural network adopted in this embodiment of the present application may only include the input layer 210 , the convolutional/pooling layer 220 and the output layer 240 .

In this application, the image to be processed can be processed by using the convolutional neural network 200 shown in FIG. 2 to obtain an enhanced image. As shown in FIG. 2 , the image to be processed is processed by the input layer 210 , the convolution layer/pooling layer 220 and the fully connected layer to output an enhanced image that is clearer and contains more texture information.

The method for image processing provided by the embodiments of this application may be executed on a server or a terminal device, or the neural network mentioned below in this application may be deployed on a server or a terminal , which can be adjusted according to actual application scenarios. The terminal device may be a mobile phone with image processing function, tablet personal computer (TPC), media player, smart TV, laptop computer (LC), personal digital assistant (PDA) ), a personal computer (PC), a camera, a video camera, a smart watch, a wearable device (WD), an autonomous vehicle, etc., which are not limited in this embodiment of the present application.

As shown in FIG. 3 , an embodiment of the present application provides a system architecture 100 . In Figure 3, a data collection device 160 is used to collect training data. In some optional implementations, for image enhancement, the sample pair included in the training data may include a lower quality image and a clear image, for example, a sample pair may include an image captured in a foggy day , and a heavily processed sharp image (or the ground truth image).

After collecting the training data, the data collection device 160 stores the training data in the database 130 , and the training device 120 obtains the target model/rule 101 by training based on the training data maintained in the database 130 . Optionally, the training set mentioned in the following embodiments of the present application may be obtained from the database 130 or obtained through user input data.

The target model/rule 101 may be the neural network after training in the embodiment of the present application.

The following describes how the training device 120 obtains the target model/rule 101 based on the training data. The training device 120 processes the input original image, and compares the output image with the original image until the image output by the training device 120 is the same as the original image. If the difference is smaller than a certain threshold, the training of the target model/rule 101 is completed.

The above-mentioned target model/rule 101 can be used to realize the neural network obtained by training the method for image processing according to the embodiment of the present application, that is, input the target model/rule 101 into the target model/rule 101 after the data to be processed (such as an image) is subjected to relevant preprocessing, The processing result can be obtained. The target model/rule 101 in the embodiment of the present application may specifically be the first neural network mentioned below in the present application, and the first neural network may be the aforementioned CNN, DNN, or RNN and other types of neural networks. It should be noted that, in practical applications, the training data maintained in the database 130 may not necessarily come from the collection of the data collection device 160, and may also be received from other devices. In addition, it should be noted that the training device 120 may not necessarily train the target model/rule 101 completely based on the training data maintained by the database 130, and may also obtain training data from the cloud or other places for model training. The above description should not be used as a reference to this application Limitations of Examples.

The target model/rule 101 trained according to the training device 120 can be applied to different systems or devices, such as the execution device 110 shown in FIG. 3 , the execution device 110 may also be referred to as a computing device. It may be a terminal, such as a mobile phone terminal, a tablet computer, a notebook computer, an augmented reality (AR)/virtual reality (virtual reality, VR), a vehicle terminal, etc., or a server or a cloud device. In FIG. 5 , the execution device 110 is configured with an input/output (I/O) interface 112 for data interaction with external devices, and the user can input data to the I/O interface 112 through the client device 140 . In this embodiment of the present application, the input data may include: pending data input by the client device.

The preprocessing module 113 and the preprocessing module 114 are used to perform preprocessing according to the input data (such as data to be processed) received by the I/O interface 112. In this embodiment of the present application, the preprocessing module 113 and the preprocessing module may also be absent. 114 (or only one of the preprocessing modules), and directly use the calculation module 111 to process the input data.

When the execution device 110 preprocesses the input data, or the calculation module 111 of the execution device 110 performs calculations and other related processing, the execution device 110 can call the data, codes, etc. in the data storage system 150 for corresponding processing , the data and instructions obtained by corresponding processing may also be stored in the data storage system 150 .

Finally, the I/O interface 112 returns the processing result to the client device 140 so as to be provided to the user. For example, if the first neural network is used for image classification and the processing result is a classification result, the I/O interface 112 The classification result obtained above is returned to the client device 140 to be provided to the user.

It should be noted that the training device 120 can generate corresponding target models/rules 101 based on different training data for different goals or tasks, and the corresponding target models/rules 101 can be used to achieve the above goals or complete The above task, thus providing the user with the desired result. In some scenarios, the execution device 110 and the training device 120 may be the same device, or located inside the same computing device. For ease of understanding, the present application will introduce the execution device and the training device separately, without limitation.

In the case shown in FIG. 3 , the user can manually give input data, and the manual setting can be operated through the interface provided by the I/O interface 112 . In another case, the client device 140 can automatically send the input data to the I/O interface 112 . If the user's authorization is required to request the client device 140 to automatically send the input data, the user can set the corresponding permission in the client device 140 . The user can view the result output by the execution device 110 on the client device 140, and the specific presentation form can be a specific manner such as display, sound, and action. The client device 140 can also act as a data collection terminal to collect the input data input into the I/O interface 112 as shown in the figure and the predicted label output from the I/O interface 112 as new sample data, and store them in the database 130 . Of course, it is also possible not to collect through the client device 140, but to use the I/O interface 112 to directly use the input data input into the I/O interface 112 and the predicted label output from the I/O interface 112 as shown in the figure as new samples The data is stored in database 130 .

It should be noted that FIG. 3 is only a schematic diagram of a system architecture provided by an embodiment of the present application, and the positional relationship between the devices, devices, modules, etc. shown in the figure does not constitute any limitation. For example, in FIG. 3 , the data The storage system 150 is an external memory relative to the execution device 110 , and in other cases, the data storage system 150 may also be placed in the execution device 110 .

As shown in FIG. 3 , the target model/rule 101 is obtained by training according to the training device 120. In this embodiment of the present application, the target model/rule 101 may be the neural network in the present application. Specifically, the neural network provided in the embodiment of the present application is the neural network in the present application. It can be CNN, deep convolutional neural networks (DCNN), recurrent neural network (RNN) and so on.

Referring to FIG. 4 , an embodiment of the present application further provides a system architecture 400 . The execution device 110 is implemented by one or more servers (such as server clusters), and optionally, cooperates with other computing devices, such as data storage, routers, load balancers and other devices; the execution device 110 can be arranged on a physical site, Or distributed over multiple physical sites. The execution device 110 can use the data in the data storage system 150 or call the program code in the data storage system 150 to implement the steps for the image processing method or the neural network training method corresponding to FIGS. 7-16 in the present application.

A user may operate respective user devices (eg, local device 401 and local device 402 ) to interact with execution device 110 . Each local device may represent any computing device, such as a personal computer, computer workstation, smartphone, tablet, smart camera, smart car or other type of cellular phone, media consumption device, wearable device, set-top box, gaming console, etc.

Each user's local device may interact with the execution device 110 through any communication mechanism/standard communication network, which may be a wide area network, a local area network, a point-to-point connection, etc., or any combination thereof. Specifically, the communication network may include a wireless network, a wired network, or a combination of a wireless network and a wired network, and the like. The wireless network includes but is not limited to: 5th-Generation (5G) system, Long Term Evolution (LTE) system, Global System for Mobile Communication (GSM) or Code Division Multiplexing address (code division multiple access, CDMA) network, wideband code division multiple access (wideband code division multiple access, WCDMA) network, wireless fidelity (wireless fidelity, WiFi), Bluetooth (bluetooth), Zigbee protocol (Zigbee), radio frequency identification Any one or a combination of any one or more of radio frequency identification (RFID), long range (Lora) wireless communication, and near field communication (NFC). The wired network may include an optical fiber communication network or a network composed of coaxial cables, and the like.

In another implementation, one or more aspects of the execution device 110 may be implemented by each local device, for example, the local device 401 may provide the execution device 110 with local data or feedback calculation results. The local device may also be referred to as a computing device.

It should be noted that all the functions of the execution device 110 can also be implemented by the local device. For example, the local device 401 implements the functions of the device 110 and provides services to the user, or provides services to the user of the local device 402, such as providing image optimization services.

Usually, in some foggy scenes or scenes with low light intensity, the captured images may be blurred or have low contrast, resulting in unclear images. For example, taking dehazing as an example, the early mainstream methods are traditional methods, which model and estimate light propagation, but artificially designed models and image prior knowledge cannot be accurately applied to complex and different types of real images. After a major breakthrough in the application of convolutional neural networks to a large number of vision tasks and achieved results, learning-based methods have become mainstream, replacing some modules in traditional methods with learnable network layers. However, the existing methods discard artificially designed modules such as light estimation, and use an end-to-end network to solve the task of dehazing, that is, the entire process of image transformation is learned by the network, and the algorithm is completely driven by data. The current end-to-end network that achieves the SOTA (State-of-the-art) effect requires a lot of computation, and it takes a lot of time to process images with higher resolutions and cannot process some images with higher resolutions in real time.

Some common dehazing methods can be, for example, using U-Net as the backbone network of the network, borrowing a method in the denoising field to gradually enhance the dehazing effect; and proposing a feature fusion module to fuse features of different scales together , to overcome the shortage of native U-Net's loss of spatial information and lack of non-adjacent level connections. However, this method has high computational complexity and takes a long time to process images, so real-time processing cannot be achieved.

For another example, a small resolution image can be processed to obtain the conversion coefficients stored in the bilateral grid, the features of the full resolution image can be extracted, and a guide map can be obtained; under the guidance of the guide map, the bilateral grid coefficients can be upsampled to obtain pixels. Transformation matrix, using the pixel transformation matrix to transform each pixel of the input image to get the final output. When obtaining the guide map, the features of the image need to be compressed, which may lead to information loss, and the method of upsampling the bilateral grid coefficients is inefficient to implement and takes a long time.

Therefore, the present application provides an image processing method, which realizes efficient image enhancement through a lightweight architecture, and even realizes real-time image processing, thereby improving the efficiency of image enhancement.

First, exemplarily, some scenarios of the application of the present application are exemplarily introduced.

For example, as shown in Figure 5, low-quality video data collected by various monitoring devices can be collected, and the low-quality video data can be stored in the memory, such as smart city, outdoor scene monitoring, indoor monitoring, in-vehicle monitoring, pet monitoring Surveillance, field shooting or field monitoring, etc. When playing the video data, image enhancement can be performed on the video data through the image processing method provided in the present application, such as defogging or increasing the contrast, so as to obtain video data with higher definition and improve the user's viewing experience.

For another example, the image processing method provided by the present application may be applied to a live video scenario. As shown in FIG. 6 , the server may send a video stream to the client used by the user. After receiving the data stream sent by the server, the client can perform image enhancement processing on the data stream by using the image processing method provided in the present application, thereby obtaining video data with higher definition and improving the user's viewing experience.

For another example, in an autonomous driving scenario, a camera set on a smart car can capture images around the vehicle, and based on the images, a driving path can be planned for the vehicle or a driving decision can be made. With the image processing method provided in the present application, the image captured by the camera can be enhanced to obtain a clearer image, especially in foggy scenes, the fog removal effect can be achieved, thereby improving the driving safety of the vehicle and improving the user experience.

Also, for example, a user can use a terminal to take pictures in a foggy environment, and the captured image may not be clear due to the influence of the fog. clarity and improve user experience.

Also for example, in some scenes with large differences in ambient light, the image captured by the user may be unclear due to low contrast. In this case, the captured image can be processed by using the image processing method provided in the The contrast of the image makes the image clearer and improves the user experience.

The image processing method provided by the present application will be introduced in detail below with reference to the foregoing scenarios and system architecture.

In addition, the image processing method provided by the present application can be implemented by a neural network, for example, the neural network can be deployed on the user's terminal, and the terminal can run the neural network, thereby realizing the image processing method provided by the present application. step.

First, referring to FIG. 7 , a schematic flowchart of an image processing method provided by the present application is as follows.

701. Obtain an input image.

The input image may be an image captured by the terminal or a received image, and the input image has multiple channels.

The input image may be obtained by user input. For example, referring to FIG. 3 , the user may send input data to the execution device 110 through the client device 140 , and the input data may carry the input image.

Alternatively, if the image processing method provided by the present application is executed by a terminal, the input image may also be an image captured by the terminal. For example, the input image may be an image captured by the terminal in a foggy scene, and needs to be dehazed to improve the clarity of the image.

702. Extract features from the information of multiple channels of the input image to obtain multiple guide maps.

Among them, the input image has multiple channels, and the present application can extract features from the dimensions of each channel to obtain a feature map of each channel dimension, that is, a guide map, and each channel can correspond to a feature map, that is, multiple guide maps can be obtained. picture. The multiple guide maps can subsequently be used to guide the upsampling of the bilateral grid data, which is equivalent to guiding the upsampling.

For example, the input image may include information of three channels, and features are extracted from the information of each channel, respectively, to obtain the corresponding features of each channel, and obtain the guide map corresponding to each channel.

703. Acquire bilateral grid data corresponding to the input image.

Wherein, the bilateral grid data may include data formed by spatial dimension information and luminance dimension information, in other words, the bilateral grid data includes data formed by luminance dimension information arranged in a preset space. It can be understood that the bilateral grid data may include data of at least three dimensions.

Specifically, the spatial dimension may be preset or determined according to the size of the input image, and the spatial dimension includes at least two dimensions, that is, the spatial dimension may include a two-dimensional space, a three-dimensional space, and the like.

For example, the spatial dimension corresponds to the size of the input image, and each position in the spatial dimension corresponds to one or more pixels in the input image. Accordingly, after the feature is extracted from the input image, the feature can be processed, The information obtained after processing is related to the brightness of the input image, and the feature is allocated to the spatial dimension to obtain bilateral grid data.

In a possible implementation manner, the input image is down-sampled to obtain the down-sampled image; features are extracted from the down-sampled image to obtain the down-sampled features, and the bilateral grid data may include the down-sampled features. The downsampling method may include various methods, such as bilinear interpolation or bicubic interpolation.

It can be understood that the input image is first downsampled to reduce the resolution of the image, and the subsequent computational complexity can be reduced to help achieve lightweight image enhancement; then features are extracted from the downsampled image, and the extracted features By performing the average pooling process, the information related to the brightness of the input image, that is, the down-sampling feature, can be obtained by mapping the down-sampling feature to the spatial dimension to obtain bilateral grid data.

The spatial dimension in the bilateral grid data may be a preset dimension, or may be a dimension determined according to the size of the input image. For example, a size can be preset, which is equivalent to the size of the spatial dimension of the bilateral grid data, and the pixels in the down-sampled image can have a corresponding relationship with this size. For example, the size can be 100*100, and the resolution of the down-sampled image can be If the ratio is 200*200, then every 4 pixels in the down-sampled image can be corresponded to one pixel in the size, and after the down-sampled features are extracted from the down-sampled image, that is, according to the down-sampled image The corresponding relationship between the pixel points and the size, the downsampling feature is allocated to the corresponding pixel points in the size, and the bilateral grid data can be obtained.

It should be noted that this application does not limit the execution order of step 702 and step 703. Step 702 may be executed first, or step 703 may be executed first, or step 702 and step 703 may be executed simultaneously, which may be adjusted according to actual application scenarios.

704. Using each guide map in the plurality of guide maps as a guide condition, perform up-sampling on the bilateral grid data to obtain multiple feature maps.

After multiple guide maps are obtained, the bilateral grid data can be up-sampled for each channel under the guidance of the guide maps corresponding to each channel, to obtain multiple feature maps corresponding to the multiple channels. Wherein, each guide map can be used to guide the selection of information related to the corresponding channel from the information of the brightness dimension in the bilateral grid data for up-sampling during up-sampling, so as to obtain a feature map corresponding to each channel.

Among them, various interpolation algorithms, such as bilinear interpolation, bicubic interpolation, or trilinear interpolation, can be used for the up-sampling method, so as to obtain feature maps with higher resolution than bilateral grid data.

Optionally, taking the processing process of one guide map (referred to as the first guide map) among the multiple guide maps as an example, using the first guide map as the guide condition, up-sampling the bilateral grid data to obtain the up-sampling feature, Then, the up-sampled features and the input image are fused to obtain the first feature map, that is, one of the multiple feature maps. The fusion method may include, but is not limited to, multiplication, splicing, or weighted fusion.

For example, the information of the luminance dimension in the bilateral grid data can be used as a coefficient, and up-sampling can be performed by means of interpolation. When , under the guidance of the guide map, one or more of the coefficients corresponding to each point or several points are selected for interpolation to obtain up-sampling features with the same size as the guide map. The specific guidance method of the guide map may include: the value corresponding to each pixel in the guide map can be understood as a brightness level, and each point in the guide map has a corresponding relationship with the point of the spatial dimension in the bilateral grid data, and can be based on the guide map. In the brightness level of each point in the figure, a coefficient matching the brightness level is selected from the multiple coefficients corresponding to each point or every few points in the bilateral grid data to perform interpolation to obtain the up-sampling feature.

Therefore, in the embodiments of the present application, the bilateral grid data can be upsampled by using the guide map as a guide, so as to obtain up-sampled features with higher resolution, and the present application uses the guide map of each channel to guide the bilateral grid data. Upsampling of grid data can obtain feature maps for each channel, so that the final output image can perform better and clearer in each channel.

Optionally, the manner of fusing the upsampling feature and the input image may include: compressing the upsampling to obtain the compressed feature. Then, the compressed feature and the input image are fused to realize feature reconstruction, and one of the corresponding feature maps is obtained. A specific fusion manner may include item-by-item product, or weighted fusion, and the like. The compression reduces the number of channels of the upsampling feature, so that the number of channels of the compressed feature is less than the number of channels of the upsampling feature, thereby reducing the amount of data. Among them, the specific method of compression can be compressed by convolution, which is equivalent to further feature extraction from up-sampling features. It can be compressed by adjusting the number of convolution kernels to obtain compression with a smaller amount of data. feature.

Therefore, in the embodiment of the present application, the upsampling feature can be compressed, and the compressed feature and the input image can be fused, and the feature reconstruction can be realized by fusing the compressed feature and the input image, so as to obtain an instance in each channel that can more accurately characterize the input image. Characteristics.

705. Fusing multiple feature maps to obtain an output image.

After multiple feature maps are obtained, the multiple feature maps can be fused, which is equivalent to restoring each channel to obtain a multi-channel output image. The number of channels of the output image and the number of channels of the input image are usually the same.

Therefore, in the embodiment of the present application, features can be extracted from the dimensions of multiple channels of the input image to obtain multiple guide maps, and under the guidance of the multiple guide maps, the low-resolution bilateral grid data are respectively up-sampled Obtaining multiple feature maps and fusing multiple feature maps with the output image is equivalent to smoothing the noise in the input image by upsampling the low-resolution bilateral grid data, improving the clarity of the input image, and realizing the input The dehazing effect of the image or enhance the contrast of the input image to get a clearer output image. In addition, the present application implements image enhancement by upsampling the low-resolution bilateral grid data. Since the resolution of the bilateral grid data is lower, the amount of computation consumed is less, thereby realizing lightweight image enhancement. .

Optionally, the specific process of fusing multiple feature maps may include: splicing multiple feature maps to obtain a spliced image, and then performing feature extraction on the spliced image at least once. The current feature is extracted from the features extracted last time to obtain at least one first feature, and the output image can be obtained by fusing the at least one feature with the input image. Therefore, in the embodiment of the present application, multiple feature maps can be fused by splicing, because the feature maps are obtained by up-sampling the bilateral grid data under the guidance of the guide map of each channel. A feature map can restore the channels of the spliced image, so that the spliced image has multiple channels, so that it can be fused with the input image to obtain an output image of multiple channels. It is equivalent to enhancing the input image from the dimension of each channel, and obtaining the enhanced output image, which improves the clarity of the image and improves the user experience.

Optionally, when splicing multiple feature maps, the multiple feature maps and the input image can also be spliced to obtain a spliced image. In this way, the details of multiple feature maps can be supplemented by the details included in the input image, such as low-frequency information such as color or brightness, so as to enrich the details of the stitched image, avoid excessive smoothing caused by the previous steps, and improve the output image quality. clarity.

The flow of the image processing method provided by the present application has been described in detail above. For ease of understanding, the following takes a specific application scenario as an example to describe the flow of the image processing method provided by the present application in more detail.

Referring to FIG. 8 , a schematic flowchart of another image processing method provided by the present application.

For ease of understanding, the image processing method for images of the present application is divided into multiple steps for introduction, such as bilateral grid generation 801 , guidance map generation 802 , feature reconstruction 803 , and image reconstruction 804 as shown in FIG. 8 , etc. .

然后基于下采样得到的图像

生成双边网格数据g，该双边网格数据包括空间维度和亮度相关的信息形成得到至少三维的数据。Among them, the bilateral grid generation 801: the full-resolution input image I can be down-sampled to obtain a low-resolution image

Then based on the downsampled image

The bilateral grid data g is generated, and the bilateral grid data includes spatial dimension and luminance-related information to form at least three-dimensional data.

Guide map generation 802: Extract features based on each channel of the full-resolution input image I, and obtain a guide map corresponding to each channel, such as the guide map G ₁ /G ₂ /G ₃ shown in FIG. 8 .

Feature reconstruction 803: For each channel, under the guidance of the corresponding guide map, upsample the bilateral grid to obtain the feature map F ₁ /F ₂ /G ₃ corresponding to each channel.

Image reconstruction 804: fuse the feature maps corresponding to each channel, and fuse the fused features with the input image to obtain the output image O. Optionally, when fusing the feature maps, the input image can also be fused to fuse the details (such as low-frequency features such as color or brightness) included in the input image with the features of each channel to obtain a fused image with richer details. image.

In the embodiment of the present application, under the guidance of the guide map of each channel, feature reconstruction can be performed based on bilateral grid data, so that more accurate and clear features can be reconstructed for each channel of the input image, and further fused with the input image, which can smooth The noise in the input image makes the input image clearer, and achieves the effect of dehazing or improving contrast.

Further, each step is described in more detail below.

1. Bilateral grid generation

即下采样图像，下采样的方式可以包括多种，如双线性插值、三线性插值或者双立方插值等。然后从低分辨率图像

中提取特征，得到下采样特征，并将下采样特征分配至对应的空间中，即可得到双边网格数据。该空间可以是预先设置的，也可以是根据输入图像的尺寸得到的空间，如基于输入图像的尺寸进行缩小，得到二维空间。Among them, the input image I of full resolution is down-sampled to obtain a low-resolution image

That is, the image is down-sampled, and the down-sampling manner may include various methods, such as bilinear interpolation, trilinear interpolation, or bicubic interpolation. then from a low resolution image

The features are extracted from the data, and the down-sampling features are obtained, and the down-sampling features are allocated to the corresponding space, and the bilateral grid data can be obtained. The space may be preset, or may be a space obtained according to the size of the input image, such as reducing the size of the input image to obtain a two-dimensional space.

之后，将该低分辨率图像

作为多层卷积的输入，提取到的特征可以用于表示低分辨率图像

的亮度，然后通过平均池化对提取到的特征进行池化处理，得到下采样特征。可以预先设定一个二维空间，该二维空间和输入图像或者低分辨率图像

的像素点具有映射关系，如输入图像中的一个或者多个像素点对应该二维空间中的一个点。然后将下采样特征映射至二维空间中的每个像素点，即可得到双边网格数据。即网格数据包括至少三维数据，其中包括二维空间，以及与输入图像的亮度相关的信息等，与输入图像的亮度相关的信息可以通过下采样特征来表示。For example, for generating bilateral grid data, see Figure 9, after obtaining low-resolution images

After that, this low resolution image

As the input of multi-layer convolution, the extracted features can be used to represent low-resolution images

Then, the extracted features are pooled through average pooling to obtain down-sampled features. A two-dimensional space can be preset, the two-dimensional space and the input image or low-resolution image

The pixels have a mapping relationship, such as one or more pixels in the input image correspond to a point in the two-dimensional space. Then the downsampled features are mapped to each pixel in the two-dimensional space, and the bilateral grid data can be obtained. That is, the grid data includes at least three-dimensional data, including a two-dimensional space, and information related to the brightness of the input image, etc. The information related to the brightness of the input image can be represented by down-sampling features.

Second, guide map generation

The input image I may include information of multiple channels, for example, an RGB image includes information of three channels. Information can be extracted from each channel of the input image I to obtain a feature map corresponding to each channel, and the feature map corresponding to each channel can be used as a guide map to guide subsequent upsampling of bilateral grid data.

Exemplarily, please refer to FIG. 10 for the way of generating the guide map. The input image I has three channels, such as channel 1, channel 2 and channel 3 as shown in FIG. 10 . For example, RGB channels can be divided into R, G, B three channels. Then perform feature extraction on each channel to obtain a feature map corresponding to each channel, such as feature maps G ₁ , G ₂ and G ₃ as shown in FIG. 10 .

Specifically, for example, as shown in FIG. 11, feature extraction can be achieved by multi-layer convolution. As shown in FIG. 11, one or more layers of convolution layers and one or more layers of parametric linear units can be modified. The rectified linear unit (PReLU) layer is used as a feature extraction network, and each channel of the input image is used as the input of the feature extraction network, and the feature map of each channel is output, that is, the feature maps G ₁ , G ₂ and G ₃ .

Therefore, in the embodiment of the present application, feature extraction is performed for each channel respectively, and multiple guide maps are obtained, which can extract more details in the input image and reduce the loss of details.

3. Feature reconstruction

Exemplarily, the feature reconstruction method of one of the channels can be as shown in FIG. 12. After obtaining the guide map and bilateral grid data, under the guidance of the guide map, the bilateral grid data is upsampled, and the method of upsampling is performed. Including but not limited to bilinear interpolation, bicubic interpolation or trilinear interpolation, etc., to obtain the up-sampling feature U, and then perform feature compression on the up-sampling feature U to obtain the compressed feature C, and then based on the input image I and the compressed feature C. Perform feature reconstruction, such as item-by-item product of the input image I and the compressed feature C, to obtain the feature map F corresponding to the current channel. For example, the resolution of the input image I may be 100*100, and the resolution of the compression feature C may also be 100*100. The pixels of the input image I correspond to the pixels of the compression feature C one-to-one. The value of each pixel of , and the value of the corresponding pixel in the compressed feature C are multiplied to obtain the feature map F. Of course, in addition to item-by-item multiplication, feature reconstruction can also be achieved by weighted fusion or splicing. This application exemplifies multiplication as an example for an exemplary introduction, which is not intended to be a limitation. The specific feature reconstruction method can be based on Actual application scene adjustment.

For example, as shown in Figure 13, the feature maps G ₁ , G ₂ and G ₃ are respectively used as guide maps to guide the up-sampling of the bilateral grid g to obtain the up-sampling feature U, and then reduce the up-sampling feature through convolution The number of channels of U to get the compressed feature C. Then, the feature map F can be obtained by multiplying the compressed feature C and the input image I.

Therefore, in the embodiment of the present application, feature reconstruction can be performed by multiplying the compressed feature and the input image, which is beneficial to the training convergence of the model, and the amount of calculation is small, which is beneficial to realize light-weight image enhancement.

4. Image reconstruction

Among them, taking the input image with three channels as an example, after the feature reconstruction is performed as described above, the feature maps F ₁ , F ₂ and F ₃ can be obtained, and the feature maps F ₁ , F ₂ and F ₃ can be spliced together, Get the stitched image. Then at least one iterative feature extraction is performed from the stitched image to obtain at least one feature. The at least one feature is spliced together and fused with the input image to obtain the final output image.

Specifically, for example, the image reconstruction method can be shown in Fig. 14. First, the feature maps F ₁ , F ₂ and F ₃ are spliced together to obtain a spliced image C ₁ . Optionally, after splicing the feature maps F ₁ , F ₂ and F ₃ , the input image can also be spliced in at the same time, so that the information included in the spliced image is richer and the loss of details is avoided. Perform at least one iterative feature extraction on _C1 , and record the features obtained by each feature extraction. Each feature extraction can be realized by one or more basic units (blocks) formed by convolution stacking, and each block can output one feature picture. Splicing at least one extracted feature to obtain a splicing feature C ₂ , and then merging the splicing feature C ₂ through a block to obtain a fused feature M, and merging the fused feature M with the input image to obtain a full-resolution enhanced output image O. The method of fusing the feature M and the input image can be weighted fusion or item-by-item product, which can be adjusted according to the actual application scenario.

Therefore, in the embodiment of the present application, the feature maps of each channel can be stitched to restore each channel of the image. And when splicing, the input image can be spliced to supplement the details in the feature map, so that the obtained spliced image has more details. Moreover, by fusing the feature M and the input image by means of multiplication, the dehazing of the input image can be realized, and an enhanced image can be obtained.

The flow of the image processing method provided by the present application has been introduced in detail. The following supplementary description is given to the neural network provided by the present application for executing the image processing method and the training method of the neural network. The neural network can be used to perform the aforementioned Fig. 7- 14 corresponds to the steps of the method.

Exemplarily, refer to FIG. 15 , which is a schematic structural diagram of a neural network provided by the present application.

Corresponding to the aforementioned FIG. 8 , it can be divided into multiple modules, such as the bilateral grid generation network, the guidance map generation network, the feature reconstruction network, and the image reconstruction network as shown in FIG. 15 .

然后基于下采样得到的图像

生成双边网格数据，该双边网格数据包括空间维度和亮度相关的信息形成得到至少三维的数据。Among them, the bilateral grid generation network can be used to: downsample the full-resolution input image I to obtain a low-resolution image

Then based on the downsampled image

Bilateral grid data is generated, and the bilateral grid data includes spatial dimension and luminance-related information to form at least three-dimensional data.

The guide map generation network is used to: extract features based on each channel of the full-resolution input image I, and obtain a guide map corresponding to each channel, such as the guide map G ₁ /G ₂ /G shown in FIG. 8 . ₃ .

The feature reconstruction network is used for: for each channel, under the guidance of the corresponding guide map, up-sampling the bilateral grid to obtain the feature map F ₁ /F ₂ /G ₃ corresponding to each channel.

The image reconstruction network is used to fuse the feature maps corresponding to each channel, and fuse the fused features with the input image to obtain the output image O. Optionally, when fusing the feature maps, the input image may also be fused, so as to fuse the details included in the input image with the features of each channel to obtain a fused image with more details.

Based on the structure of the neural network, the training method of the neural network is introduced below.

Referring to FIG. 16 , a schematic flowchart of a neural network training method provided by the present application is as follows.

1601. Obtain a training set.

The training set includes multiple image samples and labels corresponding to each image sample, and each image sample includes information of multiple channels.

Specifically, there may be various ways to obtain the training set. For example, if the application is executed by the training device 120 shown in FIG. 2, the training set may be the information extracted from the database 130, or the Data sent by client device 140 .

For example, a training set may include multiple sample pairs, each sample pair has an image sample and a corresponding ground-truth image (ie, a label), and the clarity of the ground-truth image is higher than that of the image sample. For example, after a clear multi-channel image is captured, the image is taken as the true value image, and the true value image is blurred, such as increasing fog or reducing contrast, etc., to obtain image samples, and the true value image is used as a pair of samples right.

1602. Perform at least one iterative training on the neural network using the training set to obtain a trained neural network.

Wherein, the image samples in the training set can be used as the input of the neural network to perform at least one iteration training on the neural network to obtain the trained neural network. The trained neural network can be used for image enhancement, such as implementing the steps of the corresponding methods in Figures 7-14 above.

The following takes an iterative process as an example to illustrate the training process, the following steps 16021-16026.

16021. Extract features from the information of multiple channels of the input image, respectively, to obtain multiple guide maps.

In an iterative training process, one or more image samples in the training set may be used as the input of the neural network, and this application exemplarily takes an input image as an example for exemplary introduction.

16022. Obtain bilateral grid data corresponding to the input image.

16023. Using each guide map in the plurality of guide maps as a guide condition, perform up-sampling on the bilateral grid data to obtain multiple feature maps.

16024. Fusing multiple feature maps to obtain an output image.

Wherein, steps 16021 to 16023 are similar to the processes of the foregoing steps 701 to 705, and are not repeated here.

16025. Update the neural network according to the output image and the corresponding ground-truth image to obtain the current updated neural network.

Among them, a variety of loss functions can be used, such as error square mean square, cross entropy, logarithm, exponential and other loss functions. Equivalent to using the function to calculate the offset between the output image and the ground truth image.

After the loss value is calculated, the gradient of the parameters of the neural network can be calculated based on the loss value, and the gradient can be used to represent the derivative when updating the parameters of the neural network, so that the parameters of the neural network can be updated according to the gradient, and the updated Neural Networks.

16026. Determine whether the convergence condition is met, if yes, go to step 1603, if not, go to step 16021.

After the updated neural network is obtained, it can be judged whether the convergence conditions are met, and if the convergence conditions are met, the updated neural network can be output, that is, the training of the neural network is completed. If the convergence conditions are not met, the neural network can continue to be trained, that is, step 16021 is repeatedly executed until the convergence conditions are met.

Wherein, the convergence condition may include one or more of the following: the number of times of training the neural network reaches a preset number, or the output accuracy of the neural network is higher than the preset accuracy value, or the average accuracy of the neural network is higher than the preset value average, or, the training time of the neural network exceeds the preset time, etc.

In the embodiment of the present application, when the upsampling and the input image are fused, the upsampling can be compressed to obtain the compressed feature, and then the compressed feature and the input image are multiplied item by item. Quickly feed back to the feature map, speed up the convergence speed of the neural network, and efficiently obtain a trained neural network.

1603. Output the updated neural network.

Among them, after the convergence conditions are met, the trained neural network can be output, and the neural network can be deployed in a terminal or server, such as a mobile phone, camera, smart car or monitoring equipment, such as taking pictures on mobile phones, cameras, etc. In scenarios such as autonomous driving or smart cities, it is used to enhance the collected images and improve the clarity of the images, especially in scenarios with high real-time and computational overhead requirements, the lightweight model provided by this application can be used. to achieve image enhancement.

Therefore, through the method provided in this application, a convergent neural network can be efficiently obtained, which can be applied in various lightweight scenarios, especially in some weak computing devices, and can be deployed on the computing nodes of related devices. The degraded image can be enhanced to obtain a clear and normal color result, or the contrast of the image can be improved to achieve an image enhancement effect.

In order to further facilitate the understanding of the image enhancement effect of the method provided by the present application, the following describes the image enhancement effect achieved by the method provided by the present application in combination with some common methods.

First, some common image enhancement methods can include: a fast single image haze removal algorithm using color attenuation prior (CAP), non-local image dehazing (Non-local image dehazing) , NLD), Single image haze removal using dark channel prior (DCP), Efficient image dehazing with boundary constraint and contextual regularization , BCCR), an end-to-end system for single image haze removal (DehazeNet), an all-in-one dehazing network (AOD), a multi-scale Single image dehazing via multiscale convolutional neural networks (MSCNN), robust haze removal based on patch map for single images (PMS) ), an attention-based multi-scale network for image dehazing (Griddehaze), a domain adaptation for image dehazing (DA), and the use of dense features Fusion multi-scale boosted dehazing network (multi-scale boosted dehazing network with dense feature fusion, MSBDN) and so on. Exemplarily, the running duration and peak signal-to-noise ratio (PSNR) achieved by the above-mentioned image enhancement method and the image processing method provided by the present application can be shown in FIG. 17 . Obviously, the present application The provided image processing method can obtain images with higher PSNR on the basis of lower running time, realize images with light weight and better dehazing effect, and realize the ability to process 4K images in real time.

In more detail, the following table 1 is a comparison of the output results of the above commonly used dehazing methods and the image processing methods provided in this application in the O-HAZE data set. Here, PSNR and structural similarity index measure (SSIM) are used for comparison. ,

Table 1

Obviously, the image processing method provided in this application can obtain an image with better dehazing effect on the basis of realizing high PSNR and high SSIM, and consuming less calculation amount.

More intuitively, the comparison of the image enhancement effects of the present application and some common methods can be shown in Figure 18A, Figure 18B and Figure 18C. Obviously, the image processing method provided by the present application can achieve better image dehazing effect. The PSNR and SSIM are also better, which greatly improves the clarity of the image and improves the user experience.

The flow of the method and the neural network provided by the present application are described in detail above, and the structure of the apparatus for executing the steps of the foregoing method provided by the present application is described below.

Referring to FIG. 19 , a schematic structural diagram of an image processing apparatus provided by the present application.

The image processing apparatus may include:

The transceiver module 1901 is used to acquire an input image, and the input image includes information of multiple channels;

The feature extraction module 1902 is used to extract features from the information of multiple channels of the input image to obtain multiple guide maps;

The bilateral generation module 1903 is used to obtain the bilateral grid data corresponding to the input image. The bilateral grid data includes the data formed by the information of the brightness dimension arranged in the spatial dimension, and the information of the brightness dimension is based on the features extracted from the input image. It is obtained that the resolution of the bilateral grid data is lower than the resolution of the input image, and the spatial dimension is a preset space or a space determined according to the input image;

The guidance module 1904 is used for up-sampling the bilateral grid data with each guidance picture in the plurality of guidance pictures as a guidance condition to obtain a plurality of feature maps;

The fusion module 1905 is used to fuse multiple feature maps to obtain an output image.

In a possible implementation manner, the guide module 1904 may be specifically configured to: use the first guide map as a guide condition to up-sample the bilateral grid data to obtain up-sampling features, where the first guide map is a plurality of guide maps Any one of: fusing the up-sampled feature and the input image to obtain a first feature map, and the first feature map is included in multiple feature maps.

In a possible implementation manner, the guiding module 1904 can be specifically configured to: compress the up-sampled features to obtain compressed features, and the number of channels of the compressed features is less than the number of channels of the up-sampled features; Item-by-item product to get the first feature map.

In a possible implementation, the bilateral generation module 1902 can be specifically configured to: downsample the input image to obtain a downsampled image; extract features from the downsampled image to obtain downsampled features, and the bilateral grid data includes downsampled features. Sampling features.

In a possible implementation, the fusion module 1905 can be specifically used for: splicing multiple feature maps to obtain a spliced image; performing feature extraction on the spliced image at least once to obtain at least one first feature; fusing at least one first feature and the input image to get the output image.

In a possible implementation manner, the fusion module 1905 may be specifically configured to splicing multiple feature maps and input images to obtain a spliced image.

Referring to FIG. 20 , a schematic structural diagram of a neural network training provided by the present application is described as follows.

The training device may include:

An acquisition module 2001 is used to acquire a training set, the training set includes multiple image samples and a ground truth image corresponding to each image sample, and each image sample includes information of multiple channels;

The training module 2002 is used to perform at least one iteration training on the neural network using the training set to obtain the trained neural network;

Among them, in any iterative training process, the neural network extracts features from the information of multiple channels of the input image, obtains multiple guide maps, and obtains the bilateral grid data corresponding to the input image as each of the multiple guide maps. Each guide map is used as a guide condition to upsample the bilateral grid data to obtain multiple feature maps, fuse multiple feature maps to obtain the output image, update the neural network according to the true value image corresponding to the output image and the input image, and obtain the current time In the updated neural network, the bilateral grid data includes the data formed by the information of the brightness dimension arranged in the preset space. The information of the brightness dimension is obtained according to the features extracted from the input image, and the resolution of the bilateral grid data is low. the resolution of the input image.

In a possible implementation, the training module 2002 can be specifically configured to: use the first guide map as a guide condition to upsample the bilateral grid data to obtain upsampling features, where the first guide map is a plurality of guide maps Any one of: fusing the up-sampled feature and the input image to obtain a first feature map, and the first feature map is included in multiple feature maps.

In a possible implementation, the training module 2002 can be specifically used to: compress the up-sampled features to obtain compressed features, and the number of channels of the compressed features is less than the number of channels of the up-sampled features; Item-by-item product to get the first feature map.

In a possible implementation manner, the training module 2002 can be specifically configured to: downsample the input image to obtain the downsampled image; extract features from the downsampled image to obtain the downsampled feature, and the bilateral grid data includes downsampling feature.

In a possible implementation manner, the training module 2002 can be specifically used for: splicing multiple feature maps to obtain a spliced image; performing feature extraction on the spliced image at least once to obtain at least one first feature; fusing at least one first feature and the input image to get the output image.

In a possible implementation, the training module 2002 can be specifically used to stitch multiple feature maps and input images to obtain a stitched image.

Please refer to FIG. 21 , which is a schematic structural diagram of another image processing apparatus provided by the present application, as described below.

The training device may include a processor 2101 and a memory 2102. The processor 2101 and the memory 2102 are interconnected by wires. Among them, the memory 2102 stores program instructions and data.

The memory 2102 stores program instructions and data corresponding to the steps in the aforementioned FIG. 7 to FIG. 14 .

The processor 2101 is configured to execute the method steps executed by the image processing apparatus shown in any of the foregoing embodiments in FIG. 7 to FIG. 14 .

Optionally, the image processing apparatus may further include a transceiver 2103 for receiving or sending data.

Embodiments of the present application also provide a computer-readable storage medium, where a program is stored in the computer-readable storage medium, and when the computer-readable storage medium is run on a computer, the computer is made to execute the program described in the foregoing embodiments shown in FIG. 7 to FIG. 14 . steps in the method.

Optionally, the aforementioned image processing device shown in FIG. 21 is a chip.

Please refer to FIG. 22 , which is a schematic structural diagram of another training device provided by the present application, as described below.

The training device may include a processor 2201 and a memory 2202. The processor 2201 and the memory 2202 are interconnected by wires. Among them, the memory 2202 stores program instructions and data.

The memory 2202 stores program instructions and data corresponding to the steps in the foregoing FIG. 15 to FIG. 16 .

The processor 2201 is configured to perform the method steps performed by the training apparatus shown in any of the foregoing embodiments in FIG. 15 to FIG. 16 .

Optionally, the training device may further include a transceiver 2203 for receiving or transmitting data.

Embodiments of the present application further provide a computer-readable storage medium, where a program is stored in the computer-readable storage medium, and when the computer-readable storage medium runs on a computer, the computer executes the above-described embodiments shown in FIG. 15-FIG. 16 . steps in the method.

Optionally, the aforementioned training device shown in FIG. 22 is a chip.

The embodiment of the present application also provides an image processing device, the training device may also be called a digital processing chip or a chip, the chip includes a processing unit and a communication interface, the processing unit acquires program instructions through the communication interface, and the program instructions are executed by the processing unit, The processing unit is configured to execute the method steps shown in any of the foregoing embodiments in FIGS. 7-14 .

The embodiment of the present application also provides a training device, which may also be called a digital processing chip or a chip. The chip includes a processing unit and a communication interface. The processing unit obtains program instructions through the communication interface, and the program instructions are executed by the processing unit. The unit is used to perform the steps of the method shown in any of the foregoing embodiments in FIGS. 15-16 .

The embodiments of the present application also provide a digital processing chip. The digital processing chip integrates circuits and one or more interfaces for implementing the above-mentioned processor 2101, processor 2201, or the functions of processor 2101 and processor 2201. When a memory is integrated in the digital processing chip, the digital processing chip can perform the method steps of any one or more of the foregoing embodiments. When the digital processing chip does not integrate the memory, it can be connected with the external memory through the communication interface. The digital processing chip implements the method steps in the above embodiments according to the program codes stored in the external memory.

The embodiments of the present application also provide a computer program product, which, when driving on the computer, causes the computer to execute the description in any of the foregoing embodiments in FIGS. 7 to 14 or any of the embodiments in FIGS. 15 to 16 . steps in the method.

The image processing apparatus or training apparatus provided in this embodiment of the present application may be a chip, and the chip includes: a processing unit and a communication unit. The processing unit may be, for example, a processor, and the communication unit may be, for example, an input/output interface, a pin or a circuit, etc. The processing unit can execute the computer-executed instructions stored in the storage unit, so that the chip in the server executes the deep learning training method for a computing device described in the embodiments shown in FIG. 7-FIG. 14 . Optionally, the storage unit is a storage unit in the chip, such as a register, a cache, etc., and the storage unit may also be a storage unit located outside the chip in the wireless access device, such as only Read-only memory (ROM) or other types of static storage devices that can store static information and instructions, random access memory (RAM), and the like.

Specifically, the aforementioned processing unit or processor may be a central processing unit (CPU), a network processor (neural-network processing unit, NPU), a graphics processing unit (graphics processing unit, GPU), a digital signal processor ( digital signal processor, DSP), application specific integrated circuit (ASIC) or field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. . A general purpose processor may be a microprocessor or it may be any conventional processor or the like.

Exemplarily, please refer to FIG. 23. FIG. 23 is a schematic structural diagram of a chip provided by an embodiment of the present application. The chip may be represented as a neural network processor NPU 230, and the NPU 230 is mounted as a coprocessor to the main CPU ( On the Host CPU), tasks are allocated by the Host CPU. The core part of the NPU is the arithmetic circuit 2303, which is controlled by the controller 2304 to extract the matrix data in the memory and perform multiplication operations.

In some implementations, the arithmetic circuit 2303 includes multiple processing units (process engines, PEs). In some implementations, the arithmetic circuit 2303 is a two-dimensional systolic array. The arithmetic circuit 2303 may also be a one-dimensional systolic array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 2303 is a general-purpose matrix processor.

For example, suppose there is an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit fetches the data corresponding to the matrix B from the weight memory 2302 and buffers it on each PE in the arithmetic circuit. The arithmetic circuit fetches the data of matrix A and matrix B from the input memory 2301 to perform matrix operation, and stores the partial result or final result of the matrix in the accumulator 2308 .

Unified memory 2306 is used to store input data and output data. The weight data is directly passed through a storage unit access controller (direct memory access controller, DMAC) 2305 , and the DMAC is transferred to the weight memory 2302 . Input data is also moved to unified memory 2306 via the DMAC.

A bus interface unit (BIU) 2310 is used for the interaction between the AXI bus and the DMAC and an instruction fetch buffer (instruction fetch buffer, IFB) 2309 .

The bus interface unit 2310 (bus interface unit, BIU) is used for the instruction fetch memory 2309 to obtain instructions from the external memory, and also for the storage unit access controller 2305 to obtain the original data of the input matrix A or the weight matrix B from the external memory.

The DMAC is mainly used to transfer the input data in the external memory DDR to the unified memory 2306 , the weight data to the weight memory 2302 , or the input data to the input memory 2301 .

The vector calculation unit 2307 includes a plurality of operation processing units, and further processes the output of the operation circuit if necessary, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison and so on. Mainly used for non-convolutional/fully connected layer network computations in neural networks, such as batch normalization, pixel-level summation, and upsampling of feature planes.

In some implementations, the vector computation unit 2307 can store the processed output vectors to the unified memory 2306 . For example, the vector calculation unit 2307 may apply a linear function and/or a nonlinear function to the output of the operation circuit 2303, such as linear interpolation of the feature plane extracted by the convolutional layer, such as a vector of accumulated values, to generate activation values. In some implementations, the vector computation unit 2307 generates normalized values, pixel-level summed values, or both. In some implementations, the vector of processed outputs can be used as activation input to the arithmetic circuit 2303, eg, for use in subsequent layers in a neural network.

an instruction fetch buffer 2309 connected to the controller 2304 for storing instructions used by the controller 2304;

The unified memory 2306, the input memory 2301, the weight memory 2302 and the instruction fetch memory 2309 are all On-Chip memories. External memory is private to the NPU hardware architecture.

The operation of each layer in the recurrent neural network can be performed by the operation circuit 2303 or the vector calculation unit 2307 .

Wherein, the processor mentioned in any of the above may be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more of the above-mentioned embodiments for controlling any of the above-mentioned FIG. 7-FIG. 14 or FIG. 15-FIG. 16. An integrated circuit executing the program of the method of any of the embodiments.

In addition, it should be noted that the device embodiments described above are only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be A physical unit, which can be located in one place or distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution in this embodiment. In addition, in the drawings of the device embodiments provided in the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

From the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary general-purpose hardware. Special components, etc. to achieve. Under normal circumstances, all functions completed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structures used to implement the same function can also be various, such as analog circuits, digital circuits or special circuit, etc. However, a software program implementation is a better implementation in many cases for this application. Based on this understanding, the technical solutions of the present application can be embodied in the form of software products in essence, or the parts that make contributions to the prior art. The computer software products are stored in a readable storage medium, such as a floppy disk of a computer. , U disk, removable hard disk, read only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc., including several instructions to make a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present application.

In the above-mentioned embodiments, it may be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented in software, it can be implemented in whole or in part in the form of a computer program product.

The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer may be a general purpose computer, special purpose computer, computer network, or other programmable device. The computer instructions may be stored in or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be downloaded from a website site, computer, server, or data center Transmission to another website site, computer, server, or data center is by wire (eg, coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (eg, infrared, wireless, microwave, etc.). The computer-readable storage medium may be any available medium that can be stored by a computer, or a data storage device such as a server, data center, etc., which includes one or more available media integrated. The usable media may be magnetic media (eg, floppy disks, hard disks, magnetic tapes), optical media (eg, DVDs), or semiconductor media (eg, solid state disks (SSDs)), and the like.

The terms "first", "second", "third", "fourth", etc. (if any) in the description and claims of this application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It is to be understood that data so used may be interchanged under appropriate circumstances so that the embodiments described herein can be practiced in sequences other than those illustrated or described herein. Furthermore, the terms "comprising" and "having" and any variations thereof, are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those expressly listed Rather, those steps or units may include other steps or units not expressly listed or inherent to these processes, methods, products or devices.

Finally, it should be noted that: the above are only specific embodiments of the present application, but the protection scope of the present application is not limited to this. Any person skilled in the art who is familiar with the technical scope disclosed by the present application can easily think of changes. Or replacement should be covered within the protection scope of this application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims (20)

1. A method of image processing, comprising:

acquiring an input image, wherein the input image comprises information of a plurality of channels;

extracting features from information of a plurality of channels of the input image respectively to obtain a plurality of guide graphs, wherein the guide graphs correspond to the channels one by one;

acquiring bilateral grid data corresponding to the input image, wherein the bilateral grid data comprise data formed by brightness dimension information arranged in a space dimension, the brightness dimension information is obtained according to features extracted from the input image, the resolution of the bilateral grid data is lower than that of the input image, and the space dimension is a preset space or a space determined according to the input image;

taking each guide graph in the guide graphs as a guide condition, and performing up-sampling on the bilateral grid data to obtain a plurality of characteristic graphs;

and fusing the plurality of characteristic graphs to obtain an output image.

2. The method of claim 1, wherein the upsampling the bilateral mesh data with each of the plurality of pilot maps as a pilot condition, respectively, comprises:

using a first guide graph as a guide condition, performing upsampling on the bilateral grid data to obtain an upsampling characteristic, wherein the first guide graph is any one of the plurality of guide graphs;

and fusing the up-sampling feature and the input image to obtain a first feature map, wherein the first feature map is included in the plurality of feature maps.

3. The method of claim 2, wherein said fusing the upsampled features with the input image to obtain a first feature map comprises:

compressing the up-sampling features to obtain compressed features, wherein the number of channels of the compressed features is less than that of the up-sampling features;

and carrying out item-by-item product (element-wise product) on the compressed features and the input image to obtain the first feature map.

4. The method according to any one of claims 1-3, wherein the obtaining bilateral mesh data corresponding to the input image comprises:

down-sampling the input image to obtain a down-sampled image;

extracting features from the downsampled image to obtain downsampled features;

and determining the bilateral grid data according to the downsampling characteristics.

5. The method according to any one of claims 1-4, wherein said fusing the plurality of feature maps to obtain the output image comprises:

splicing the plurality of characteristic graphs to obtain a spliced image;

performing at least one time of feature extraction on the spliced image to obtain at least one first feature;

and fusing the at least one first feature and the input image to obtain the output image.

6. The method of claim 5, wherein said stitching the plurality of feature maps to obtain a stitched image comprises:

and splicing the plurality of feature maps and the input image to obtain the spliced image.

7. A neural network training method, comprising:

acquiring a training set, wherein the training set comprises a plurality of image samples and a true value image corresponding to each image sample, and each image sample comprises information of a plurality of channels;

performing at least one iterative training on the neural network by using the training set to obtain a trained neural network;

wherein, in any iterative training process, the neural network extracts features from the information of a plurality of channels of the input image respectively to obtain a plurality of guide maps, the plurality of guide graphs correspond to the plurality of channels one by one, the bilateral grid data corresponding to the input image are obtained, and each guide graph in the plurality of guide graphs is used as a guide condition, up-sampling the bilateral grid data to obtain a plurality of characteristic graphs, fusing the characteristic graphs to obtain an output image, updating the neural network according to the true value images corresponding to the output image and the input image to obtain the updated neural network at the current time, the bilateral mesh data includes data formed of information of a luminance dimension arranged in a preset space, the information of the brightness dimension is obtained according to features extracted from the input image, and the resolution of the bilateral mesh data is lower than that of the input image.

8. The method of claim 7, wherein the upsampling the bilateral mesh data with each of the plurality of pilot maps as a pilot condition, respectively, comprises:

using a first guide graph as a guide condition, performing upsampling on the bilateral grid data to obtain an upsampling characteristic, wherein the first guide graph is any one of the plurality of guide graphs;

and fusing the up-sampling feature and the input image to obtain a first feature map, wherein the first feature map is included in the plurality of feature maps.

9. An apparatus for image processing, comprising:

the receiving and sending module is used for acquiring an input image, and the input image comprises information of a plurality of channels;

the characteristic extraction module is used for extracting characteristics from information of a plurality of channels of the input image respectively to obtain a plurality of guide graphs, and the guide graphs correspond to the channels one to one;

the bilateral generation module is used for acquiring bilateral grid data corresponding to the input image, wherein the bilateral grid data comprise data formed by brightness dimension information arranged in a space dimension, the brightness dimension information is obtained according to features extracted from the input image, the resolution of the bilateral grid data is lower than that of the input image, and the space dimension is a preset space or a space determined according to the input image;

the guiding module is used for respectively taking each guiding graph in the plurality of guiding graphs as a guiding condition and carrying out up-sampling on the bilateral grid data to obtain a plurality of characteristic graphs;

and the fusion module is used for fusing the plurality of characteristic graphs to obtain an output image.

10. The apparatus according to claim 9, wherein the guidance module is specifically configured to:

using a first guide graph as a guide condition, performing upsampling on the bilateral grid data to obtain an upsampling characteristic, wherein the first guide graph is any one of the plurality of guide graphs;

and fusing the up-sampling feature and the input image to obtain a first feature map, wherein the first feature map is included in the plurality of feature maps.

11. The apparatus according to claim 10, wherein the guidance module is specifically configured to:

compressing the up-sampling features to obtain compressed features, wherein the number of channels of the compressed features is less than that of the up-sampling features;

and carrying out item-by-item product on the compressed characteristic and the input image to obtain the first characteristic map.

12. The apparatus according to any one of claims 9 to 11, wherein the bilateral generation module is specifically configured to:

down-sampling the input image to obtain a down-sampled image;

extracting features from the downsampled image to obtain downsampled features;

and determining the bilateral grid data according to the downsampling characteristics.

13. The device according to any one of claims 9 to 12, wherein the fusion module is specifically configured to:

splicing the plurality of characteristic graphs to obtain a spliced image;

performing at least one time of feature extraction on the spliced image to obtain at least one first feature;

and fusing the at least one first feature and the input image to obtain the output image.

14. The apparatus of claim 13,

the fusion module is specifically configured to splice the plurality of feature maps and the input image to obtain the spliced image.

15. An exercise device, comprising:

the device comprises an acquisition module, a processing module and a processing module, wherein the acquisition module is used for acquiring a training set, the training set comprises a plurality of image samples and a true value image corresponding to each image sample, and each image sample comprises information of a plurality of channels;

the training module is used for carrying out at least one iterative training on the neural network by using the training set to obtain the trained neural network;

wherein, in any iterative training process, the neural network extracts features from the information of a plurality of channels of the input image respectively to obtain a plurality of guide maps, the plurality of guide graphs correspond to the plurality of channels one by one, the bilateral grid data corresponding to the input image are obtained, and each guide graph in the plurality of guide graphs is used as a guide condition, up-sampling the bilateral grid data to obtain a plurality of characteristic graphs, fusing the characteristic graphs to obtain an output image, updating the neural network according to the true value images corresponding to the output image and the input image to obtain the updated neural network at the current time, the bilateral mesh data includes data formed of information of a luminance dimension arranged in a preset space, the information of the brightness dimension is obtained according to features extracted from the input image, and the resolution of the bilateral mesh data is lower than that of the input image.

16. The apparatus of claim 15, wherein the training module is specifically configured to:

using a first guide graph as a guide condition, performing upsampling on the bilateral grid data to obtain an upsampling characteristic, wherein the first guide graph is any one of the plurality of guide graphs;

and fusing the up-sampling feature and the input image to obtain a first feature map, wherein the first feature map is included in the plurality of feature maps.

17. An image processing apparatus comprising a processor coupled to a memory, the memory storing a program, the program instructions stored by the memory when executed by the processor implementing the method of any of claims 1 to 6.

18. An exercise apparatus comprising a processor coupled to a memory, the memory storing a program, the program instructions stored by the memory when executed by the processor implementing the method of claim 7 or 8.

19. A computer readable storage medium comprising a program which, when executed by a processing unit, performs the method of any of claims 1 to 8.

20. A computer program product, characterized in that it comprises a software code for performing the method according to any one of claims 1 to 8.

Images