WO2022151813A1 - Electronic device, front image signal processor, and image processing method - Google Patents

Abstract

Embodiments of the present application provide an electronic device, a front image signal processor, and an image processing method. The method comprises: acquiring a scene image of a photography scene; identifying, according to the scene image, an object existing in the photography scene; performing white balance correction on the scene image to obtain a corrected image; and when a color vector of the object in the photography scene and a pre-assigned color vector of the object have a color cast, performing color reduction processing on the corrected image to obtain a restored image.

Description

Electronic device, front image signal processor and image processing method

This application claims the priority of the Chinese patent application filed on January 12, 2021 with the application number 202110035973.9 and the application name "Electronic Equipment, Front Image Signal Processor and Image Processing Method", the entire contents of which are approved by Reference is incorporated in this application.

technical field

The present application relates to the technical field of image processing, and in particular, to an electronic device, a front-end image signal processor, and an image processing method.

Background technique

At present, users usually use electronic devices with cameras (such as digital cameras, smart phones, etc.) to capture images, so as to record what happens around them, the scenery they see, and the like anytime and anywhere. In order to provide a higher shooting experience, it is not only required to improve the camera pixels of the electronic device, but also to improve the color reproduction capability of the electronic device. In the related art, in order to make the photographed image truly reflect the color of the photographed object, an automatic white balance technology is proposed, which aims to make the photographed object under different light source illumination scenes have the same color as the human eye sees under the same light source illumination scene. reduction. However, electronic devices have poor color reproduction capabilities based on white balance.

SUMMARY OF THE INVENTION

Embodiments of the present application provide an electronic device, a front image signal processor, and an image processing method, which can improve the color reproduction capability of the electronic device.

In a first aspect, an embodiment of the present application provides an electronic device, including: a camera for collecting a scene image of a shooting scene; a front image signal processor for recognizing objects existing in the shooting scene according to the scene image ; an application processor for performing white balance correction on the scene image to obtain a corrected image; and when there is a color shift between the color vector of the object in the corrected image and the pre-assigned color vector of the object, according to The color shift performs color restoration processing on the corrected image to obtain a restored image.

In a second aspect, an embodiment of the present application further provides a front image signal processor, including: a data interface for acquiring a scene image of a shooting scene from a camera; and transmitting the scene image to an application processor for white balance correction , and receive the corrected image returned after the application processor performs white balance correction; a neural network processing unit is used to identify the scene image through an object recognition model, so as to identify the object existing in the shooting scene; image A signal processing unit, configured to perform color restoration processing on the corrected image according to the color shift when there is a color shift between the color vector of the object in the corrected image and the pre-assigned color vector of the object to obtain a restoration image.

In a third aspect, the embodiments of the present application further provide an image processing method, which acquires a scene image of a shooting scene; performs white balance correction on the scene image to obtain a corrected image; and identifies images existing in the shooting scene according to the scene image object; when the color vector of the object in the corrected image has a color deviation from the pre-assigned color vector of the object, perform color restoration processing on the corrected image according to the color deviation to obtain a restored image.

Description of drawings

In order to illustrate the technical solutions in the embodiments of the present application more clearly, the following briefly introduces the accompanying drawings that are used in the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative effort.

FIG. 1 is a schematic structural diagram of an electronic device provided by an embodiment of the present application.

FIG. 2 is an example diagram of an object existing in a scene image being recognized and obtained according to an embodiment of the present application.

FIG. 3 is a schematic diagram of a refined structure of the front image signal processor in FIG. 1 .

FIG. 4 is an example diagram of sampling pixel points determined in an embodiment of the present application.

5 is a schematic diagram of the corresponding relationship between the color shift direction and the preset similarity, which is associated with the pre-assigned color vector in the embodiment of the application.

FIG. 6 is an example diagram of a color shift vector calculated in an embodiment of the present application.

FIG. 7 is an example diagram of interpolation processing in an embodiment of the present application.

FIG. 8 is a schematic structural diagram of a pre-image signal processor provided by an embodiment of the present application.

FIG. 9 is a schematic flowchart of an image processing method provided by an embodiment of the present application.

Detailed ways

The technical solutions provided in the embodiments of the present application can be applied to various scenarios where data communication is required, which is not limited in the embodiments of the present application.

This embodiment provides an electronic device, the electronic device includes:

A camera, used to collect scene images of the shooting scene;

a front image signal processor, configured to identify objects existing in the shooting scene according to the scene image;

An application processor, configured to perform white balance correction on the scene image to obtain a corrected image; and when there is a color shift between the color vector of the object in the corrected image and the pre-assigned color vector of the object, according to the The color cast performs color restoration processing on the corrected image to obtain a restored image.

In some embodiments, the pre-image signal processor is configured to perform state statistics on the scene image to obtain state information required by the application processor to perform white balance correction; and perform optimization processing on the scene image, Get the optimized scene image;

The application processor is configured to perform white balance correction on the optimized scene image according to the state information to obtain the corrected image.

In some embodiments, the pre-image signal processor includes:

an image signal processing unit, configured to perform state statistics on the scene image to obtain the state information; and perform a first optimization process on the scene image;

A neural network processing unit for performing a second optimization process on the scene image after the first optimization process; and performing object recognition on the scene image after the second optimization process through an object recognition model, so as to identify the shooting scene objects that exist in .

In some embodiments, the application processor is configured to determine a sampling pixel point for color sampling from the object in the object area in the corrected image, and use the color vector of the sampling pixel point as the the color vector of the object; and calculating the vector difference between the color vector and the pre-assigned color vector, and determining whether there is a color shift between the color vector and the pre-assigned color vector according to the vector difference.

In some embodiments, the application processor is configured to determine a difference threshold based on the color shift direction of the color vector and the pre-assigned color vector, and determine the difference between the color vector and the color vector based on the difference threshold and the vector difference. Whether there is a color cast in the pre-assigned color vector.

In some embodiments, the application processor is configured to calculate a difference vector between the color vector and the pre-assigned color vector, and use the difference vector as a color shift vector of the sampled pixel point; and according to the Perform interpolation processing on the color shift vector of the sampled pixel points to obtain the color shift vector of the non-sampled pixel points in the corrected image; A restoration process is performed to obtain the restored image.

In some embodiments, the application processor is configured to perform interpolation processing according to more than one interpolation algorithm according to the color shift vector of the sampled pixel point, to obtain a plurality of candidate color shift vectors of the non-sampled pixel point; and According to the plurality of candidate color shift vectors, the color shift vectors of the non-sampled pixels are obtained.

In some embodiments, the application processor is configured to calculate the average value of components of the plurality of candidate color shift vectors in each dimension, and obtain the color shift of the non-sampling pixel point according to the average value of each dimension component amount; or

Weighted summation is performed on components of the plurality of candidate color shift vectors in each dimension, and the color shift vector of the non-sampled pixel point is obtained according to the weighted sum value of each dimension component.

In some embodiments, the application processor is configured to obtain the recognition confidence level corresponding to the object, and perform correction processing on the color shift vector of the sampling pixel point according to the recognition confidence level, so as to obtain the correction of the sampling pixel point and performing interpolation processing according to the corrected color shift vector of the sampled pixel points to obtain the color shift vector of the non-sampled pixel points in the corrected image.

Please refer to FIG. 1 , which is a schematic structural diagram of an electronic device 100 according to an embodiment of the present application. The electronic device 100 includes a camera 110, a front image signal processor 120 and an application processor 130, wherein,

a camera 110, used for collecting scene images of the shooting scene;

The front image signal processor 120 is used for identifying objects existing in the shooting scene according to the scene image;

The application processor 130 is configured to perform white balance correction on the scene image to obtain a corrected image; and when there is a color shift between the color vector of the aforementioned object in the corrected image and the pre-assigned color vector of the aforementioned object, perform a color shift on the corrected image according to the color shift. Color restoration processing to obtain restored images.

It should be noted that, in the embodiments of the present application, the entity presentation form of the electronic device is not specifically limited, and the entity presentation form of the electronic device may be a mobile electronic device such as a smartphone, a tablet computer, a palmtop computer, and a notebook computer, or may be a mobile electronic device. Desktop computers, TVs and other stationary electronic equipment.

As above, the electronic device provided by this application at least includes the camera 110 , the front image signal processor 120 and the application processor 130 .

The camera 110 is composed of multiple parts, mainly including a lens, a motor, and an image sensor. Among them, the lens is used to project the external light signal to the image sensor; the image sensor is used to photoelectrically convert the light signal projected by the lens to convert the light signal into a usable electrical signal to obtain the original image data; and the motor is used to drive The lens moves, thereby adjusting the distance between the lens and the image sensor to satisfy the imaging formula (or lens imaging formula, Gaussian imaging formula, etc.), so that the image is clear.

Based on the hardware capability of the camera 110, in this embodiment of the present application, the camera 110 is configured to collect a scene image of a shooting scene. The shooting scene can be understood as the scene that the camera 110 is aimed at after being enabled, that is, the scene where the camera 110 can convert the light signal into corresponding image data. For example, after the electronic device enables the camera 110 according to the user operation, if the user controls the camera 110 of the electronic device 100 to aim at a scene including an object, the scene including the object is the shooting scene of the camera 110 .

According to the above description, those skilled in the art should understand that the shooting scene does not refer to a specific scene, but a scene that is aimed in real time following the direction of the camera 110 . Usually, the shooting scene does not include only a single object, and various objects may exist in it. For example, when shooting a portrait in a shooting scene, the shooting scene of the camera 110 includes not only the "target person" to be shot, but also other objects such as grass, trees, and buildings.

Usually, a specific object has a specific color, such as the sky is usually blue, the clouds are usually white, and the fire hydrant is usually red. Based on this, in the embodiments of the present application, different objects are assigned colors corresponding to them in advance based on experience, which are denoted as pre-assigned color vectors, thereby establishing a correspondence between objects and pre-assigned color vectors. In this way, the object in the shooting scene can be identified, and the pre-assigned color vector of the object in the shooting scene can be compared with the object in the photographed image after white balance by using the object relationship between the object and the pre-assigned color vector. Then, according to the comparison result, it can be judged whether the white balance is abnormal, that is, it can be judged whether the image after the white balance has a color cast. Based on this, the front image signal processor 120 is configured to identify objects existing in the shooting scene according to the scene image according to the configured identification strategy. The configuration of the recognition strategy is not specifically limited here, and can be configured by those of ordinary skill in the art according to actual needs, including but not limited to target-based object recognition methods and artificial intelligence-based object recognition methods. For example, please refer to FIG. 2 , which shows a scene image, and the front image signal processor 120 recognizes an object "fire hydrant" existing in the corresponding shooting scene by identifying the scene image.

The application processor 130 performs white balance correction on the aforementioned scene image according to the configured white balance strategy, and obtains a scene image after the white balance correction, which is recorded as a corrected image. The configuration of the white balance strategy is not specifically limited here, and can be configured by those of ordinary skill in the art according to actual needs, including but not limited to white balance correction methods based on grayscale world and white balance correction methods based on color temperature estimation.

The application processor 130 is further configured to obtain the color vector of the object in the identified shooting scene in the corrected image; and obtain the pre-assigned color vector corresponding to the object in the shooting scene according to the correspondence between the object and the pre-assigned color vector. and judge whether the color vector of the object in the shooting scene and the pre-assigned color vector have color shift; and when the color vector of the object in the shooting scene and the pre-assigned color vector have color shift, according to the color vector of the object in the shooting scene and the pre-assigned color vector Color shift of color vector Perform color restoration processing on the corrected image to eliminate color shift, and record the corrected image after color restoration processing as a restored image.

As can be seen from the above, the electronic device 100 provided by the present application includes a camera 110 , a front image signal processor 120 and an application processor 130 . Wherein, the scene image of the shooting scene is collected by the camera 110, and the objects existing in the shooting scene are identified by using the scene image by the front image signal processor 120, and the white balance correction is performed on the scene image by the application processor 130 to obtain a corrected image, And when there is a color deviation between the color vector of the object in the shooting scene in the corrected image and the pre-assigned color vector of the object, the application processor 130 performs color restoration processing on the corrected image to obtain a restored image. In this way, by identifying the color shift existing in the white balance result and performing the color restoration processing accordingly, the stability of the color restoration capability of the electronic device 100 can be ensured.

Optionally, in an embodiment, in order to improve the efficiency of image processing, the application processor 130 is configured to perform white balance correction on the scene image while the front image signal processor 120 recognizes objects existing in the shooting scene according to the scene image. to get the corrected image.

Optionally, in an embodiment, the pre-image signal processor 120 is configured to perform state statistics on the scene image to obtain the state information required by the application processor 130 to perform white balance correction; and perform optimization processing on the scene image to obtain Optimized scene image;

The application processor 130 is configured to perform white balance correction on the optimized scene image according to the state information to obtain a corrected image.

It should be noted that, in this embodiment of the present application, the pre-image signal processor 120 is further configured to perform state statistics on the scene image, so as to obtain state information required by the application processor 130 to perform white balance correction. In addition, the pre-image signal processor 120 is further configured to perform optimization processing on the scene image according to the configured optimization strategy after obtaining the state information required by the application processor 130 to perform white balance correction by statistics to obtain an optimized scene image . The configuration of the optimization strategy is not specifically limited here, and can be flexibly configured by those of ordinary skill in the art according to the processing performance of the pre-image signal processor 120 and actual needs. The optimization process optimizes the scene image.

After the aforementioned state information and the optimized scene image obtained through statistics are obtained, the pre-image signal processor 120 also transmits the aforementioned state information and the aforementioned optimized scene image to the application processor 130 .

In addition, the application processor 130 is further configured to perform white balance correction on the optimized scene image according to the aforementioned state information to obtain a corrected image.

Optionally, in an embodiment, please refer to FIG. 3 , the pre-image signal processor 120 includes:

The image signal processing unit 1201 is used to perform state statistics on the scene image to obtain the state information required by the application processor 130 to perform white balance correction; and perform the first optimization process on the scene image;

The neural network processing unit 1202 is used to perform a second optimization process on the scene image after the first optimization process; and perform object recognition on the scene image after the second optimization process through an object recognition model, to identify the object that exists.

As shown in FIG. 3 , the pre-image signal processor 120 includes an image signal processing unit 1201 and a neural network processing unit 1202 . The image signal processing unit 1201 is configured to perform state statistics on the scene image to obtain the state information required by the application processor 130 to perform white balance correction. In addition, the image signal processing unit 1201 is further configured to perform a first optimization process on the aforementioned scene image, including but not limited to dead pixel correction processing, Domain noise reduction processing, 3D noise reduction processing, linearization processing and black level correction processing and other non-artificial intelligence-based optimization processing methods. Of course, optimization processing methods not listed in this application may also be included.

In addition, the neural network processing unit 1202 is configured to perform a second optimization process on the scene image after the image signal processing unit 1201 performs the first optimization process. The manner in which the neural network processing unit 1202 processes the image data may be to read data blocks in a row manner, and process the data blocks in a row manner. For example, the neural network processing unit 1202 reads the data block in a multi-line manner, and processes the data block in a multi-line manner. It can be understood that one frame of image data may have multiple rows of data blocks, that is, the neural network processing unit 1202 may process a part of one frame of image data, such as n rows of data blocks, where n is a positive integer, such as 2, 4, 5 Wait. When the neural network processing unit 1202 has not completely processed a frame of image data, the neural network processing unit 1202 may have a built-in cache to store the data of the multi-line data blocks processed by the neural network processing unit 1202 in the process of processing one frame of image data.

It should be noted that, in the data stream, the neural network processing unit 1202 can complete the processing according to the preset time. The preset time is, for example, 30fps=33ms (milliseconds). In other words, the preset time for the neural network processing unit 1202 to process one frame of image is 33ms, which can ensure that the neural network processing unit 1202 can realize real-time data transmission on the basis of rapidly processing image data.

The second optimization processing performed by the neural network processing unit 1202 includes, but is not limited to, optimization processing methods based on artificial intelligence such as night scene algorithms, HDR algorithms, blurring algorithms, noise reduction algorithms, and super-resolution algorithms. Of course, optimization processing methods not listed in this application may also be included.

It can be seen from the above that the optimization process performed by the pre-image signal processor 120 is divided into two parts, namely, the first optimization process based on non-artificial intelligence performed by the image signal processing unit 1201, and the first optimization process based on artificial intelligence performed by the neural network processing unit 1202. Intelligent second optimization processing.

It should be noted that, in this embodiment of the present application, the neural network processing unit 1202 is further deployed with an object recognition model, and the object recognition model is configured to recognize objects existing in the input image. The architecture and training method of the object recognition model are not specifically limited here, and can be selected by those of ordinary skill in the art according to actual needs.

Exemplarily, in the embodiments of the present application, a convolutional neural network model is used as the basic model for training the object recognition model. In addition, sample images including different objects (for example, objects with a specific color may be preferentially selected) are also acquired, and object labels of objects existing in the sample images are manually marked. After that, the convolutional neural network model is supervised by using the obtained sample objects and the correspondingly calibrated object labels until the convolutional neural network model converges, and the converged convolutional neural network model is used as the object for object recognition. Identify the model.

Optionally, in one embodiment, the application processor 130 is configured to determine sampling pixels for color sampling from the aforementioned objects in the object area in the corrected image, and use the color vectors of the sampling pixels as the color vectors of the aforementioned objects. color vector; and calculating the vector difference between the aforementioned color vector and the pre-assigned color vector, and judging whether there is a color shift between the aforementioned object's color vector and the pre-assigned color vector according to the calculated vector difference.

In the embodiment of the present application, color sampling is performed on the identified object in the shooting scene, and the color vector obtained by sampling is used to identify whether there is a color shift in the white balance correction result.

Wherein, the application processor 130 firstly determines the sampling pixel points for color sampling from the aforementioned object in the object area in the corrected image. The selection of sampling pixel points is not specifically limited here, and can be selected by those of ordinary skill in the art as required. For example, the pixel point located at the geometric center of the object area in the corrected image may be used as the sampling pixel point, or a pixel point may be randomly selected from the aforementioned object in the object area of the corrected image as the sampling pixel point.

After determining the sampled pixel points for color sampling, the application processor 130 uses the color vector of the sampled pixel point as the color vector of the aforementioned object. For example, please refer to FIG. 4, which shows a scene image of a shooting scene. According to the scene image, the front image signal processor 120 identifies an object "fire hydrant" existing in the shooting scene, and the application processor 130 uses the object The pixel point of the geometric center of the image area of the "fire hydrant" in the illustrated scene image is determined as the sampling pixel point, and the color vector of the sampling pixel point is used as the color vector of the object "fire hydrant".

As above, after sampling the color vector of the aforementioned object, the application processor 130 further calculates the vector difference between the color vector of the aforementioned object and its pre-allocated color vector, which can be expressed as:

r ₃ =(r ₁ -r ₂ )/256;

g ₃ =(g ₁ -g ₂ )/256;

b ₃ =(b ₁ -b ₂ )/256;

where diff represents the vector difference between the color vector of the aforementioned object and its pre-assigned color vector, r ₁ represents the component value of the aforementioned pre-assigned color vector in the red channel, g ₁ represents the component value of the aforementioned pre-assigned color vector in the green channel, b ₁ represents the component value of the aforementioned pre-assigned color vector in the blue channel, r ₂ represents the component value of the aforementioned color vector in the red channel, g ₂ represents the component value of the aforementioned color vector in the green channel, and b ₂ represents the aforementioned color vector in the blue channel. component value.

As above, after calculating the vector difference between the color vector of the object and the pre-assigned color vector, the application processor 130 can determine whether there is a color shift between the color vector of the object and the pre-assigned color vector according to the calculated vector difference.

For example, a difference threshold for determining the presence of color shift may be pre-configured, and accordingly, by comparing whether the aforementioned vector difference is greater than or equal to the difference threshold, it can be determined whether there is color shift according to the comparison result. Wherein, if the aforementioned vector difference is greater than or equal to the difference threshold, it is determined that there is a color deviation between the aforementioned color vector and the aforementioned pre-assigned color vector (represented by the difference vector between the aforementioned color vector and the aforementioned pre-assigned color vector), if the aforementioned vector difference is less than the difference. If the threshold is set, it is determined that there is no color shift between the aforementioned color vector and the aforementioned pre-assigned color vector.

It should be noted that the value of the difference threshold is not specifically limited in the embodiments of the present application, and can be configured by those of ordinary skill in the art according to actual needs. The difference threshold may be configured as a fixed value, or the value of the difference threshold may be dynamically determined. value.

Optionally, in one embodiment, the application processor 130 is configured to determine a difference threshold based on the color shift direction of the aforementioned color vector and the aforementioned pre-assigned color vector, and to judge the aforementioned color vector and the aforementioned color vector based on the difference threshold and the aforementioned vector difference. Whether there is a color cast in the pre-allocated color vector.

In the embodiment of the present application, considering that the sensitivity of human eyes to different colors is different, the color shift direction is used to dynamically determine the value of the difference threshold.

Wherein, for different pre-assigned color vectors, according to the sensitivity of the human eye to different colors, in the embodiment of the present application, the corresponding relationship between the color shift direction and the difference threshold associated with each pre-assigned color vector is pre-established. For example, please refer to Figure 5. The ellipse in the figure represents the corresponding relationship between the color shift direction and the difference threshold associated with the pre-assigned color vectors. There are 18 ellipses, which represent the colors associated with each of the 18 pre-assigned color vectors. The correspondence between the bias direction and the difference threshold. Taking the ellipse numbered "7" as an example, for the pre-assigned color vector associated with the ellipse, the difference threshold corresponding to when the color shift direction is biased towards blue or red is significantly larger than that corresponding to when the color shift direction is biased towards green. , because the human eye is more sensitive to green.

As above, based on the established correspondence between the color shift direction and the difference threshold associated with the pre-assigned color vector, when the application processor 130 identifies the color shift, it first determines that the color vector of the identified object in the shooting scene is compared with It pre-allocates the color shift direction of the color vector, and then determines the difference threshold corresponding to the color shift direction of the aforementioned color vector according to the corresponding relationship between the color shift direction associated with the pre-assigned color vector and the difference threshold, and then judges the aforementioned Whether the vector difference of the color vectors is greater than or equal to the difference threshold corresponding to the aforementioned color shift direction, wherein, if the vector difference of the aforementioned color vector is greater than or equal to the aforementioned difference threshold corresponding to the aforementioned color shift direction, it is determined that the aforementioned color vector and the aforementioned pre-assigned color vector There is a color shift (here, the aforementioned vector difference and color shift direction are used to characterize), otherwise, it is determined that there is no color shift between the aforementioned color vector and the aforementioned pre-assigned color vector.

For example, referring to FIG. 6 , the application processor 130 determines that the color vector of the object "fire hydrant" and its pre-assigned color vector have a color shift by identifying the color shift of the object "fire hydrant" in the shooting scene shown in the figure. The arrow represents the color shift, wherein the direction of the arrow represents the color shift direction, and the length of the arrow represents the difference threshold, that is, the longer the length, the greater the difference threshold.

Optionally, in one embodiment, the application processor 130 is configured to calculate the difference vector between the aforementioned color vector and the pre-assigned color vector, and use the aforementioned difference vector as the color shift vector of the sampling pixel; and according to the sampling pixel Perform interpolation processing on the color shift vector of the corrected image to obtain the color shift vector of the non-sampling pixel in the corrected image; and perform color restoration processing on each pixel according to the color shift vector of each pixel in the corrected image to obtain a restored image.

In this embodiment of the present application, the application processor 130 is configured to calculate a difference vector between the foregoing color vector and the foregoing pre-assigned color vector, and use the difference vector as a color shift vector of the sampling pixel. Then, according to a preconfigured interpolation strategy, the application processor 130 interpolates according to the color shift vector of the sampled pixel points to obtain the color shift vector of the non-sampled pixel points in the corrected image. So far, the color shift vectors of all pixels in the corrected image, including sampling pixels and non-sampling pixels, are known, and each pixel can be color-coded according to the color shift vector of each pixel in the corrected image. Restoration processing to obtain a restored image.

The interpolation processing performed in the above embodiments of the present application can be understood as a process of estimating the color shift vectors of other pixel points in the entire corrected image through the known color shift vectors of discrete pixels.

For example, referring to FIG. 7 , in the corrected image on the upper side of FIG. 7 , the front image signal processor 120 has identified 8 different objects in total, and the black solid circles shown in FIG. 7 represent the objects corresponding to these identified objects. Sampling pixel points, the arrows represent the corresponding color shift vector. As shown in FIG. 7 , the application processor 130 obtains the color shift vectors of all pixels in the corrected image by interpolation according to the color shift vectors of the sampled pixels.

It should be noted that the interpolation strategy used in the embodiments of the present application is not specifically limited, and can be selected by those of ordinary skill in the art according to actual needs, including but not limited to recent field interpolation, bilinear interpolation, or bicubic interpolation, etc.

Optionally, in one embodiment, the application processor 130 is configured to perform interpolation processing according to more than one interpolation algorithm according to the color shift vector of the sampled pixel points, to obtain a plurality of candidate color shift vectors of the non-sampled pixel points; and According to the multiple candidate color shift vectors, the color shift vectors of the non-sampling pixels are obtained;

Wherein, for the same non-sampling pixel point, the neighborhood pixel points selected by the application processor 130 during interpolation according to different interpolation strategies are different.

It should be noted that, in the embodiment of the present application, a single interpolation strategy is not used for interpolation processing, but multiple interpolation strategies are combined to perform interpolation processing. There is no specific limitation on which interpolation strategy is adopted and the number of interpolation strategies, which can be configured according to the processing capability of the application processor 130 in the art. In addition, the embodiment of the present application does not specifically limit which fusion strategy is adopted, and can be configured by those of ordinary skill in the art according to actual needs.

For example, three different interpolation strategies are pre-configured in the embodiment of the present application, which are denoted as interpolation strategy A, interpolation strategy B, and interpolation strategy C, respectively. During interpolation processing, for a non-sampling pixel point, the application processor 130 uses interpolation strategy A to obtain the candidate color shift vector A of the non-sampling pixel point according to the color shift vector of the sampled pixel point, and obtains the candidate color shift vector A of the non-sampled pixel point by interpolation strategy B. The candidate color shift vector B of the non-sampling pixel point, and the candidate color shift vector C of the non-sampling pixel point obtained by interpolation using an interpolation strategy C. Finally, the application processor 130 fuses the candidate color shift vector A, the candidate color shift vector B, and the candidate color shift vector C into one vector according to the configured fusion strategy, which is used as the color shift vector of the non-sampled pixel point.

Optionally, in one embodiment, the application processor 130 is configured to calculate the average value of the components of the multiple candidate color shift vectors in each dimension, and obtain the color shift of the non-sampling pixel point according to the average value of each dimension component. amount; or

Weighted summation is performed on the components of the multiple candidate color shift vectors in each dimension, and the color shift vector of the non-sampling pixel point is obtained according to the weighted sum value of each dimension component.

Two optional fusion strategies are further provided in the embodiments of the present application.

First, for a non-sampling pixel, the application processor 130 calculates the average value of the components of its multiple candidate color shift vectors in each dimension, and obtains the color shift vector of the non-sampling pixel according to the average value of each dimension component. .

Second, assign weights for weighted summation to different interpolation strategies in advance. For example, with the weight sum value being 1 as a constraint, if the accuracy of an interpolation strategy is higher, the assigned weight will be higher. For a non-sampling pixel, the application processor 130 performs a weighted summation of the components of the multiple candidate color shift vectors of the non-sampling pixel in each dimension according to the weight corresponding to each interpolation strategy, The weighted sum value of , obtains the color shift vector of the non-sampled pixel point.

Optionally, in one embodiment, the application processor 130 is configured to obtain the recognition confidence level corresponding to the aforementioned object, and perform correction processing on the color shift vector of the aforementioned sampling pixel point according to the recognition confidence level, to obtain the corrected sample pixel point. color shift vector; and performing interpolation processing on the corrected color shift vector of the sampled pixel points to obtain the color shift vector of the non-sampled pixel points in the corrected image.

In the embodiment of the present application, the magnitude of the color restoration is determined according to the recognition confidence of the object recognized by the pre-image signal processor 120 .

Wherein, the application processor 130 first obtains the recognition confidence of the pre-image signal processor 120 to identify the object in the shooting scene, and performs correction processing on the color shift vector of the aforementioned sampling pixel points according to the recognition confidence, which can be expressed as:

V'=V*α;

Among them, V' represents the corrected color shift vector of the sampled pixel point, V represents the color shift vector of the sampled pixel point obtained by calculation, and α represents the recognition confidence of the aforementioned object.

After completing the correction of the color shift vector of the sampling pixel, the application processor 130 further performs interpolation processing according to the corrected color shift vector of the sampling pixel, so as to obtain the color shift vector of the non-sampling pixel in the corrected image, For details, reference may be made to the relevant descriptions in the above embodiments, which will not be repeated here.

The embodiment of the present application also provides a pre-image signal processor, and the device includes:

a data interface for acquiring a scene image of a shooting scene from a camera; and transmitting the scene image to an application processor for white balance correction, and receiving a corrected image returned after the application processor performs white balance correction;

a neural network processing unit, configured to recognize the scene image through an object recognition model, so as to recognize the object existing in the shooting scene;

An image signal processing unit, configured to perform color restoration processing on the corrected image according to the color deviation when the color vector of the object in the corrected image and the pre-assigned color vector of the object have a color deviation, to obtain Restore the image.

In some embodiments, the image signal processing unit is configured to determine a sampling pixel point for color sampling from the object in the object area in the corrected image, and use the color vector of the sampling pixel point as a the color vector of the object; and calculating the vector difference between the color vector and the pre-assigned color vector, and judging whether there is a color shift between the color vector and the pre-assigned color vector according to the vector difference.

In some embodiments, the image signal processing unit is configured to determine a difference threshold based on the color shift direction between the color vector and the pre-assigned color vector, and determine the color based on the difference threshold and the vector difference Whether there is a color shift between the vector and the pre-assigned color vector.

In some embodiments, the image signal processing unit is configured to calculate a difference vector between the color vector and the pre-assigned color vector, and use the difference vector as a color shift vector of the sampling pixel; and Perform interpolation processing according to the color shift vector of the sampled pixel points to obtain the color shift vector of the non-sampled pixel points in the corrected image; A color restoration process is performed to obtain the restored image.

In some embodiments, the image signal processing unit is configured to perform interpolation processing according to more than one interpolation algorithm according to the color shift vectors of the sampled pixels to obtain multiple candidate color shift vectors of the non-sampled pixels ; and obtaining the color shift vector of the non-sampling pixel point according to the plurality of candidate color shift vectors.

In some embodiments, the image signal processing unit is configured to obtain the recognition confidence corresponding to the object, and perform correction processing on the color shift vector of the sampling pixel point according to the recognition confidence to obtain the sampling pixel point-corrected color shift vector; and performing interpolation processing according to the corrected color shift vector of the sampled pixel point to obtain the color shift vector of the non-sampled pixel point in the corrected image.

The present application also provides a pre-image signal processor. As shown in FIG. 8 , the pre-image signal processor 200 includes:

a data interface 210 for acquiring a scene image of the shooting scene from the camera; and transmitting the scene image to the application processor for white balance correction, and receiving the corrected image returned after the application processor performs white balance correction;

The neural network processing unit 220 is used to identify the scene image through the object recognition model, to identify the object existing in the shooting scene;

The image signal processing unit 230 is configured to perform color restoration processing on the corrected image according to the color shift to obtain a restored image when there is a color shift between the color vector of the object in the corrected image and the pre-assigned color vector of the object.

It should be noted that the front image signal processor provided in the present application can be applied to an electronic device having a camera and an application processor to improve the color reproduction capability of the electronic device.

Wherein, the type of the data interface 210 is not specifically limited in the embodiments of the present application, including but not limited to a mobile industry processor interface (Mobile Industry Processor Interface, MIPI), a PCI-E interface, and the like.

The shooting scene can be understood as the scene that the camera is aimed at after being enabled, that is, the scene where the camera can convert the light signal into corresponding image data. For example, after the electronic device enables the camera according to the user operation, if the user controls the camera of the electronic device to aim at a scene including an object, the scene including the object is the shooting scene of the camera.

From the above description, those of ordinary skill in the art should understand that the shooting scene does not refer to a specific scene, but a scene that is aimed in real time following the direction of the camera. Usually, the shooting scene does not include only a single object, and various objects may exist in it. For example, when shooting a portrait in a shooting scene, the shooting scene of the camera not only includes the "target person", but also other objects such as grass, trees, and buildings.

Usually, a specific object has a specific color, such as the sky is usually blue, the clouds are usually white, and the fire hydrant is usually red. Based on this, in the embodiments of the present application, different objects are assigned colors corresponding to them in advance based on experience, which are denoted as pre-assigned color vectors, thereby establishing a correspondence between objects and pre-assigned color vectors. In this way, the object in the shooting scene can be identified, and the pre-assigned color vector of the object in the shooting scene can be compared with the object in the photographed image after white balance by using the object relationship between the object and the pre-assigned color vector. Then, according to the comparison result, it can be judged whether the white balance is abnormal, that is, it can be judged whether the image after the white balance has a color cast. Based on this, the data interface 210 is configured to acquire the scene image of the shooting scene from the camera.

It should be noted that, in this embodiment of the present application, the neural network processing unit 220 is deployed with an object recognition model, and the object recognition model is configured to recognize objects existing in the input image. The architecture and training method of the object recognition model are not specifically limited here, and can be selected by those of ordinary skill in the art according to actual needs.

Exemplarily, in the embodiments of the present application, a convolutional neural network model is used as the basic model for training the object recognition model. In addition, sample images including different objects (for example, objects with a specific color may be preferentially selected) are also acquired, and object labels of objects existing in the sample images are manually marked. After that, the convolutional neural network model is supervised by using the obtained sample objects and the correspondingly calibrated object labels until the convolutional neural network model converges, and the converged convolutional neural network model is used as the object for object recognition. Identify the model.

Correspondingly, the neural network processing unit 220 is configured to recognize the scene image through the object recognition model, so as to recognize the object existing in the shooting scene.

For example, please refer to FIG. 2 , which shows a scene image. The pre-image signal processor 200 recognizes the scene image through the neural network processing unit 220, and identifies an object "fire hydrant" in the corresponding shooting scene.

In addition, the data interface 210 is further configured to transmit the scene image to the application processor to perform white balance correction, and to receive the corrected image returned after the application processor performs the white balance correction.

The application processor performs white balance correction on the aforementioned scene image according to the configured white balance strategy, and obtains the scene image after the white balance correction, which is recorded as a corrected image. The configuration of the white balance strategy is not specifically limited here, and can be configured by those of ordinary skill in the art according to actual needs, including but not limited to white balance correction methods based on grayscale world and white balance correction methods based on color temperature estimation. After the corrected image is obtained from the correction, the corrected image is returned to the data interface 210.

The image signal processing unit is configured to obtain the identified color vector of the object in the shooting scene in the corrected image; and obtain the pre-assigned color vector corresponding to the object in the shooting scene according to the correspondence between the object and the pre-assigned color vector; And judge whether the color vector of the object in the shooting scene and the pre-assigned color vector have color shift; The color cast of the vector performs color restoration processing on the corrected image to eliminate the color cast, and records the corrected image after the color restoration processing as the restored image.

It can be seen from the above that the front image signal processor 200 provided by this application obtains the scene image of the shooting scene from the camera through the data interface 210; and transmits the scene image to the application processor for white balance correction, and receives the application processor for white balance correction. The corrected image returned after the balance correction; the scene image is recognized based on the object recognition model by the neural network processing unit 220 to identify the object existing in the shooting scene; and the color vector of the object in the corrected image by the image signal processing unit 230 When there is a color shift from the pre-assigned color vector of the object, the corrected image is subjected to color restoration processing according to the color shift to obtain a restored image. In this way, by identifying the color shift existing in the white balance result, and performing color restoration processing accordingly, the color restoration capability of the electronic device can be improved.

Optionally, in an embodiment, the image signal processing unit 230 is configured to perform state statistics on the scene image to obtain state information required by the application processor to perform white balance correction;

The data interface 210 is used to transmit the aforementioned status information and the scene image to the application processor for white balance correction, and receive the corrected image returned after the application processor performs the white balance correction.

It should be noted that, in this embodiment of the present application, the image signal processing unit 230 is further configured to perform state statistics on the scene image, so as to obtain state information required by the application processor to perform white balance correction. In addition, the data interface 210 is configured to transmit the aforementioned status information and the aforementioned scene image to the application processor for white balance correction, and to receive the corrected image returned after the application processor performs white balance correction.

Optionally, in an embodiment, the image signal processing unit 230 is further configured to perform a first optimization process on the scene image after obtaining the state information from statistics;

The neural network processing unit 220 is further configured to perform a second optimization process on the scene image after the first optimization process;

The data interface 210 is used to transmit the foregoing status information and the scene image after the second optimization process to the application processor for white balance correction, and receive the corrected image returned after the application processor performs white balance correction.

The image signal processing unit 230 is further configured to perform a first optimization process on the aforementioned scene image, including but not limited to dead pixel correction processing, temporal noise reduction, after the state information required by the application processor to perform white balance correction is statistically obtained. Processing, 3D noise reduction processing, linearization processing and black level correction processing and other non-artificial intelligence-based optimization processing methods. Of course, optimization processing methods not listed in this application may also be included.

In addition, the neural network processing unit 220 is configured to perform a second optimization process on the scene image after the image signal processing unit 230 performs the first optimization process. The manner in which the neural network processing unit 220 processes the image data may be to read data blocks in a row manner, and process the data blocks in a row manner. For example, the neural network processing unit 220 reads the data block in a multi-line manner, and processes the data block in a multi-line manner. It can be understood that one frame of image data may have multiple rows of data blocks, that is, the neural network processing unit 220 may process a part of one frame of image data, such as n rows of data blocks, where n is a positive integer, such as 2, 4, and 5. Wait. When the neural network processing unit 220 has not finished processing a frame of image data, the neural network processing unit 220 may have a built-in cache to store the data of the multi-line data blocks processed by the neural network processing unit 220 in the process of processing one frame of image data.

It should be noted that, in the data stream, the neural network processing unit 220 can complete the processing according to the preset time. The preset time is, for example, 30fps=33ms (milliseconds). In other words, the preset time for the neural network processing unit 220 to process one frame of image is 33ms, which can ensure that the neural network processing unit 220 can realize real-time data transmission on the basis of fast processing image data.

The second optimization processing performed by the neural network processing unit 220 includes, but is not limited to, optimization processing methods based on artificial intelligence such as night scene algorithms, HDR algorithms, blurring algorithms, noise reduction algorithms, and super-resolution algorithms. Of course, optimization processing methods not listed in this application may also be included.

It can be seen from the above that the pre-image signal processor 200 performs two optimization processes through the image signal processing unit 230 and the neural network processing unit 220 respectively, which are the first optimization process based on non-artificial intelligence performed by the image signal processing unit 230 respectively, and artificial intelligence-based second optimization processing performed by the neural network processing unit 220 .

The data interface 210 is further configured to transmit the foregoing status information and the scene image after the second optimization process to the application processor for white balance correction, and receive a corrected image returned after the application processor performs white balance correction.

Optionally, in an embodiment, the neural network processing unit 220 is configured to perform object recognition on the scene image after the second optimization process through the object recognition model, so as to recognize the objects existing in the shooting scene.

Optionally, in one embodiment, the image signal processing unit 230 is configured to determine sampling pixels for color sampling from the object in the object area in the corrected image, and use the color vector of the sampling pixels as the color of the object. vector; and calculating the vector difference between the color vector and the pre-allocated color vector, and determining whether there is a color shift between the color vector and the pre-allocated color vector according to the vector difference.

In the embodiment of the present application, color sampling is performed on the identified object in the shooting scene, and the color vector obtained by sampling is used to identify whether there is a color shift in the white balance correction result.

Wherein, the image signal processing unit 230 firstly determines the sampling pixel points for color sampling from the aforementioned object in the object area in the corrected image. The selection of sampling pixel points is not specifically limited here, and can be selected by those of ordinary skill in the art as required. For example, the pixel point located at the geometric center of the object area in the corrected image may be used as the sampling pixel point, or a pixel point may be randomly selected from the aforementioned object in the object area of the corrected image as the sampling pixel point.

After determining the sampled pixel points for color sampling, the image signal processing unit 230 uses the color vector of the sampled pixel point as the color vector of the aforementioned object. For example, please refer to FIG. 4, which shows a scene image of a shooting scene. According to the scene image, the front image signal processor 200 recognizes an object "fire hydrant" existing in the shooting scene, and the image signal processing unit 230 will The pixel point of the object "fire hydrant" in the geometric center of the image area in the illustrated scene image is determined as the sampling pixel point, and the color vector of the sampling pixel point is used as the color vector of the object "fire hydrant".

As above, after sampling the color vector of the aforementioned object, the image signal processing unit 230 further calculates the vector difference between the color vector of the aforementioned object and its pre-assigned color vector, which can be expressed as:

r ₃ =(r ₁ -r ₂ )/256;

g ₃ =(g ₁ -g ₂ )/256;

b ₃ =(b ₁ -b ₂ )/256;

where diff represents the vector difference between the color vector of the aforementioned object and its pre-assigned color vector, r ₁ represents the component value of the aforementioned pre-assigned color vector in the red channel, g ₁ represents the component value of the aforementioned pre-assigned color vector in the green channel, b ₁ represents the component value of the aforementioned pre-assigned color vector in the blue channel, r ₂ represents the component value of the aforementioned color vector in the red channel, g ₂ represents the component value of the aforementioned color vector in the green channel, and b ₂ represents the aforementioned color vector in the blue channel. component value.

As above, after calculating the vector difference between the color vector of the object and the pre-assigned color vector, the image signal processing unit 230 can determine whether there is a color shift between the color vector of the object and the pre-assigned color vector according to the calculated vector difference.

For example, a difference threshold for determining the presence of color shift may be pre-configured, and accordingly, by comparing whether the aforementioned vector difference is greater than or equal to the difference threshold, it can be determined whether there is color shift according to the comparison result. Wherein, if the aforementioned vector difference is greater than or equal to the difference threshold, it is determined that there is a color deviation between the aforementioned color vector and the aforementioned pre-assigned color vector (represented by the difference vector between the aforementioned color vector and the aforementioned pre-assigned color vector), if the aforementioned vector difference is less than the difference. If the threshold is set, it is determined that there is no color shift between the aforementioned color vector and the aforementioned pre-assigned color vector.

It should be noted that the value of the difference threshold is not specifically limited in the embodiments of the present application, and can be configured by those of ordinary skill in the art according to actual needs. The difference threshold may be configured as a fixed value, or the value of the difference threshold may be dynamically determined. value.

Optionally, in one embodiment, the image signal processing unit 230 is configured to determine a difference threshold based on the color shift direction of the color vector and the pre-assigned color vector, and determine whether the color vector and the pre-assigned color vector are not based on the difference threshold and the vector difference. There is a color cast.

In the embodiment of the present application, considering that the sensitivity of human eyes to different colors is different, the color shift direction is used to dynamically determine the value of the difference threshold.

Wherein, for different pre-assigned color vectors, according to the sensitivity of the human eye to different colors, in the embodiment of the present application, the corresponding relationship between the color shift direction and the difference threshold associated with each pre-assigned color vector is pre-established. For example, please refer to Figure 5. The ellipse in the figure represents the corresponding relationship between the color shift direction and the difference threshold associated with the pre-assigned color vectors. There are 18 ellipses, which represent the colors associated with each of the 18 pre-assigned color vectors. The correspondence between the bias direction and the difference threshold. Taking the ellipse numbered "7" as an example, for the pre-assigned color vector associated with the ellipse, the difference threshold corresponding to when the color shift direction is biased towards blue or red is significantly larger than that corresponding to when the color shift direction is biased towards green. , because the human eye is more sensitive to green.

As above, based on the established corresponding relationship between the color shift direction and the difference threshold associated with the pre-assigned color vector, when the image signal processing unit 230 performs color shift identification, it first determines that the color vector of the identified object in the shooting scene is compared with Based on the color shift direction of its pre-assigned color vector, then according to the corresponding relationship between the color shift direction and the difference threshold associated with the pre-assigned color vector, determine the difference threshold corresponding to the color shift direction of the aforementioned color vector, and then judge Whether the vector difference of the aforementioned color vectors is greater than or equal to the difference threshold corresponding to the aforementioned color shift direction, wherein, if the aforementioned vector difference of the aforementioned color vector is greater than or equal to the aforementioned difference threshold value corresponding to the aforementioned color shift direction, then it is determined that the aforementioned color vector and the aforementioned pre-assigned color The vector has a color shift (here, the aforementioned vector difference and color shift direction are used to characterize it), otherwise, it is determined that the aforementioned color vector and the aforementioned pre-assigned color vector do not have a color shift.

For example, referring to FIG. 6 , the image signal processing unit 230 determines that the color vector of the object "fire hydrant" and its pre-assigned color vector have a color shift by identifying the color shift of the object "fire hydrant" in the shooting scene shown in the figure. The arrow shown represents the color shift, wherein the direction of the arrow represents the color shift direction, and the length of the arrow represents the difference threshold, that is, the longer the length, the greater the difference threshold.

Optionally, in one embodiment, the image signal processing unit 230 is configured to calculate the difference vector between the color vector and the pre-assigned color vector, and use the difference vector as the color shift vector of the sampling pixel; The color shift vector is interpolated to obtain the color shift vector of the non-sampling pixels in the corrected image; and according to the color shift vector of each pixel in the corrected image, color restoration is performed on each pixel to obtain a restored image.

In this embodiment of the present application, the image signal processing unit 230 is configured to calculate the difference vector between the aforementioned color vector and the aforementioned pre-assigned color vector, and use the difference vector as the color shift vector of the sampling pixel. Then, the image signal processing unit 230 obtains the color shift vector of the non-sampled pixel points in the corrected image by interpolating according to the color shift vector of the sampled pixel points according to the pre-configured interpolation strategy. So far, the color shift vectors of all pixels in the corrected image, including sampling pixels and non-sampling pixels, are known, and each pixel can be color-coded according to the color shift vector of each pixel in the corrected image. Restoration processing to obtain a restored image.

The interpolation processing performed in the above embodiments of the present application can be understood as a process of estimating the color shift vectors of other pixel points in the entire corrected image through the known color shift vectors of discrete pixels.

For example, referring to FIG. 7 , in the corrected image on the upper side of FIG. 7 , the front image signal processor 200 has identified 8 different objects through the neural network processing unit 220 , and the black solid circles shown in FIG. 7 represent the corresponding The sampled pixels of these identified objects, and the arrows represent the corresponding color shift vectors. As shown in FIG. 7 , the image signal processing unit 230 obtains the color shift vectors of all pixels in the corrected image by interpolation according to the color shift vectors of the sampled pixels.

It should be noted that the interpolation strategy adopted is not specifically limited in the embodiments of the present application, and can be selected by those of ordinary skill in the art according to actual needs, including but not limited to recent field interpolation, bilinear interpolation or bicubic interpolation, etc.

Optionally, in one embodiment, the image signal processing unit 230 is configured to perform interpolation processing according to more than one interpolation algorithm according to the color shift vector of the sampled pixel point, to obtain multiple candidate color shift vectors of the non-sampled pixel point; And according to the multiple candidate color shift vectors, the color shift vectors of the non-sampling pixels are obtained.

Wherein, for the same non-sampling pixel point, the neighborhood pixel points selected by the application processor 130 during interpolation according to different interpolation strategies are different.

It should be noted that, in the embodiment of the present application, a single interpolation strategy is not used for interpolation processing, but multiple interpolation strategies are combined to perform interpolation processing. There is no specific limitation on which interpolation strategy is adopted and the number of interpolation strategies, which can be configured according to the processing capability of the image signal processing unit 230 in the art. In addition, the embodiment of the present application does not specifically limit which fusion strategy is adopted, and can be configured by those of ordinary skill in the art according to actual needs.

For example, three different interpolation strategies are pre-configured in the embodiment of the present application, which are denoted as interpolation strategy A, interpolation strategy B, and interpolation strategy C, respectively. When performing interpolation processing, for a non-sampling pixel point, the image signal processing unit 230 uses interpolation strategy A to obtain the candidate color shift vector A of the non-sampling pixel point according to the color shift vector of the sampled pixel point, and uses interpolation strategy B to interpolate the candidate color shift vector A. Obtain the candidate color shift vector B of the non-sampling pixel point, and use the interpolation strategy C to interpolate to obtain the candidate color shift vector C of the non-sampling pixel point. Finally, the image signal processing unit 230 fuses the candidate color shift vector A, the candidate color shift vector B, and the candidate color shift vector C into one vector according to the configured fusion strategy, which is used as the color shift vector of the non-sampling pixel point.

Optionally, in one embodiment, the image signal processing unit 230 is configured to calculate the average value of the components of the multiple candidate color shift vectors in each dimension, and obtain the color of the non-sampling pixel point according to the average value of each dimension component. bias vector; or

Weighted summation is performed on the components of the multiple candidate color shift vectors in each dimension, and the color shift vector of the non-sampling pixel point is obtained according to the weighted sum value of each dimension component.

Two optional fusion strategies are further provided in the embodiments of the present application.

First, for a non-sampling pixel, the image signal processing unit 230 calculates the average value of the components of its multiple candidate color shift vectors in each dimension, and obtains the color shift of the non-sampling pixel according to the average value of each dimension component. quantity.

Second, different interpolation strategies are pre-allocated with weights for weighted summation. For example, with the weight sum value being 1 as a constraint, if the accuracy of an interpolation strategy is higher, the assigned weight will be higher. For a non-sampling pixel, the image signal processing unit 230 performs weighted summation of the components of the multiple candidate color shift vectors of the non-sampling pixel in each dimension according to the weight corresponding to each interpolation strategy, The weighted sum of the components obtains the color shift vector of the unsampled pixel.

Optionally, in one embodiment, the image signal processing unit 230 is configured to obtain the recognition confidence corresponding to the aforementioned object, and perform correction processing on the color shift vector of the sampled pixel points according to the recognition confidence degree to obtain the corrected sample pixel point. color shift vector; and performing interpolation processing on the corrected color shift vector of the sampled pixel points to obtain the color shift vector of the non-sampled pixel points in the corrected image.

In the embodiment of the present application, the magnitude of the color restoration is determined according to the recognition confidence of the object recognized by the neural network recognition unit 220 .

Wherein, the image signal processing unit 230 first obtains the recognition confidence of the object in the shooting scene recognized by the neural network recognition unit 220, and performs correction processing on the color shift vector of the aforementioned sampling pixel points according to the recognition confidence, which can be expressed as:

V'=V*α;

Among them, V' represents the corrected color shift vector of the sampled pixel point, V represents the color shift vector of the sampled pixel point obtained by calculation, and α represents the recognition confidence of the aforementioned object.

After completing the correction of the color shift vector of the sampling pixel, the image signal processing unit 230 further performs interpolation processing according to the corrected color shift vector of the sampling pixel, so as to obtain the color shift vector of the non-sampling pixel in the corrected image , for details, reference may be made to the relevant descriptions in the above embodiments, which will not be repeated here.

The embodiment of the present application also provides an image processing method, the method includes:

Get the scene image of the shooting scene;

performing white balance correction on the scene image to obtain a corrected image;

Identify objects existing in the shooting scene according to the scene image;

When there is a color shift between the color vector of the object in the corrected image and the pre-assigned color vector of the object, color restoration processing is performed on the corrected image according to the color shift to obtain a restored image.

In some embodiments, the image processing method further includes:

Determine the sampling pixel points for color sampling from the object in the object area in the corrected image, and use the color vector of the sampling pixel point as the color vector of the object;

Calculate the vector difference between the color vector and the pre-assigned color vector, and determine whether there is a color shift between the color vector and the pre-assigned color vector according to the vector difference.

In some embodiments, determining whether there is a color shift between the color vector and the pre-assigned color vector according to the vector difference includes:

A difference threshold is determined based on the color shift direction between the color vector and the pre-assigned color vector, and whether there is a color shift between the color vector and the pre-assigned color vector is determined based on the difference threshold and the vector difference.

In some embodiments, performing color restoration processing on the corrected image according to the color shift to obtain a restored image includes:

Calculate the difference vector between the color vector and the pre-assigned color vector, and use the difference vector as the color shift vector of the sampling pixel;

Perform interpolation processing according to the color shift vector of the sampling pixel to obtain the color shift vector of the non-sampling pixel in the corrected image;

According to the color shift vector of each pixel in the corrected image, color restoration processing is performed on each pixel to obtain the restored image.

In some embodiments, performing interpolation processing according to the color shift vector of the sampled pixel points to obtain the color shift vector of the non-sampled pixel points in the corrected image, including:

Obtaining the recognition confidence level corresponding to the object, and performing correction processing on the color shift vector of the sampling pixel point according to the recognition confidence level, to obtain the corrected color shift vector of the sampling pixel point;

Perform interpolation processing according to the corrected color shift vector of the sampled pixel points to obtain the color shift vector of the non-sampled pixel points in the corrected image.

Please refer to FIG. 9 , the present application also provides an image processing method, as shown in FIG. 9 , the image processing method includes:

In 310, a scene image of the shooting scene is obtained;

In 320, performing white balance correction on the scene image to obtain a corrected image;

In 330, identifying objects existing in the shooting scene according to the scene image;

In 340, when there is a color shift between the color vector of the aforementioned object in the corrected image and the pre-assigned color vector of the aforementioned object, perform color restoration processing on the corrected image according to the color shift to obtain a restored image.

It should be noted that the execution sequence of 320 and 330 is not affected by the size of the sequence number, it may be that 320 is executed first and then 330 is executed, 330 may be executed first and then 320 is executed, or 320 and 330 may be executed simultaneously.

Optionally, in an embodiment, performing white balance correction on the scene image to obtain a corrected image, including:

Perform state statistics on the scene image to obtain the state information required for white balance correction;

Perform white balance correction on the scene image according to the aforementioned state information to obtain a corrected image.

Optionally, in an embodiment, the white balance correction is performed on the scene image according to the foregoing state information, and before the corrected image is obtained, the method further includes:

After the aforementioned state information is obtained by statistics, the first optimization processing is performed on the scene image;

Perform a second optimization process on the scene image after the first optimization process;

Perform white balance correction on the scene image according to the foregoing state information to obtain a corrected image, including:

Perform white balance correction on the scene image after the second optimization process according to the aforementioned state information to obtain a corrected image.

Optionally, in an embodiment, identifying objects existing in the shooting scene according to the scene image includes:

The object recognition model is used to perform object recognition on the scene image after the second optimization processing, so as to recognize the objects existing in the shooting scene.

Optionally, in an embodiment, the image processing method provided by this application further includes:

Determine the sampling pixel points for color sampling from the aforementioned object in the object area in the corrected image, and use the color vector of the sampling pixel point as the color vector of the object;

Calculate the vector difference between the color vector and the pre-assigned color vector, and judge whether there is a color shift between the color vector and the pre-assigned color vector according to the vector difference.

Optionally, in one embodiment, determining whether there is a color shift between the color vector and the pre-assigned color vector according to the vector difference includes:

The difference threshold is determined based on the color shift direction of the color vector and the pre-assigned color vector, and whether there is a color shift between the color vector and the pre-assigned color vector is determined based on the difference threshold and the vector difference.

Optionally, in one embodiment, performing color restoration processing on the corrected image according to the color shift to obtain a restored image, including:

Calculate the difference vector between the color vector and the pre-allocated color vector, and use the difference vector as the color shift vector of the sampling pixel;

Perform interpolation processing according to the color shift vector of the sampled pixel points to obtain the color shift vector of the non-sampled pixel points in the corrected image;

According to the color shift vector of each pixel in the corrected image, color restoration is performed on each pixel to obtain a restored image.

Optionally, in an embodiment, interpolation processing is performed according to the color shift vector of the sampled pixels to obtain the color shift vector of the non-sampled pixels in the corrected image, including:

Perform interpolation processing according to more than one interpolation algorithm according to the color shift vector of the sampled pixel point to obtain a plurality of candidate color shift vectors of the non-sampled pixel point;

According to the multiple candidate color shift vectors, the color shift vectors of the non-sampled pixels are obtained.

Optionally, in an embodiment, the color shift vector of the non-sampling pixel is obtained according to a plurality of candidate color shift vectors, including:

Calculate the average value of the components of multiple candidate color shift vectors in each dimension, and obtain the color shift vector of the non-sampling pixel point according to the average value of each dimension component; or

Weighted summation is performed on the components of the multiple candidate color shift vectors in each dimension, and the color shift vector of the non-sampling pixel point is obtained according to the weighted sum value of each dimension component.

Optionally, in one embodiment, interpolation processing is performed according to the color shift vector of the sampled pixel points to obtain the color shift vector of the non-sampled pixel points in the corrected image, including:

Obtaining the recognition confidence level corresponding to the object, and correcting the color shift vector of the sampled pixel points according to the recognition confidence level to obtain the corrected color shift vector of the sampled pixel point;

Interpolate according to the corrected color shift vector of the sampled pixel points to obtain the color shift vector of the non-sampled pixel points in the corrected image.

It should be noted that the image processing method provided by this application can be executed by the electronic device provided by this application, and it can also be executed by the pre-image signal processor provided by this application. The relevant description of the device or the front image signal processor will not be repeated here.

The electronic device, the front image signal processor, and the image processing method provided by the embodiments of the present application have been described in detail above. The principles and implementations of the present application are described herein by using specific examples, and the descriptions of the above embodiments are only used to help the understanding of the present application. At the same time, for those skilled in the art, according to the idea of the application, there will be changes in the specific embodiments and application scope. In summary, the content of this specification should not be construed as a limitation to the application.

Claims (20)

An electronic device comprising: A camera, used to collect scene images of the shooting scene; a front image signal processor, configured to identify objects existing in the shooting scene according to the scene image; An application processor, configured to perform white balance correction on the scene image to obtain a corrected image; and when there is a color shift between the color vector of the object in the corrected image and the pre-assigned color vector of the object, according to the The color cast performs color restoration processing on the corrected image to obtain a restored image. The electronic device according to claim 1, wherein the pre-image signal processor is configured to perform state statistics on the scene image to obtain state information required by the application processor to perform white balance correction; and The scene image is optimized to obtain an optimized scene image; The application processor is configured to perform white balance correction on the optimized scene image according to the state information to obtain the corrected image. The electronic device according to claim 2, wherein the front image signal processor comprises: an image signal processing unit, configured to perform state statistics on the scene image to obtain the state information; and perform a first optimization process on the scene image; A neural network processing unit for performing a second optimization process on the scene image after the first optimization process; and performing object recognition on the scene image after the second optimization process through an object recognition model, so as to identify the shooting scene objects that exist in . 1. The electronic device according to claim 1, wherein the application processor is configured to determine sampling pixels for color sampling from the object within the object area in the corrected image, and use the sampling pixels to The color vector of the object is used as the color vector of the object; and the vector difference between the color vector and the pre-assigned color vector is calculated, and whether there is a color shift between the color vector and the pre-assigned color vector is judged according to the vector difference. . 4. The electronic device of claim 4, wherein the application processor is configured to determine a difference threshold based on a color shift direction of the color vector and the pre-assigned color vector, and to determine a difference threshold based on the difference threshold and the vector difference Determine whether there is a color deviation between the color vector and the pre-assigned color vector. The electronic device according to claim 4, wherein the application processor is configured to calculate a difference vector between the color vector and the pre-assigned color vector, and use the difference vector as the color of the sampled pixel. and performing interpolation processing according to the color shift vector of the sampled pixels to obtain the color shift vector of the non-sampled pixels in the corrected image; and according to the color shift vector of each pixel in the corrected image, to Each pixel is subjected to color restoration processing to obtain the restored image. The electronic device according to claim 6, wherein the application processor is configured to perform interpolation processing according to more than one interpolation algorithm according to the color shift vector of the sampled pixel points, to obtain a plurality of the non-sampled pixel points candidate color shift vectors; and obtaining color shift vectors of the non-sampled pixel points according to the plurality of candidate color shift vectors. The electronic device according to claim 7, wherein the application processor is configured to calculate an average value of components of the plurality of candidate color shift vectors in each dimension, and obtain the The color shift vector of the sampled pixel; or Weighted summation is performed on components of the plurality of candidate color shift vectors in each dimension, and the color shift vector of the non-sampled pixel point is obtained according to the weighted sum value of each dimension component. The electronic device according to claim 6, wherein the application processor is configured to obtain the recognition confidence corresponding to the object, and perform correction processing on the color shift vector of the sampling pixel point according to the recognition confidence, to obtain the corrected color shift vector of the sampling pixel; and performing interpolation processing according to the corrected color shift vector of the sampling pixel to obtain the color shift vector of the non-sampling pixel in the corrected image. A front image signal processor, comprising: A data interface for obtaining a scene image of a shooting scene from a camera; and transmitting the scene image to an application processor for white balance correction, and receiving a corrected image returned after the application processor performs white balance correction; a neural network processing unit, configured to recognize the scene image through an object recognition model, so as to recognize the object existing in the shooting scene; An image signal processing unit, configured to perform color restoration processing on the corrected image according to the color deviation when the color vector of the object in the corrected image and the pre-assigned color vector of the object have a color deviation, to obtain Restore the image. The front-end image signal processor according to claim 10, wherein the image signal processing unit is configured to determine the sampling pixel points for color sampling from the object in the object area in the corrected image, and Taking the color vector of the sampling pixel as the color vector of the object; and calculating the vector difference between the color vector and the pre-allocated color vector, and judging the color vector and the pre-allocation according to the vector difference Whether the color vector has a color cast. The front-end image signal processor according to claim 11, wherein the image signal processing unit is configured to determine a difference threshold based on the color shift direction of the color vector and the pre-assigned color vector, and based on the difference threshold and the vector difference to determine whether there is a color shift between the color vector and the pre-assigned color vector. The front-end image signal processor according to claim 10, wherein the image signal processing unit is configured to calculate a difference vector between the color vector and the pre-assigned color vector, and use the difference vector as the The color shift vector of the sampled pixels; and performing interpolation processing according to the color shift vector of the sampled pixels to obtain the color shift vector of the non-sampled pixels in the corrected image; and according to each pixel in the corrected image The color shift vector of , and performing color restoration processing on each pixel to obtain the restored image. The front-end image signal processor according to claim 13, wherein the image signal processing unit is configured to perform interpolation processing according to more than one interpolation algorithm according to the color shift vector of the sampled pixel points to obtain the non- multiple candidate color shift vectors of the sampling pixel; and obtaining the color shift vector of the non-sampled pixel according to the multiple candidate color shift vectors. The front-end image signal processor according to claim 13, wherein the image signal processing unit is configured to obtain the recognition confidence corresponding to the object, and to determine the color deviation of the sampling pixel according to the recognition confidence. performing correction processing on the sampled pixel point to obtain the corrected color shift vector of the sampled pixel point; and performing interpolation processing according to the corrected color shift vector of the sampled pixel point to obtain the color shift vector of the non-sampled pixel point in the corrected image. An image processing method, comprising: Get the scene image of the shooting scene; performing white balance correction on the scene image to obtain a corrected image; Identify objects existing in the shooting scene according to the scene image; When there is a color shift between the color vector of the object in the corrected image and the pre-assigned color vector of the object, a color restoration process is performed on the corrected image according to the color shift to obtain a restored image. The image processing method according to claim 16, wherein the image processing method further comprises: Determine the sampling pixel points for color sampling from the object in the object area in the corrected image, and use the color vector of the sampling pixel point as the color vector of the object; Calculate the vector difference between the color vector and the pre-assigned color vector, and determine whether there is a color shift between the color vector and the pre-assigned color vector according to the vector difference. The image processing method according to claim 17, wherein the determining whether there is a color shift between the color vector and the pre-assigned color vector according to the vector difference comprises: A difference threshold is determined based on the color shift direction of the color vector and the pre-assigned color vector, and whether there is a color shift between the color vector and the pre-assigned color vector is determined based on the difference threshold and the vector difference. The image processing method according to claim 16, wherein the performing color restoration processing on the corrected image according to the color shift to obtain a restored image comprises: Calculate the difference vector of the color vector and the pre-assigned color vector, and use the difference vector as the color shift vector of the sampling pixel; Perform interpolation processing according to the color shift vector of the sampling pixel to obtain the color shift vector of the non-sampling pixel in the corrected image; According to the color shift vector of each pixel in the corrected image, color restoration processing is performed on each pixel to obtain the restored image. The image processing method according to claim 19, wherein the performing interpolation processing according to the color shift vector of the sampling pixel points to obtain the color shift vector of the non-sampling pixel points in the corrected image, comprising: Obtaining the recognition confidence level corresponding to the object, and performing correction processing on the color shift vector of the sampling pixel point according to the recognition confidence level, to obtain the corrected color shift vector of the sampling pixel point; Perform interpolation processing according to the corrected color shift vector of the sampled pixel points to obtain the color shift vector of the non-sampled pixel points in the corrected image.

Images