CN113610884B - Image processing method, device, electronic equipment and computer readable storage medium - Google Patents

Abstract

The embodiment of the application discloses an image processing method, an image processing device, electronic equipment and a computer readable storage medium. The method comprises the following steps: identifying a foreground region in the first image to obtain a first foreground identification result; performing blurring processing on the first image based on the first foreground identification result to obtain a first blurring image; determining one or more image areas selected in the first virtual image by the selection operation in response to the selection operation for the first virtual image; identifying foreground areas of the image areas to obtain second foreground identification results corresponding to the image areas; and carrying out blurring processing on the first image or the first blurring image based on a second foreground identification result of each image area to obtain a second blurring image. The image processing method, the device, the electronic equipment and the computer readable storage medium can improve the accuracy of image foreground identification and improve the blurring effect of images.

Description

Image processing method, device, electronic device and computer readable storage medium

Technical Field

The present application relates to the field of imaging technology, and in particular to an image processing method, device, electronic device and computer-readable storage medium.

Background technique

After an electronic device captures an image through an imaging device (such as a camera), in order to highlight the object of interest in the image, it divides the image into a foreground area and a background area, and blurs the background area of the image to achieve an image effect that highlights the object of interest in the foreground area. For some images where the foreground area and the background area are easily confused, it is easy to mistakenly blur part of the foreground area or omit part of the background area from being blurred, resulting in a poor blurring effect of the image.

Summary of the invention

The embodiments of the present application disclose an image processing method, device, electronic device and computer-readable storage medium, which can improve the accuracy of image foreground recognition and enhance the blurring effect of the image.

The present application discloses an image processing method, including:

Identify a foreground area in the first image to obtain a first foreground identification result;

Performing blur processing on the first image based on the first foreground recognition result to obtain a first blur image;

In response to a selection operation on the first blurred image, determining one or more image areas selected in the first blurred image by the selection operation;

Identify the foreground area of each of the image areas to obtain a second foreground identification result corresponding to each of the image areas;

Based on the second foreground recognition results of each of the image regions, blurring is performed on the first image or the first blurred image to obtain a second blurred image.

The present application discloses an image processing device, including:

A first recognition module, used to recognize a foreground area in the first image and obtain a first foreground recognition result;

A blur module, configured to blur the first image based on the first foreground recognition result to obtain a first blurred image;

an area selection module, configured to respond to a selection operation on the first blurred image and determine one or more image areas selected by the selection operation in the first blurred image;

A second recognition module, used to recognize the foreground area of each of the image areas, and obtain a second foreground recognition result corresponding to each of the image areas;

The blurring module is further configured to perform blurring processing on the first blurred image based on the second foreground recognition results of each of the image regions to obtain a second blurred image.

An embodiment of the present application discloses an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor implements the method described above.

An embodiment of the present application discloses a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method described above is implemented.

The image processing method, device, electronic device and computer-readable storage medium disclosed in the embodiments of the present application, after blurring the first image based on the first foreground recognition result to obtain the first blurred image, in response to the selection operation on the first blurred image, determine one or more image areas selected by the selection operation in the first blurred image, identify the foreground area of each image area, obtain the second foreground recognition result corresponding to each image area, and then blur the first image or the first blurred image based on the second foreground recognition result of each image area to obtain the second blurred image. After the first blurred image is initially blurred, the user can select the image area that needs to be further optimized, and perform foreground recognition on the selected image area again, thereby improving the accuracy of foreground recognition, and performing secondary blurring on the first image or the first blurred image based on the more accurate second foreground recognition result can improve the situation where part of the foreground area is mistakenly blurred or part of the background area is omitted and not blurred, thereby improving the blurring effect of the image. In addition, the user can select the image area that needs to be further optimized in the first blurred image, which meets the different needs of the user and improves the interactivity with the user.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

FIG1 is a block diagram of an image processing circuit in one embodiment;

FIG2 is a flow chart of an image processing method in one embodiment;

FIG3A is a schematic diagram of performing a selection operation on a first blurred image in one embodiment;

FIG3B is a schematic diagram showing a selection box in one embodiment;

FIG3C is a schematic diagram of performing a selection operation on a first blurred image in another embodiment;

FIG3D is a schematic diagram showing a selection box in another embodiment;

FIG3E is a schematic diagram of adjusting the size of a selection box in one embodiment;

FIG4 is a flow chart of an image processing method in another embodiment;

FIG5 is a schematic diagram of correcting a first foreground recognition result using a second foreground recognition result of a selected image region in one embodiment;

FIG6 is a schematic diagram of correcting a first depth map using a second foreground recognition result of a selected image region in one embodiment;

FIG7 is a flow chart of an image processing method in another embodiment;

FIG8 is a schematic diagram of obtaining a local hair cutout result corresponding to an image area in one embodiment;

FIG9 is a block diagram of an image processing apparatus according to an embodiment;

FIG. 10 is a structural block diagram of an electronic device in one embodiment.

Detailed ways

The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

It should be noted that the terms "including" and "having" and any variations thereof in the embodiments of the present application and the accompanying drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device including a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products or devices.

It is understood that the terms "first", "second", etc. used in this application may be used herein to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish a first element from another element. For example, without departing from the scope of this application, a first foreground recognition result may be referred to as a second foreground recognition result, and similarly, a second foreground recognition result may be referred to as a first foreground recognition result. Both the first foreground recognition result and the second foreground recognition result are foreground recognition results, but they are not the same foreground recognition result.

An embodiment of the present application provides an electronic device. The electronic device includes an image processing circuit, which can be implemented using hardware and/or software components and can include various processing units that define an ISP (Image Signal Processing) pipeline. FIG1 is a block diagram of an image processing circuit in one embodiment. For ease of explanation, FIG1 only shows various aspects of image processing technology related to an embodiment of the present application.

As shown in FIG1 , the image processing circuit includes an ISP processor 140 and a control logic 150. The image data captured by the imaging device 110 is first processed by the ISP processor 140, which analyzes the image data to capture image statistics that can be used to determine one or more control parameters of the imaging device 110. The imaging device 110 may include one or more lenses 112 and an image sensor 114. The image sensor 114 may include a color filter array (such as a Bayer filter), and the image sensor 114 may obtain light intensity and wavelength information captured by each imaging pixel, and provide a set of raw image data that can be processed by the ISP processor 140. The attitude sensor 120 (such as a three-axis gyroscope, a Hall sensor, an accelerometer, etc.) may provide the acquired image processing parameters (such as anti-shake parameters) to the ISP processor 140 based on the interface type of the attitude sensor 120. The attitude sensor 120 interface may use a SMIA (Standard Mobile Imaging Architecture) interface, other serial or parallel camera interfaces, or a combination of the above interfaces.

It should be noted that, although only one imaging device 110 is shown in FIG1 , in the embodiment of the present application, at least two imaging devices 110 may be included, each imaging device 110 may correspond to one image sensor 114, or multiple imaging devices 110 may correspond to one image sensor 114, which is not limited here. The working process of each imaging device 110 may refer to the above description.

In addition, the image sensor 114 may also send raw image data to the gesture sensor 120 , and the gesture sensor 120 may provide the raw image data to the ISP processor 140 based on the interface type of the gesture sensor 120 , or the gesture sensor 120 may store the raw image data in the image memory 130 .

The ISP processor 140 processes the raw image data pixel by pixel in a variety of formats. For example, each image pixel may have a bit depth of 8, 10, 12, or 14 bits, and the ISP processor 140 may perform one or more image processing operations on the raw image data and collect statistical information about the image data. The image processing operations may be performed at the same or different bit depth precisions.

The ISP processor 140 may also receive image data from the image memory 130. For example, the gesture sensor 120 interface sends raw image data to the image memory 130, and the raw image data in the image memory 130 is then provided to the ISP processor 140 for processing. The image memory 130 may be a portion of a memory device, a storage device, or an independent dedicated memory within an electronic device, and may include a DMA (Direct Memory Access) feature.

When receiving raw image data from the image sensor 114 interface or from the posture sensor 120 interface or from the image memory 130, the ISP processor 140 may perform one or more image processing operations, such as time domain filtering. The processed image data may be sent to the image memory 130 for further processing before being displayed. The ISP processor 140 receives the processed data from the image memory 130 and performs image data processing in the original domain and in the RGB and YCbCr color spaces on the processed data. The image data processed by the ISP processor 140 may be output to the display 160 for viewing by the user and/or further processed by a graphics engine or GPU (Graphics Processing Unit). In addition, the output of the ISP processor 140 may also be sent to the image memory 130, and the display 160 may read the image data from the image memory 130. In one embodiment, the image memory 130 may be configured to implement one or more frame buffers.

The statistical data determined by the ISP processor 140 may be sent to the control logic 150. For example, the statistical data may include image sensor 114 statistical information such as vibration frequency of a gyroscope, automatic exposure, automatic white balance, automatic focus, flicker detection, black level compensation, lens 112 shading correction, etc. The control logic 150 may include a processor and/or microcontroller that executes one or more routines (such as firmware), and the one or more routines may determine control parameters of the imaging device 110 and control parameters of the ISP processor 140 based on the received statistical data. For example, the control parameters of the imaging device 110 may include attitude sensor 120 control parameters (such as gain, integration time for exposure control, anti-shake parameters, etc.), camera flash control parameters, camera anti-shake displacement parameters, lens 112 control parameters (such as focal length for focus or zoom), or a combination of these parameters. The ISP control parameters may include gain levels and color correction matrices for automatic white balance and color adjustment (for example, during RGB processing), and lens 112 shading correction parameters.

Exemplarily, the image processing method provided by the embodiment of the present application is described in conjunction with the image processing circuit of FIG. 1. The ISP processor 140 may obtain a first image from the imaging device 110 or the image memory 130, identify a foreground area in the first image, obtain a first foreground recognition result, and perform blur processing on the first image based on the first foreground recognition result to obtain a first blurred image. The ISP processor 140 may output the first blurred image to the display 160 for display. The user may select an image area to be optimized according to the first blurred image displayed by the display 160, and the ISP processor 140 may respond to a selection operation for the first blurred image, determine one or more image areas selected by the selection operation in the first blurred image, and identify a foreground area of each image area, obtain a second foreground recognition result corresponding to each image area, and then perform blur processing on the first blurred image based on the second foreground recognition result of each image area to obtain a second blurred image. Optionally, the ISP processor 140 may output the second blurred image to the display 160 for display, or store the second blurred image in the image memory 130.

As shown in FIG2 , in one embodiment, an image processing method is provided, which can be applied to the above-mentioned electronic devices, which may include but are not limited to mobile phones, smart wearable devices, tablet computers, PCs (Personal Computers), vehicle-mounted terminals, digital cameras, etc., and the present application embodiment does not limit this. The image processing method may include the following steps:

Step 210: Identify a foreground area in the first image to obtain a first foreground identification result.

The first image may include a foreground area and a background area, wherein the foreground area may refer to the image area where the target object in the first image is located, and the background area may refer to the image area other than the target object in the first image. The target object may be an object of interest in the first image, for example, the first image may be a person image, and the target object may be a person portrait in the person image; the first image may be an animal image, and the target object may be an animal in the animal image, etc.; the first image may be a building image, and the target object may be a building in the building image, etc., but is not limited thereto.

The first image may be a color image, for example, an image in RGB (Red Green Blue) format or an image in YUV (Y represents brightness, U and V represent chrominance) format, etc. The first image may be an image pre-stored in a memory of the electronic device, or may be an image captured in real time by a camera of the electronic device.

The electronic device may perform foreground recognition on the first image to obtain a first foreground recognition result, and the first foreground recognition result may be used to mark the foreground area in the first image. As an embodiment, the electronic device may obtain depth information of each pixel in the first image, and the depth information may be used to characterize the distance between the photographed object and the camera. The larger the depth information, the farther the distance. The depth information corresponding to the foreground area and the background area is quite different. Therefore, the depth information corresponding to each pixel in the first image may be used to divide the foreground area and the background area in the first image. For example, the background area may be an area composed of pixels whose depth information is greater than a first threshold, and the foreground area may be an area composed of pixels whose depth information is less than a second threshold, and so on.

In some embodiments, the electronic device may extract image features of the first image and analyze the image features to determine the foreground area of the first image. Optionally, the image features may include but are not limited to edge features, color features, position features, etc.

In some embodiments, the electronic device may also use a neural network to determine the foreground area in the first image, and the first image may be input into a pre-trained object segmentation model, and the target object contained in the first image may be identified through the object segmentation model to obtain the foreground area corresponding to the target object. The object segmentation model may be trained based on multiple groups of sample training images, each group of sample training images may include sample images, and each sample image may be annotated with a foreground area. The image segmentation model may include but is not limited to a network based on the deeplab semantic segmentation algorithm, a U-Net network structure, FCN (Fully Convolutional Networks), etc., which are not limited here.

It should be noted that the electronic device may also use other methods to identify the foreground area in the first image, and the method of identifying the foreground area is not limited in the embodiment of the present application.

Step 220: blurring the first image based on the first foreground recognition result to obtain a first blurred image.

The electronic device can determine the foreground area and background area of the first image based on the first foreground recognition result, and blur the background area in the first image to obtain a first blurred image, wherein the blurring can be implemented by using Gaussian filters, mean blur processing, median blur processing, etc., which are not limited here.

In some embodiments, the electronic device may also first blur the first image, and then, based on the first foreground recognition result, fuse the blurred first image with the first image before blurring to obtain the first blurred image. The fusion method may include but is not limited to fusion by taking the average, fusion by assigning different weight coefficients, Alpha fusion processing, etc. Taking Alpha fusion processing as an example, Alpha fusion processing may assign an Alpha value to each pixel in the first image before blurring and the first image after blurring, so that the first image before blurring and the first image after blurring have different transparencies. The first foreground recognition result may be used as the Alpha value of the first image after blurring, and the first image after blurring may be fused with the first image before blurring.

In some embodiments, the electronic device may perform depth estimation on the first image to obtain a depth estimation result of the first image, and the depth estimation result may include depth information of each pixel in the first image. The first image may be divided into regions according to the depth estimation result, and pixels with the same or similar depth information may be divided into the same image region. The blur parameters corresponding to each image region may be determined according to the depth information of the pixels in each divided image region, and then blur processing may be performed on each image region according to the blur parameters corresponding to each image region. The blur parameters may be used to describe the degree of blur, for example, may include parameters such as blur strength and blur coefficient. An image region with greater depth information may correspond to a greater degree of blur, and an image region with less depth information may correspond to a smaller degree of blur, so that different image regions may be blurred to different degrees.

Step 230 , in response to a selection operation on the first blurred image, determining one or more image regions selected by the selection operation in the first blurred image.

After the electronic device performs blurring processing on the first image, the first blurred image can be displayed on the display device, and the user can view the first blurred image and select the image area to be optimized. Optionally, the image area to be optimized can be a boundary between the foreground area and the background area, which is prone to blurring errors or missed blurring, such as the boundary between the hair and the background in a portrait image, or a gap in the hair.

The electronic device may respond to the selection operation on the first blurred image and determine one or more image areas selected in the first blurred image by the selection operation. The selection operation may include but is not limited to a variety of interactive operation modes such as touch operation, voice operation, sight interaction operation, and gesture operation.

As an implementation mode, the selection operation may be a touch operation performed by a user on a touch screen. The electronic device may obtain one or more touch positions of the selection operation on the screen, and for each touch position, a selection box corresponding to each touch position may be formed according to the area size, and the image area corresponding to each selection box in the first blurred image may be determined. The touch position of the selection operation on the screen may include touch coordinates, and the user may perform multiple touches on the screen to select multiple image areas that need to be optimized at the same time. The electronic device may obtain multiple touch positions at the same time to determine multiple image areas, and then optimize the blurring of the multiple image areas selected by the user (i.e., multiple touches and one optimization method). The electronic device may also obtain the currently detected touch position each time it detects that the user performs a selection operation to obtain the corresponding image area, and optimize the blurring of the image area (i.e., multiple touches and multiple optimization methods).

The above-mentioned area size can be a fixed size pre-set by the user according to actual needs, or it can be a fixed size uniformly set by the R&D personnel of the electronic device before leaving the factory. The area size can also be dynamically adjusted according to the image resolution, image size, etc. of the first blurred image. For example, if the image size is larger, the corresponding area size can also be larger, etc., but it is not limited to this. After determining each touch position, the electronic device can form a selection box corresponding to each touch position based on the area size. The selection box can be any shape such as a rectangle, square, polygon, circle, etc., which is not limited here. The touch position can be at a specific position of the selection box, for example, it can be at the center of the selection box, or it can be at a corner point of the selection box (such as the upper left corner point, the upper right corner point, etc.).

The electronic device may display a selection box on the screen in a preset display mode (such as a preset color, line, etc.). The image content in the selection box in the first blurred image is the selected image area. The user can intuitively know the selected image area through the displayed selection box. In some embodiments, the user can adjust the size of the selection box according to actual needs. If the electronic device detects an adjustment operation triggered for a target selection box, the size of the target selection box may be adjusted according to the adjustment operation. The target selection box refers to the selection box whose size the user needs to adjust. The operation mode of the adjustment operation may be different from the operation mode of the selection operation. For example, the selection operation may be a single-click operation, and the adjustment operation may be a sliding operation, or the selection operation may be a single-click operation, and the adjustment operation may be a double-click operation, etc., but is not limited thereto.

For example, please refer to FIG. 3A and FIG. 3B , FIG. 3A is a schematic diagram of selecting a first blurred image in an embodiment, and FIG. 3B is a schematic diagram of displaying a selection box in an embodiment. As shown in FIG. 3A and FIG. 3B , the electronic device 10 can display a first blurred image 310 on the screen, and the user can select the area to be optimized and blurred according to actual needs, and can select by touch. The electronic device 10 can display a selection box 320 on the screen according to the touch position of the user, and the image content in the selection box 320 is the selected image area.

As another example, please refer to Figures 3C and 3D. Figure 3C is a schematic diagram of performing a selection operation on the first blurred image in another embodiment, and Figure 3D is a schematic diagram of displaying a selection box in another embodiment. As shown in Figures 3C and 3D, the electronic device 10 can display a first blurred image 330 on the screen, and the user can perform multiple touch operations in the first blurred image 310 according to actual needs. The electronic device 10 can display a selection box 340 corresponding to each touch position on the screen according to the multiple touch positions touched by the user, and the image content in each selection box 340 is the selected image area. Figure 3E is a schematic diagram of adjusting the size of the selection box in one embodiment. As shown in Figure 3E, the user can adjust the size of the selection box 340 according to actual needs, thereby adjusting the selected image area.

Step 240 , identifying the foreground area of each image area, and obtaining a second foreground identification result corresponding to each image area.

After determining one or more image areas selected by the user, the electronic device may re-identify the foreground areas of each image area respectively, and obtain a second foreground recognition result corresponding to each image area. Optionally, since the first blurred image is an image that has been blurred, in order to ensure the accuracy of foreground recognition, after determining each image area, the electronic device may crop the regional image at the same image position from the first image according to the image position of each image area in the first blurred image, and perform foreground recognition on the regional image cropped from the first image, so as to obtain a second foreground recognition result corresponding to each image area. Since the image area is only a local area in the first blurred image, the cropped regional image is also a local image in the first image. By re-performing foreground recognition on the local image, a more refined and accurate second foreground recognition result can be obtained to correct the first foreground recognition result and improve the accuracy of foreground recognition.

Step 250: Based on the second foreground recognition results of each image area, blurring is performed on the first image or the first blurred image to obtain a second blurred image.

In some embodiments, after obtaining the second foreground recognition results corresponding to each image area, the electronic device may correct the first foreground recognition results according to the second foreground recognition results corresponding to each image area to obtain a corrected target foreground recognition result, and blur the first image based on the target foreground recognition result to obtain a second blurred image. Since the recognition accuracy of the foreground and background of the target foreground recognition result is higher, a second blurred image with a better blurring effect can be obtained.

In some embodiments, after obtaining the second foreground recognition results corresponding to each image area, the electronic device may also directly perform blur processing on each image area in the first blurred image according to the second foreground recognition results corresponding to each image area to obtain the second blurred image. Performing a secondary blur processing only on each image area selected in the first blurred image can improve the blurring effect while reducing the amount of calculation and improving the processing effect.

In the embodiment of the present application, after the first blurring process is performed on the first image, the user can select the image area that needs to be further optimized, and perform foreground recognition on the selected image area again, thereby improving the accuracy of foreground recognition, and performing secondary blurring on the first image or the first blurred image based on the more accurate second foreground recognition result, thereby improving the situation where part of the foreground area is mistakenly blurred or part of the background area is omitted and not blurred, thereby improving the blurring effect of the image. In addition, the user can select the image area that needs to be further optimized in the first blurred image, thereby meeting the different needs of the user and improving the interactivity with the user.

As shown in FIG. 4 , in another embodiment, an image processing method is provided. The method can be applied to the above-mentioned electronic device. The method may include the following steps:

Step 402: Identify a foreground area in a first image to obtain a first foreground identification result.

The description of step 402 may refer to the description of step 210 in the above embodiment, which will not be repeated here.

Step 404: perform depth estimation on the first image to obtain a depth estimation result.

The electronic device may perform depth estimation on the first image, determine the depth information of each pixel in the first image, and obtain a depth estimation result. The method in which the electronic device performs depth estimation on the first image may be a software depth estimation method, or a method in which the depth information is calculated in combination with a hardware device, etc. The software depth estimation method may include but is not limited to a method of performing depth estimation using a neural network such as a depth estimation model, which may be obtained by training a depth training set, and the depth training set may include multiple sample images and a depth map corresponding to each sample image, etc. The depth estimation method combined with the hardware device may include but is not limited to depth estimation using multiple cameras (such as dual cameras), depth estimation using structured light, depth estimation using TOF (Time of flight, time of flight), etc. The embodiment of the present application does not limit the method of depth estimation.

It should be noted that the execution order of step 402 and step 404 is not limited here, and step 404 may be executed first and then step 402, or step 402 and step 404 may be executed simultaneously.

Step 406 , blurring the first image according to the first foreground recognition result and the depth estimation result to obtain a first blurred image.

The depth estimation result of the first image may include the depth information of each pixel in the first image, and the first image may be divided into a plurality of image blocks according to the depth information of each pixel, so as to determine the blur parameter corresponding to each image block. For example, the depth information of each pixel divided into the same image block may belong to the same depth value interval, or the difference between the depth information of each pixel divided into the same image block is less than the depth threshold, etc.

The electronic device may divide the first image into a foreground area and a background area according to the depth information of each pixel. Since the foreground area and the background area divided in the depth estimation result are divided based on the depth information, the edge of the foreground area may not be accurate enough, while the foreground area identified by the first foreground recognition result is more accurate. Therefore, in some embodiments, the depth estimation result may be corrected according to the first foreground recognition result to obtain a first depth map. Optionally, the edge information of the foreground area divided in the depth estimation result may be adjusted based on the first foreground recognition result to obtain a first depth map, and then the first depth map may be used to blur the first image to obtain a first blurred image. The edge information may include the coordinates of the pixel points marked as edge pixels.

Optionally, the electronic device may compare the first foreground recognition result with the foreground area divided in the depth estimation result to determine whether the edge information of the foreground area in the first foreground recognition result is consistent with the edge information of the foreground area in the depth estimation result. If not, the edge information of the foreground area in the depth estimation result may be directly modified to the edge information of the foreground area in the first foreground recognition result, or the edge information of the foreground area in the depth estimation result may be fused with the edge information of the foreground area in the first foreground recognition result. Optionally, the fusion method may include but is not limited to pixel mean fusion, fusion according to different weight coefficients, etc. Since the recognition accuracy of the foreground area by the first foreground recognition result is greater than that of the depth estimation result, the weight coefficient corresponding to the edge information of the foreground area in the first foreground recognition result may be greater than the weight coefficient of the edge information of the foreground area in the depth estimation result. It should be noted that other methods may also be used to correct and adjust the depth estimation result, which are not limited here.

After obtaining the first depth map, the electronic device can divide the background area according to the foreground area and the background area divided in the first depth map, and the depth information of each pixel in the background area, to obtain multiple background sub-areas. The electronic device can then determine the blur parameters corresponding to each background sub-area based on the depth information of the pixels in each background sub-area, so as to perform blur processing with different blur strengths on each background sub-area according to the blur parameters corresponding to each background sub-area.

Step 408 , in response to a selection operation on the first blurred image, determining one or more image regions selected by the selection operation in the first blurred image.

Step 410: Identify the foreground area of each image area to obtain a second foreground identification result corresponding to each image area.

The description of steps 408 to 410 may refer to the description of steps 230 to 240 in the above embodiment, which will not be repeated here.

Step 412: The second foreground recognition result corresponding to each image region is merged with the first foreground recognition result to obtain a target foreground recognition result.

In some embodiments, the electronic device may correct the first foreground recognition result according to the second foreground recognition result corresponding to each selected image area, and merge the second foreground recognition result corresponding to each image area with the first foreground recognition result to obtain a more accurate target foreground recognition result.

As a specific implementation, the electronic device may replace the foreground recognition results corresponding to each image area in the first foreground recognition result with the second foreground recognition results corresponding to each image area, respectively, to obtain a target foreground recognition result. The first foreground recognition result may include a foreground mask of the first image, and the foreground mask may be used to mark the position of the foreground area of the first image. After determining each image area selected by the user, the electronic device may determine, based on the image position of each image area in the first blurred image, a mask area in the foreground mask having the same image position as each image area, that is, the mask area corresponding to each image area. The second foreground recognition result of each image area may include a local foreground mask corresponding to each image area, and the mask area corresponding to each image area in the foreground mask of the first image may be replaced with the corresponding local foreground mask to obtain a more accurate target foreground mask, which may be used as the target foreground recognition result.

FIG5 is a schematic diagram of correcting a first foreground recognition result using a second foreground recognition result of a selected image area in one embodiment. As shown in FIG5 , the electronic device performs foreground recognition on a first image 510 to obtain a first foreground recognition result 520, and can perform blur processing on the first image 510 based on the first foreground recognition result 520 and the depth estimation result of the first image 510 to obtain a first blurred image 530. The user can select an image area 532 to be optimized in the first blurred image 530 displayed on the screen of the electronic device, and can perform local foreground recognition on the image area 532 to obtain a second foreground recognition result 540 corresponding to the image area 532. The foreground recognition result 522 corresponding to the image area 532 in the first foreground recognition result 520 can be replaced with the second foreground recognition result 540 to obtain a target foreground recognition result 550. If there are multiple selected image areas, the foreground recognition result corresponding to each image area 5 in the first foreground recognition result can be replaced with the corresponding second foreground recognition result in sequence.

It should be noted that other methods may also be used to fuse the second foreground recognition results corresponding to each image area with the first foreground recognition results. For example, the foreground recognition results corresponding to each image area in the first foreground recognition result may be fused with the second foreground recognition results corresponding to each image area by weighted average, etc., which are not limited here.

Step 414: correct the depth estimation result according to the target foreground recognition result to obtain a target depth map, and blur the first image based on the target depth map to obtain a second blurred image.

The depth estimation result can be corrected according to the more accurate target foreground recognition result to obtain a more accurate target depth map of the foreground area, and the electronic device can re-blur the first image according to the target depth map to obtain a second blurred image with a better blur effect. It can be understood that the method of correcting the depth estimation result according to the target foreground recognition result can be similar to the method of correcting the depth estimation result according to the first foreground recognition result described in the above embodiment, and the method of blurring the first image according to the target depth map can be similar to the method of blurring the first image according to the first depth map described in the above embodiment, and will not be repeated here.

In an embodiment of the present application, after the first image is initially blurred, the user can select image areas that need to be further optimized, and the electronic device can perform foreground recognition on each selected image area, and fuse the second foreground recognition results corresponding to each image area with the first foreground recognition result to correct the first foreground recognition result and obtain a more accurate target foreground recognition result, so that the first image can be blurred based on the target foreground recognition result to obtain a second blurred image with a better blurring effect, thereby improving the accuracy of foreground recognition and the image blurring effect.

In some embodiments, in addition to using the second foreground recognition results of each selected image area to blur the entire first image again in the above embodiments, the second foreground recognition results of each selected image area can also be used directly to perform local blurring on the first blurred image to reduce the amount of calculation and improve image processing efficiency. After obtaining the second foreground recognition results corresponding to each image area, the electronic device can correct the first depth map based on the second foreground recognition results of each image area, and can adjust the edge information corresponding to each image area in the first depth map based on the second foreground recognition results of each image area to obtain the second depth map.

As a specific implementation, a depth map area having the same image position as each selected image area in the first depth map can be determined based on the image position of each selected image area in the first blurred image, and edge information of the foreground area contained in the corresponding depth map area can be corrected based on the second foreground recognition result of each selected image area, thereby adjusting the edge information of the foreground area in the first depth map.

Taking the first image area among the selected image areas as an example, the first image area can be any selected image area. The edge information of the foreground area in the second foreground recognition result of the first image area can be obtained, and the edge information of the foreground area in the second foreground recognition result of the first image area can be compared with the edge information of the foreground area contained in the first depth map area corresponding to the first image area in the first depth map to determine whether the two are consistent. If they are inconsistent, the edge information of the foreground area contained in the first depth map area can be modified to the edge information of the foreground area in the second foreground recognition result of the first image area. Optionally, if the two are inconsistent, the edge information of the foreground area contained in the first depth map area can also be fused with the edge information of the foreground area in the second foreground recognition result of the first image area. The fusion method can include but is not limited to the mean fusion of pixels, fusion according to different weight coefficients, etc., which are not limited here.

Since the second foreground recognition results of each image area are more accurate, the second foreground recognition results of each image area are used to correct the edge information of the foreground area of the corresponding depth map area in the first depth map, so as to obtain a second depth map that more accurately divides the foreground area and the background area.

As an implementation, the electronic device may use the second depth map to blur the first image to obtain a more accurate second blurred image. As another implementation, the electronic device may also blur each image area in the first blurred image according to the depth information of each image area in the second depth map to obtain a second blurred image.

Furthermore, the depth map area corresponding to each image area in the second depth map accurately divides the foreground area and the background area. The blur parameters corresponding to each image area can be re-determined according to the depth information of the background area contained in the depth map area corresponding to each image area in the second depth map, and the background area contained in each image area can be blurred according to the blur parameters corresponding to each image area to obtain a second blurred image.

FIG6 is a schematic diagram of correcting a first depth map using a second foreground recognition result of a selected image area in one embodiment. As shown in FIG6 , the electronic device may perform depth estimation on the first image 610 to obtain a depth estimation result, and correct the depth estimation result using the first foreground recognition result of the first image to obtain a first depth map 620. The first image 610 may be defocused according to the first depth map 620 to obtain a first defocused image 630. The user may select an image area to be optimized in the first defocused image 630, and the electronic device may determine an image area 632 according to the user's selection operation, and perform foreground recognition on the image area 632 to obtain a second foreground recognition result 640 corresponding to the image area 632. According to the second foreground recognition result 640 corresponding to the image area 632, the edge information of the foreground area in the depth map area 622 having the same image position as the image area 632 in the first depth map 620 may be adjusted to obtain a second depth map 650.

Optionally, after obtaining the second depth map 650, the electronic device may blur the image area 632 in the first blurred image 630 according to the depth information of the depth map area having the same image position as the image area 632 in the second depth map 650 to obtain a second blurred image.

In an embodiment of the present application, the first depth map can be corrected according to the second foreground recognition result of each image area selected by the user, and the image area in the first blurred image can be blurred based on the corrected second depth map to obtain a second blurred image with a better blurring effect. The second foreground recognition result of each selected image area is directly used to perform local blurring in the first blurred image, thereby improving the accuracy of foreground recognition and the image blurring effect, reducing the amount of calculation, and improving image processing efficiency.

As shown in FIG. 7 , in another embodiment, an image processing method is provided, which can be applied to the above electronic device. The method may include the following steps:

Step 702, identifying the portrait area and the hair area of the first image, and obtaining a portrait segmentation result that meets the accuracy requirement.

In an embodiment of the present application, the first image may include a portrait image, which refers to an image containing a portrait. The portrait area in the portrait image is the foreground area, and the area other than the portrait area is the background area. Because the hair area of the person is particularly prone to misidentification when the foreground portrait area of the portrait image is identified, such as the hair part of the hair area is mistakenly identified as the background area, or the background area is mistakenly identified as the hair area, etc., resulting in an inaccurate foreground portrait area. Therefore, in an embodiment of the present application, the portrait area and the hair area of the first image can be identified separately to obtain a portrait segmentation result that meets the accuracy conditions, and the portrait segmentation result can be used to mark the position of the portrait area in the first image. The portrait segmentation result that meets the accuracy conditions can accurately locate the hair area in the first image, and the portrait segmentation result is more accurate. Optionally, the accuracy condition may be set using one or more accuracy indicators. For example, the accuracy indicator may include the sum of absolute differences (SAD), mean squared error (MSE), gradient error, etc. between the obtained portrait segmentation result and the true portrait segmentation result; the accuracy condition may include one or more of the error between the obtained portrait segmentation result and the true portrait segmentation result being less than a SAD threshold, less than an MSE threshold, less than a gradient error threshold, etc.

In some embodiments, the electronic device may first identify the portrait area in the first image, obtain a portrait segmentation map corresponding to the first image, and then identify the hair area in the first image based on the portrait segmentation map to obtain a hair cutout result of the first image. The portrait segmentation map may be corrected according to the hair cutout result to obtain a portrait segmentation result that meets the accuracy requirements.

Specifically, the way in which the electronic device identifies the portrait area in the first image may include but is not limited to utilizing a portrait segmentation method based on graph theory, a portrait segmentation method based on clustering, a portrait segmentation method based on semantics, an instance-based portrait segmentation method, a portrait segmentation method based on a deeplab series network model, a segmentation method based on a U-net network (U-Net), or a portrait segmentation method based on a fully convolutional network (FCN).

Taking the example of an electronic device identifying the portrait area of the first image through a portrait segmentation model to obtain a portrait segmentation map, the portrait segmentation model can be a model with a U-Net structure, and the portrait segmentation model can include an encoder and a decoder, the encoder can include multiple downsampling layers, and the decoder can include multiple upsampling layers. The portrait segmentation model can first perform multiple downsampling convolution processing on the first image through the multiple downsampling layers of the encoder, and then perform multiple upsampling processing through the multiple upsampling layers of the decoder to obtain a portrait segmentation map. In the portrait segmentation model, jump connections can be implemented between downsampling layers and upsampling layers with the same resolution, and the features of the downsampling layers and upsampling layers with the same resolution are fused to make the upsampling process more accurate.

Optionally, the portrait segmentation model can be trained based on a portrait sample set, which may include multiple portrait sample images carrying portrait labels. The portrait labels can be used to mark the portrait areas in the portrait sample images. For example, the portrait labels may include a portrait mask. The pixel points belonging to the portrait area in the portrait mask may correspond to a first pixel value, and the pixel points belonging to the background area may correspond to a second pixel value. The portrait area of the sample image can be accurately marked by the binarized portrait mask.

In some embodiments, before the first image is input into the portrait segmentation model, the first image may be scaled and/or rotated according to the input size of the portrait segmentation model to obtain a first image that meets the input size, and then the first image is input into the portrait segmentation model for portrait recognition. For example, if the input size of the portrait segmentation model is a vertical size (the side of the image parallel to the horizontal line is shorter than the side perpendicular to the horizontal line), if the first image is a horizontal size (the side of the image parallel to the horizontal line is longer than the side perpendicular to the horizontal line), the first image may be rotated 90 degrees clockwise or counterclockwise first; or if the input size of the portrait segmentation model is smaller than the image size of the first image, the first image may be reduced first to obtain a first image consistent with the input size, etc. It can ensure that the input first image is compatible with the portrait segmentation model and the accuracy of the output portrait segmentation map is guaranteed.

After obtaining the portrait segmentation map, the electronic device may perform channel stitching on the portrait segmentation map and the first image to obtain a stitched image, and identify the hair area in the first image based on the stitched image to obtain a portrait segmentation result that meets the accuracy requirements.

In some embodiments, the portrait segmentation map obtained by the electronic device may be a single-channel image. Further, the portrait segmentation map may be a single-channel three-value image. In the portrait segmentation map, the pixel points identified as belonging to the portrait area may correspond to the first pixel value, the pixel points identified as belonging to the background area may correspond to the second pixel value, and the pixel points identified as belonging to the junction area of the portrait and the background may correspond to the third pixel value. For example, taking the portrait segmentation map as a grayscale image as an example, the grayscale value corresponding to the pixel points identified as belonging to the portrait area may be 0, the grayscale value corresponding to the pixel points identified as belonging to the background area may be 255, and the grayscale value corresponding to the pixel points identified as belonging to the junction area of the portrait and the background may be 127.5, etc., but it is not limited thereto.

The single-channel portrait segmentation map and the first image can be channel-joined. The first image is a three-channel image (such as an RGB image or an HSV image, etc.). The portrait segmentation map can be joined to the fourth channel of the first image to obtain a four-channel joined image. Optionally, before the portrait segmentation map and the first image are channel-joined, the first image and the portrait segmentation map can be normalized respectively, and then the normalized first image and the portrait segmentation map are channel-joined. The normalization method may include subtracting the mean from the pixel value of each pixel in the image and dividing it by the variance to obtain the normalized pixel value, or directly removing the gray value area (such as 255) from the pixel value of each pixel to obtain the normalized pixel value, etc., but is not limited thereto. Normalizing the first image and the portrait segmentation map first and then performing channel-joining can improve the accuracy and efficiency of subsequent hair area recognition.

The electronic device can perform hair matting on the spliced image, identify the hair area, obtain the hair matting result of the first image, and obtain the portrait segmentation result that meets the accuracy condition based on the hair matting result and the portrait segmentation map. The hair matting method may include but is not limited to Poisson Matting method, Bayes Matting based on Bayesian theory, data-driven machine learning matting method or closed surface matting method and other traditional matting methods that do not use deep learning, or deep learning-based matting methods using artificial neural networks such as Convolutional Neutral Network (CNN).

As a specific implementation, the electronic device may input the spliced image into a first hair cutout model, extract features of the spliced image through the first hair cutout model, and determine the hair region in the first image according to the features, so as to obtain a portrait segmentation result that meets the accuracy condition. The first hair cutout model may be obtained by training based on a first training set, and the first training set includes a plurality of portrait sample images with hair regions annotated, and the portrait sample images may carry hair labels, and the hair labels may be used to annotate the hair regions in the portrait sample images. Optionally, in order to ensure that the portrait segmentation result that meets the accuracy condition is obtained, the first hair cutout model may be trained according to the accuracy condition, so that the predicted hair region output by the first hair cutout model meets the accuracy condition, for example, the error between the predicted hair region output by the first hair cutout model and the real hair region of the portrait sample image may be less than a set SAD threshold, less than an MSE threshold, less than a gradient error threshold, etc., but is not limited thereto.

The first hair cutout model may also be a model using a network architecture such as U-Net. The first hair cutout model may include an encoder and a decoder. The first hair cutout model outputs a hair cutout result of the first image based on the input spliced image. The hair cutout result may include a hair mask corresponding to the first image. The hair mask may include position information of the hair area in the first image, which can be used to mark the hair area in the first image.

The portrait segmentation map can be corrected according to the hair cutout result output by the first hair cutout model to obtain a portrait segmentation result that meets the accuracy condition. Further, the pixel values of the pixel points in the portrait segmentation map that belong to the intersection area of the portrait and the background can be adjusted according to the hair cutout result to determine whether each pixel point in the intersection area of the portrait and the background belongs to the hair area or the background area. Each pixel point in the portrait segmentation map that is identified as belonging to the intersection area of the portrait and the background can be searched to determine whether it corresponds to the hair area or the background area in the hair cutout result, and each pixel point in the portrait segmentation map that is identified as belonging to the intersection area of the portrait and the background can be accurately divided into regions using the hair cutout result.

As another implementation, the hair cutout result may be a single-channel hair mask, and the region of the portrait segmentation map that matches the hair cutout result may be directly replaced with the hair cutout result to obtain a portrait segmentation result that meets the accuracy requirement. Using the portrait segmentation model and the first hair cutout model to identify the portrait region and the hair region of the first image, respectively, can improve the stability and accuracy of the obtained portrait segmentation result that meets the accuracy requirement.

In other embodiments, the electronic device may also directly input the first image into an image processing model, and identify the portrait area and the hair area of the first image through the image processing model to obtain a portrait segmentation result that meets the accuracy requirements. Optionally, the image processing model may be a neural network with a dual encoding and decoding structure, and the image processing model may be obtained by training portrait sample images that carry both portrait labels and hair labels. This approach can reduce the amount of calculation required to obtain a portrait segmentation result that meets the accuracy requirements, thereby improving image processing efficiency.

Step 704: blur the first image based on the portrait segmentation result that meets the accuracy requirement to obtain a first blurred image.

In some embodiments, the electronic device may also perform depth estimation on the first image to obtain a depth estimation result of the first image, and correct the depth estimation result according to a portrait segmentation result that meets the accuracy conditions to obtain a first depth map of the first image, and then use the first depth map to blur the first image to obtain a first blurred image.

Step 706 , in response to a selection operation on the first blurred image, determining one or more image regions selected by the selection operation in the first blurred image.

The description of step 706 may refer to the relevant description in the above embodiments, which will not be repeated here.

Step 708, identifying the hair area of each image area, and obtaining the local hair cutout result corresponding to each image area.

For portrait images, the area where the blurring effect is poor is usually the boundary area between the hair and the background. In order to improve the blurring effect of the image, after determining one or more image areas selected by the user, the embodiment of the present application can identify the hair area in each image area again to obtain the local hair cutout result corresponding to each image area, so as to refine the local hair identification in the first image.

Methods for identifying hair areas in each image area may include but are not limited to traditional cutout methods that do not use deep learning, such as Poisson cutout method, Bayesian cutout based on Bayesian theory, data-driven machine learning cutout method, or closed surface cutout method, or deep learning-based cutout methods using artificial neural networks such as convolutional neural networks.

As a specific implementation, the electronic device can identify the hair area in each image area through the second hair cutout model to obtain the local hair cutout result corresponding to each image area. The network architecture of the second hair cutout model and the first hair cutout model can be the same or similar, wherein the second hair cutout model can be trained based on a second training set, and the second training set includes a plurality of sample images randomly cropped from the portrait sample images of the first training set.

Optionally, the sample images in the second training set can be randomly cropped from the portrait sample images of the first training set according to the area size. Since the portrait sample images of the first training set carry hair labels, directly cropping the image area of the same size as the selection box from the portrait sample images of the first training set as the sample image for the second hair cutout model can improve the local hair recognition capability of the second hair cutout model and improve the training efficiency.

In some embodiments, before the electronic device recognizes the hair area in each image area through the second hair cutout model, the first area image corresponding to each image area may be cropped from the first image, and the second area image corresponding to each image area may be cropped from the portrait segmentation map. The first area image corresponding to each image area may refer to an image formed by image content having the same image position as each image area in the first image, and the second area image corresponding to each image area refers to an image formed by image content having the same image position as each image area in the portrait segmentation map.

The first region image and the second region image corresponding to each image region can be channel-joined to obtain an input image corresponding to each image region, and the input image corresponding to each image region can be input into the second hair cutout model. The second hair cutout model can be used to identify the hair region contained in the input image corresponding to each input image region, and obtain a local hair cutout result corresponding to each image region.

Exemplarily, the local hair cutout results corresponding to each image region in the above embodiment are now described in conjunction with FIG8. As shown in FIG8, the screen of the electronic device can display a first blurred image 810, and the user can select an image region 812 to be optimized. The electronic device can crop a first region image 822 having the same image position as the image region 812 from the first image 820, and crop a second region image 832 having the same image position as the image region 812 from the portrait segmentation image 830. The first region image 822 and the second region image 832 can be channel-joined, and the first region image 822 can be joined as the fourth channel of the second region image 832 to obtain a four-channel input image 840. The input image 840 can be input into the second hair cutout model, and the hair region of the input image 840 can be identified by the second hair cutout model to obtain a local hair cutout result 850 corresponding to the image region 812.

In some embodiments, before inputting the input images of each image region into the second hair cutout model, the input images may be preprocessed so that the preprocessed input images are compatible with the second hair cutout model. The input images may be scaled and/or rotated according to the size requirement corresponding to the second hair cutout model to obtain input images that meet the size requirement, which is the size requirement of the second hair cutout model for the input images. For example, if the cropped first region image and the second region image are both the size of the selection box, which is smaller than the size requirement of the second hair cutout model, the input images may be enlarged first; or if the spliced input images are vertical images (the side of the image parallel to the horizontal line is shorter than the side perpendicular to the horizontal line), and the size requirement of the second hair cutout model is horizontal size (the side of the image parallel to the horizontal line is longer than the side perpendicular to the horizontal line), the input images may be rotated 90 degrees clockwise or counterclockwise first. The input image is first scaled and/or rotated so that the input image meets the size requirement, and then the input image that meets the size requirement is input into the second hair cutout model for hair area recognition, thereby ensuring the accuracy of the recognition result.

Furthermore, after scaling and/or rotating the input image, the input image may be normalized in a manner similar to the normalization of the first image and the portrait segmentation image described in the above embodiment, and the normalized input image is then input into the second hair cutout model for hair region recognition. Optionally, after cropping to obtain the first region image and the second region image, the first region image and the second region image may be normalized respectively, and then the normalized first region image and the second region image may be channel stitched.

After obtaining the local hair cutout result output by the second hair cutout model, the local hair cutout result may be scaled and/or rotated according to the original size of the input image to obtain a local hair cutout result having the same size as the image area.

Step 710: Based on the local hair cutout results of each image region, blur the first image or the first blurred image to obtain a second blurred image.

As an implementation method, the electronic device may fuse the local hair cutout results of each image area with the portrait segmentation results that meet the accuracy conditions to obtain a target portrait segmentation result. The portrait segmentation results that meet the accuracy conditions may be corrected based on the local hair cutout results of each image area to obtain a more accurate target portrait segmentation result. Specifically, the image content that has the same position as each image area in the portrait segmentation results that meet the accuracy conditions may be replaced with the local hair cutout results corresponding to each image area. After obtaining the target portrait segmentation result, the depth estimation result of the first image may be corrected based on the target foreground recognition result to obtain a target depth map, and the first image may be blurred based on the target depth map to obtain a second blurred image.

As another implementation, the electronic device may correct the first depth map using the local hair cutout results of each image area, and may adjust the hair edges corresponding to each image area in the first depth map based on the local hair cutout results of each image area to obtain a second depth map. Then, using the depth information of each image area in the second depth map, blur each image area in the first blurred image to obtain a second blurred image. Optionally, the second depth map may also be used to blur the first image to obtain a more accurate second blurred image.

It should be noted that in addition to being applied to blur optimization scenarios, the solution of the present application can also be applied to other image processing scenarios. For example, after performing local hair cutouts on each selected image area to obtain a more accurate portrait area, the boundary area between the hair and the background can be color adjusted or the light spot blurred based on the more accurate portrait area. This is not limited here.

In the embodiment of the present application, an interactive hair cutout rendering optimization method is adopted. The user can select the image area that needs to be optimized according to actual needs, and perform local hair cutout on the selected image area, thereby improving the accuracy of hair cutout and the accuracy of portrait recognition, thereby improving the hair significance effect of the blurred processing result, avoiding the situation where the background of the hair area is blurred or the hair area is blurred by mistake, and improving the interactivity with the user.

As shown in FIG. 9 , in one embodiment, an image processing device 900 is provided, which can be applied to the above-mentioned electronic device. The image processing device 900 may include a first recognition module 910 , a blurring module 920 , a region selection module 930 and a second recognition module 940 .

The first recognition module 910 is used to recognize a foreground area in a first image and obtain a first foreground recognition result.

The blur module 920 is used to blur the first image based on the first foreground recognition result to obtain a first blurred image.

The area selection module 930 is used to respond to a selection operation on the first blurred image and determine one or more image areas selected by the selection operation in the first blurred image.

In one embodiment, the area selection module 930 is also used to obtain one or more touch positions of the selection operation on the screen; for each touch position, form a selection box corresponding to each touch position according to the area size; and determine the image area corresponding to each selection box in the first blurred image.

In one embodiment, the region selection module 930 is further configured to adjust the size of the target selection box according to the adjustment operation if an adjustment operation triggered on the target selection box is detected.

The second recognition module 940 is used to recognize the foreground area of each image area and obtain a second foreground recognition result corresponding to each image area.

The blur module 920 is further configured to blur the first blurred image based on the second foreground recognition result of each image region to obtain a second blurred image.

In the embodiment of the present application, after the first blurring process is performed on the first image, the user can select the image area that needs to be further optimized, and perform foreground recognition on the selected image area again, thereby improving the accuracy of foreground recognition, and performing secondary blurring on the first image or the first blurred image based on the more accurate second foreground recognition result, thereby improving the situation where part of the foreground area is mistakenly blurred or part of the background area is omitted and not blurred, thereby improving the blurring effect of the image. In addition, the user can select the image area that needs to be further optimized in the first blurred image, thereby meeting the different needs of the user and improving the interactivity with the user.

In one embodiment, the blur module 920 includes a fusion unit and a blur unit.

The fusion unit is used to fuse the second foreground recognition results corresponding to each image area with the first foreground recognition results to obtain a target foreground recognition result.

In one embodiment, the fusion unit is further used to replace the foreground recognition results corresponding to each image region in the first foreground recognition results with the second foreground recognition results corresponding to each image region, respectively, to obtain a target foreground recognition result.

The blurring unit is used to blur the first image based on the target foreground recognition result to obtain a second blurred image.

In one embodiment, the image processing device 900 , in addition to the first recognition module 910 , the blurring module 920 , the region selection module 930 , and the second recognition module 940 , further includes a depth estimation module.

The depth estimation module is used to perform depth estimation on the first image to obtain a depth estimation result, wherein the depth estimation result includes depth information of each pixel in the first image.

The blurring unit is further used to correct the depth estimation result according to the target foreground recognition result to obtain a target depth map, and blur the first image based on the target depth map to obtain a second blurred image.

In an embodiment of the present application, after the first image is initially blurred, the user can select image areas that need to be further optimized, and the electronic device can perform foreground recognition on each selected image area, and fuse the second foreground recognition results corresponding to each image area with the first foreground recognition result to correct the first foreground recognition result and obtain a more accurate target foreground recognition result, so that the first image can be blurred based on the target foreground recognition result to obtain a second blurred image with a better blurring effect, thereby improving the accuracy of foreground recognition and the image blurring effect.

In one embodiment, the blur module 920 is further configured to correct the depth estimation result according to the first foreground recognition result to obtain a first depth map of the first image, and blur the first image according to the first depth map to obtain a first blurred image.

In one embodiment, the blur module 920 is also used to adjust the edge information corresponding to each image area in the first depth map based on the second foreground recognition result of each image area to obtain a second depth map; and blur each image area in the first blurred image according to the depth information of each image area in the second depth map to obtain a second blurred image.

In an embodiment of the present application, the first depth map can be corrected according to the second foreground recognition result of each image area selected by the user, and the image area in the first blurred image can be blurred based on the corrected second depth map to obtain a second blurred image with a better blurring effect. The second foreground recognition result of each selected image area is directly used to perform local blurring in the first blurred image, thereby improving the accuracy of foreground recognition and the image blurring effect, reducing the amount of calculation, and improving image processing efficiency.

In one embodiment, the first image includes a portrait image. The first recognition module 910 is further configured to recognize a portrait region and a hair region of the first image to obtain a portrait segmentation result that meets the accuracy requirement.

In one embodiment, the first recognition module 910 includes a portrait segmentation unit, a splicing unit and a hair cutout unit.

The portrait segmentation unit is used to identify the portrait area of the first image and obtain a portrait segmentation map.

The stitching unit is used to perform channel stitching on the portrait segmentation image and the first image to obtain a stitched image.

The hair cutout unit is used to identify the hair area in the first image according to the spliced image to obtain a portrait segmentation result that meets the accuracy condition.

In one embodiment, the hair cutout unit is further used to input the stitched image into a first hair cutout model, extract features of the stitched image through the first hair cutout model, and determine the hair area in the first image according to the features to obtain a portrait segmentation result that meets the accuracy conditions, wherein the first hair cutout model is trained based on a first training set, and the first training set includes multiple portrait sample images with hair areas marked.

In one embodiment, the second recognition module 940 is further used to identify the hair area in each image area through a second hair cutout model to obtain a local hair cutout result corresponding to each image area, wherein the second hair cutout model is trained based on a second training set, and the second training set includes multiple sample images randomly cropped from portrait sample images of the first training set.

In one embodiment, the image processing device 900 further includes a cropping module and a splicing module.

The cropping module is used to crop first region images corresponding to each image region from the first image; and to crop second region images corresponding to each image region from the portrait segmentation image.

The stitching module is used to perform channel stitching on the first region image and the second region image corresponding to each image region to obtain the input image corresponding to each image region.

The second recognition module 940 is further used to input the input image corresponding to each image area into the second hair cutout model, and identify the hair area in the input image corresponding to each image area through the second hair cutout model to obtain the local hair cutout result corresponding to each image area.

In one embodiment, the above-mentioned image processing device 900 further includes a pre-processing module.

The preprocessing module is used to scale and/or rotate the input image according to the size requirement corresponding to the second hair cutout model after the stitching module obtains the input image corresponding to each image area, so as to obtain an input image that meets the size requirement.

In the embodiment of the present application, an interactive hair cutout rendering optimization method is adopted. The user can select the image area that needs to be optimized according to actual needs, and perform local hair cutout on the selected image area, thereby improving the accuracy of hair cutout and the accuracy of portrait recognition, thereby improving the hair significance effect of the blurred processing result, avoiding the situation where the background of the hair area is blurred or the hair area is blurred by mistake, and improving the interactivity with the user.

Fig. 10 is a block diagram of the structure of an electronic device in an embodiment. As shown in Fig. 10, the electronic device 1000 may include one or more of the following components: a processor 1010, a memory 1020 coupled to the processor 1010, wherein the memory 1020 may store one or more computer programs, and the one or more computer programs may be configured to implement the methods described in the above embodiments when executed by one or more processors 1010.

The processor 1010 may include one or more processing cores. The processor 1010 uses various interfaces and lines to connect various parts of the entire electronic device 1000, and executes various functions and processes data of the electronic device 1000 by running or executing instructions, programs, code sets or instruction sets stored in the memory 1020, and calling data stored in the memory 1020. Optionally, the processor 1010 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 1010 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing display content; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 1010, but may be implemented separately through a communication chip.

The memory 1020 may include a random access memory (RAM) or a read-only memory (ROM). The memory 1020 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 1020 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc. The data storage area may also store data created by the electronic device 1000 during use, etc.

It can be understood that the electronic device 1000 may include more or fewer structural elements than those in the above structural block diagram, for example, including a power module, physical buttons, a WiFi (Wireless Fidelity) module, a speaker, a Bluetooth module, a sensor, etc., and is not limited here.

An embodiment of the present application discloses a computer-readable storage medium storing a computer program, wherein the computer program implements the method described in the above embodiment when executed by a processor.

An embodiment of the present application discloses a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program can be executed by a processor to implement the methods described in the above embodiments.

Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a ROM, etc.

As used herein, any reference to memory, storage, database, or other medium may include non-volatile and/or volatile memory. Suitable non-volatile memory may include ROM, programmable ROM (Programmable ROM, PROM), erasable PROM (Erasable PROM, EPROM), electrically erasable PROM (Electrically Erasable PROM, EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which is used as an external cache memory. By way of illustration and not limitation, RAM may be in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (Double Data Rate SDRAM, DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), memory bus direct RAM (RDRAM) and direct memory bus dynamic RAM (DRDRAM).

It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. Those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present application.

In the various embodiments of the present application, it should be understood that the size of the serial numbers of the above-mentioned processes does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

The above is a detailed introduction to an image processing method, device, electronic device, and computer-readable storage medium disclosed in the embodiments of the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core idea of the present application. At the same time, for those of ordinary skill in the art, according to the idea of the present application, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as limiting the present application.

Claims (17)

1. An image processing method, comprising:

identifying a foreground region in the first image to obtain a first foreground identification result;

performing blurring processing on the first image based on the first foreground identification result to obtain a first blurring image;

Determining one or more image areas selected in the first virtual image by the selection operation in response to the selection operation for the first virtual image; the image area is an area where blurring errors or blurring leakage occurs;

Identifying foreground areas of the image areas to obtain second foreground identification results corresponding to the image areas;

based on the second foreground identification result of each image area, carrying out blurring processing on the first image or the first blurring image to obtain a second blurring image;

The step of identifying the foreground region of each image region to obtain a second foreground identification result corresponding to each image region comprises the following steps:

And cutting out an area image with the same image position from the first image according to the image position of each image area in the first blurring image, and carrying out foreground recognition on the area image cut out from the first image so as to obtain a second foreground recognition result corresponding to each image area.

2. The method according to claim 1, wherein blurring the first image or the first blurring image based on the second foreground identification result of each image area, to obtain a second blurring image, includes:

Fusing the second foreground recognition results corresponding to the image areas with the first foreground recognition results to obtain target foreground recognition results;

and blurring the first image based on the target foreground identification result to obtain a second blurring image.

3. The method according to claim 2, wherein the fusing the second foreground recognition result corresponding to each image area with the first foreground recognition result to obtain a target foreground recognition result includes:

And respectively replacing the foreground recognition results corresponding to the image areas in the first foreground recognition results with the second foreground recognition results corresponding to the image areas to obtain target foreground recognition results.

4. The method of claim 2, wherein prior to blurring the first image based on the first foreground recognition result, the method further comprises:

Performing depth estimation on the first image to obtain a depth estimation result, wherein the depth estimation result comprises depth information of each pixel point in the first image;

the blurring processing is performed on the first image based on the target foreground recognition result to obtain a second blurring image, including:

correcting the depth estimation result according to the target foreground identification result to obtain a target depth map;

and blurring the first image based on the target depth map to obtain a second blurring image.

5. The method of claim 1, wherein prior to blurring the first image based on the first foreground recognition result, the method further comprises:

Performing depth estimation on the first image to obtain a depth estimation result, wherein the depth estimation result comprises depth information of each pixel point in the first image;

The blurring processing is performed on the first image based on the first foreground recognition result to obtain a first blurring image, including:

Correcting the depth estimation result according to the first foreground identification result to obtain a first depth map of the first image;

and carrying out blurring processing on the first image according to the first depth map to obtain a first blurring image.

6. The method of claim 5, wherein blurring the first image or the first blurred image based on the second foreground identification result of each of the image areas to obtain a second blurred image, comprising:

Based on a second foreground identification result of each image area, adjusting edge information corresponding to each image area in the first depth map to obtain a second depth map;

And according to the depth information of each image region in the second depth map, blurring processing is carried out on each image region in the first blurring image, so as to obtain a second blurring image.

7. The method of any one of claims 1-6, wherein the first image comprises a portrait image; the step of identifying the foreground region in the first image to obtain a first foreground identification result comprises the following steps:

and identifying the portrait area and the hair area of the first image to obtain a portrait segmentation result meeting the precision condition.

8. The method of claim 7, wherein the identifying the portrait area and the hair area of the first image to obtain the portrait segmentation result satisfying the accuracy condition includes:

Identifying a human image area of the first image to obtain a human image segmentation map;

channel stitching is carried out on the portrait segmentation drawing and the first image, so that a stitched image is obtained;

and identifying the hair region in the first image according to the spliced image so as to obtain a portrait segmentation result meeting the precision condition.

9. The method of claim 8, wherein the identifying the hair region in the first image from the stitched image to obtain the portrait segmentation result satisfying the accuracy condition comprises:

Inputting the spliced image into a first hairline matting model, extracting the characteristics of the spliced image through the first hairline matting model, and determining the hair region in the first image according to the characteristics so as to obtain a portrait segmentation result meeting the precision condition, wherein the first hairline matting model is obtained based on a first training set, and the first training set comprises a plurality of portrait sample images marked with the hair region.

10. The method according to claim 9, wherein the identifying the foreground region of each image region to obtain the second foreground identification result corresponding to each image region includes:

and identifying hair areas in the image areas through a second hair matting model to obtain local hair matting results corresponding to the image areas, wherein the second hair matting model is obtained based on training of a second training set, and the second training set comprises a plurality of sample images obtained by randomly cutting human image sample images of the first training set.

11. The method of claim 10 further including, prior to said identifying hair regions in each of said image regions by a second hair cut model:

cropping a first region image corresponding to each image region from the first image;

Clipping a second region image corresponding to each image region from the portrait segmentation drawing;

Channel stitching is carried out on the first area image and the second area image corresponding to each image area, and input images corresponding to each image area are obtained;

and inputting the input images corresponding to the image areas into a second hairline matting model.

12. The method of claim 11, wherein after said obtaining the input image corresponding to each of the image areas, the method further comprises:

And performing scaling and/or rotation processing on the input image according to the size requirement corresponding to the second hairline matting model to obtain the input image meeting the size requirement.

13. The method of any of claims 1-6, 8-12, wherein the determining the one or more image regions selected by the selecting operation in the first blurred image comprises:

Acquiring one or more touch positions of the selection operation on a screen;

forming a selection frame corresponding to each touch position according to the area size aiming at each touch position;

and determining an image area corresponding to each selection frame in the first blurring image.

14. The method of claim 13, wherein after the forming of the selection boxes corresponding to the respective touch positions according to the area size, the method further comprises:

and if the adjustment operation triggered by the target selection frame is detected, adjusting the size of the target selection frame according to the adjustment operation.

15. An image processing apparatus, comprising:

the first identification module is used for identifying a foreground region in the first image to obtain a first foreground identification result;

The blurring module is used for blurring the first image based on the first foreground identification result to obtain a first blurring image;

A region selection module for determining one or more image regions selected by the selection operation in the first virtual image in response to the selection operation for the first virtual image; the image area is an area where blurring errors or blurring leakage occurs;

The second recognition module is used for recognizing the foreground region of each image region to obtain a second foreground recognition result corresponding to each image region;

The blurring module is further configured to perform blurring processing on the first blurring image based on a second foreground recognition result of each image area, so as to obtain a second blurring image;

The second recognition module is further configured to cut out an area image with the same image position from the first image according to the image position of each image area in the first blurring image, and perform foreground recognition on the area image cut out from the first image, so as to obtain a second foreground recognition result corresponding to each image area.

16. An electronic device comprising a memory and a processor, the memory having stored therein a computer program which, when executed by the processor, causes the processor to implement the method of any of claims 1 to 14.

17. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the method according to any of claims 1 to 14.