WO2023015981A1 - Image processing method and related device therefor - Google Patents

Abstract

An image processing method and a related device therefor, which relate to the field of image processing. The method comprises: a first camera collecting a first image and a second camera collecting a second image (S10); obtaining a mask block according to the first image; fusing the first image and the second image, so as to obtain a first fused image (S30); according to the mask block, determining a first image block in the first image and determining a second image block in the second image (S40); fusing the first image block and the second image block, so as to obtain a fused image block (S50); and fusing the first fused image and the fused image block, so as to obtain a third image (S60). By means of fusing the content of the clearer second image and the same area in the low-definition first image, missing details are recovered, and fusion is performed a plurality of times, so as to obtain a high-definition image.

Description

Image processing method and related equipment

This application claims the priority of a Chinese patent application filed with the State Intellectual Property Office on August 12, 2021, with application number 202110923642.9, and application title "Image processing method and related equipment", the entire contents of which are incorporated in this application by reference middle.

technical field

The present application relates to the field of image processing, in particular to an image processing method and related equipment.

Background technique

With the widespread use of electronic devices, taking pictures with electronic devices has become a daily behavior in people's lives. Taking electronic devices as mobile phones as an example, various multi-frame image synthesis algorithms have emerged to improve image quality, such as: multi-frame noise reduction, multi-frame super-resolution, etc.

However, in some high dynamic range (high dynamic range, HDR) scenes or in backlit scenes, due to the specular reflection on some areas of the object surface in the scene to be shot, the details of these high-brightness areas are lost, and the related technology is not effective. handle the situation. Therefore, how to restore the details in the high-brightness area of the image has become an urgent problem to be solved.

Contents of the invention

The present application provides an image processing method and related equipment, which can perform image restoration processing on low-resolution areas in an image to restore details, thereby improving user experience.

In order to achieve the above object, the application adopts the following technical solutions:

In a first aspect, an image processing method is provided, which is applied to an electronic device including a first camera and a second camera, and the method includes:

The electronic device starts the camera; displays a preview interface, and the preview interface includes a first control; detects a first operation on the first control; in response to the first operation, the first camera captures a first image and the second camera captures a second image, the second The definition of an image is lower than that of the second image, the first image includes a first region, and the first region is a region in the first image whose resolution is less than a preset threshold; a mask block is obtained according to the first image, and the mask The block corresponds to the first area; the first image and the second image are fused to obtain the first fused image; according to the mask block, the first image block in the first image and the second image block in the second image are determined , the first image block corresponds to the mask block, and the second image block corresponds to the mask block; the first image block and the second image block are fused to obtain a fused image block; the first fused image is fused with the fused image block , to get the third image.

Exemplarily, the first control may be the camera key 11 .

The embodiment of the present application provides an image processing method, by determining the mask block corresponding to the first region missing details from the first low-resolution image, and then obtaining the mask block corresponding to the first image from the first image An image block, and obtaining a second image block corresponding to the mask block from a second image with high definition and rich details, and fusing the first image block and the second image block to obtain a clear fused image block; Then, the first fused image obtained by merging the first image and the second image is further fused with the fused image block to restore missing details and obtain a high-definition third image.

In a possible implementation manner of the first aspect, obtaining the mask block according to the first image includes: inputting the first image into a segmentation model for segmentation, and generating a mask block; A region is segmented, and a mask block corresponding to the first region is generated.

Exemplarily, the segmentation model may be: a fully convolutional neural network.

In this implementation, the first image can be finely segmented through the segmentation model to obtain multiple segmented image regions, which facilitates the subsequent independent repair of regions with severe local missing details in the first image without affecting the second image. An image of the surrounding area.

In a possible implementation manner of the first aspect, merging the first image and the second image to obtain the first fused image includes: merging the first image and the second image using the first fusion model to obtain the first Blend images.

In this implementation, since the second image has a higher resolution than the first image, after the fusion of the first image and the second image, the resolution of the overall image can be improved, and a higher-definition first fusion can be obtained. image.

In a possible implementation manner of the first aspect, merging the first image block and the second image block to obtain the fused image block includes: merging the first image block and the second image block using a second fusion model, Get the fused image block.

In this implementation, since the resolution of the first image is lower than that of the second image, the resolution of the first image block is also lower than that of the second image block, and even there is no Any details. Therefore, by fusing the first image block that is unclear and lacks details with the second image block that is clear and rich in details, a higher-definition fused image block can be obtained.

In a possible implementation manner of the first aspect, merging the first fused image and the fused image block to obtain the third image includes: merging the first fused image and the fused image block using a third fusion model to obtain the third image Three images.

In this implementation, the first fused image has improved overall definition compared with the first image, and the fused image block has locally improved definition compared with the first image block in the first image, and the second A fused image is fused with the fused image block, and a part in the first fused image can be further repaired to obtain a third image with higher definition.

In a possible implementation manner of the first aspect, the method further includes: when the mask block is not obtained according to the first image, fusing the first image and the second image using the first fusion model to obtain the first fusion image .

In this implementation, no mask block is obtained, indicating that the first image does not have areas with severe partial loss of details, but the overall definition of the first image is still very low. Therefore, the first image and the second image can be combined Blending is performed to improve the clarity of the image.

In a possible implementation manner of the first aspect, the method further includes: registering the first image and the second image. In this implementation manner, registration can improve the accuracy of fusing the first image and the second image.

In a possible implementation manner of the first aspect, the method further includes: registering the first image block and the second image block. In this implementation manner, registration can improve the accuracy of fusing the first image block and the second image block.

In a possible implementation manner of the first aspect, registration includes: global registration and/or local registration, global registration is used to register all content in multiple images, and local registration is used to represent Register local content in multiple images. In this implementation, the alignment accuracy of all content in multiple images can be improved through global registration, and the alignment accuracy of local content in multiple images can be improved through local registration.

In a possible implementation of the first aspect, the method further includes: using the training image set and adding random highlight noise to train the first fusion model to obtain the second fusion model, wherein the training image set includes the original image , the original image is annotated with a mask block. In this implementation, due to the addition of random highlight noise during training, when the first image block and the second image block are fused using the trained second fusion model, the second image block with higher definition The proportion of the weight of the first image block is larger, so that the fused image block obtained by fusion can obtain more details from the second image block.

In a possible implementation manner of the first aspect, the third fusion model is a Laplace fusion model. In this implementation, when using the Laplacian fusion model for fusion, the Laplacian fusion model can first decompose the first fused image and the fused image block into different spatial frequency bands, and then in each spatial frequency band layer Fusion is performed separately, so that through the frequency division processing, the fusion of the first fusion image and the fusion image blocks can be made more natural, the connection is more delicate, and the obtained third image is of higher quality.

In a second aspect, an image processing apparatus is provided, and the apparatus includes a unit for performing each step in the above first aspect or any possible implementation manner of the first aspect.

In a third aspect, an electronic device is provided, including a camera module, a processor, and a memory; the camera module is used to collect a first image and a second image, and the definition of the first image is lower than that of the second image , the first image includes a first area, and the first area is an area whose resolution in the first image is less than a preset threshold; the memory is used to store a computer program that can run on the processor; the processor is used to execute the first A step of processing in the image processing method provided in any possible implementation manner of the aspect or the first aspect.

In a possible implementation of the third aspect, the camera module includes a wide-angle camera, a main camera, and a telephoto camera; the wide-angle camera is used to obtain the first image after the processor obtains a camera instruction; The second image is obtained after the processor obtains the photographing instruction, or; the main camera is used to obtain the first image after the processor obtains the photographing instruction; the telephoto camera is used to obtain the first image after the processor obtains the photographing instruction second image.

In a fourth aspect, a chip is provided, including: a processor, configured to call and run a computer program from a memory, so that a device installed with the chip executes the chip as provided in the first aspect or any possible implementation manner of the first aspect. The steps of processing in the image processing method.

In a fifth aspect, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program. The computer program includes program instructions. Steps of performing processing in the image processing method provided in any possible implementation manner of the aspect.

In a sixth aspect, a computer program product is provided, the computer program product includes a computer-readable storage medium storing a computer program, and the computer program enables the computer to execute the image provided in the first aspect or any possible implementation manner of the first aspect The step in the processing method that performs the processing.

For the beneficial effects of the second aspect to the sixth aspect, reference may be made to the beneficial effects of the first aspect above, which will not be repeated here.

Description of drawings

Fig. 1 is a schematic diagram of an image obtained by using related technologies;

FIG. 2 is a schematic diagram of an application scenario provided by an embodiment of the present application;

FIG. 3 is a schematic diagram of an image processing method provided in an embodiment of the present application;

Fig. 4 is a schematic flow chart of another image processing method provided by the embodiment of the present application;

FIG. 5 is a schematic diagram of image processing by a segmentation model provided in an embodiment of the present application;

FIG. 6 is a schematic diagram of image processing when obtaining a mask block provided by an embodiment of the present application;

FIG. 7 is a schematic diagram of a display interface for zooming when taking pictures and previewing provided by an embodiment of the present application;

FIG. 8 is a schematic diagram of the process of multi-camera zooming during photo preview provided by the embodiment of the present application;

Fig. 9 is a schematic diagram of a hardware system applicable to the device of the present application;

Fig. 10 is a schematic diagram of a software system applicable to the device of the present application;

FIG. 11 is a schematic structural diagram of an image processing device provided by an embodiment of the present application;

FIG. 12 is a schematic structural diagram of a chip provided in the embodiment of the application.

Detailed ways

The technical solution in this application will be described below with reference to the accompanying drawings.

In the description of the embodiments of this application, unless otherwise specified, "/" means or, for example, A/B can mean A or B; "and/or" in this article is only a description of the association of associated objects A relationship means that there may be three kinds of relationships, for example, A and/or B means: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of the present application, "plurality" refers to two or more than two.

Hereinafter, the terms "first" and "second" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance or implicitly specifying the quantity of indicated technical features. Thus, a feature defined as "first" and "second" may explicitly or implicitly include one or more of these features. In the description of this embodiment, unless otherwise specified, "plurality" means two or more.

First of all, some terms used in the embodiments of the present application are explained to facilitate the understanding of those skilled in the art.

1. RGB (red, green, blue) color space refers to a color model related to the structure of the human visual system. According to the structure of the human eye, all colors are seen as different combinations of red, green and blue.

2. A pixel value refers to a set of color components corresponding to each pixel in a color image located in the RGB color space. For example, each pixel corresponds to a group of three primary color components, wherein the three primary color components are red component R, green component G and blue component B respectively.

3. Image registration refers to the matching of geographic coordinates of different images obtained by different imaging methods in the same area. Among them, it includes the processing of three aspects: geometric correction, projection transformation and unified scale.

4. Field of view (FOV), which is used to indicate the maximum angle range that the camera can capture. If the object to be photographed is within the angle range, the object to be photographed will be captured by the camera. If the object to be photographed is outside the angle range, the object to be photographed will not be captured by the camera.

Generally, the larger the field of view of the camera, the larger the shooting range and the shorter the focal length; while the smaller the field of view of the camera, the smaller the shooting range and the longer the focal length. Therefore, the camera can be divided into a main camera, a wide-angle camera, and a telephoto camera due to different field of view angles. Among them, the field of view of the wide-angle camera is larger than that of the main camera, and the focal length is smaller, which is suitable for close-up shooting; while the field of view of the telephoto camera is smaller than that of the main camera, and the focal length is longer. Suitable for remote shooting.

5. Backlighting, backlighting is a situation where the subject is just between the light source and the camera. In this state, it is easy to cause insufficient exposure of the subject. Therefore, in general, users should try to avoid shooting objects under backlight conditions.

The above is a brief introduction to the nouns involved in the embodiments of the present application, and details will not be repeated below.

With the widespread use of electronic devices, taking pictures with electronic devices has become a daily behavior in people's lives. Taking electronic devices as mobile phones as an example, various multi-frame image synthesis algorithms have emerged to improve image quality, such as: multi-frame noise reduction, multi-frame super-resolution, etc.

However, in some high dynamic range (high dynamic range, HDR) scenes or in backlit scenes, due to the specular reflection on some areas of the object surface in the scene to be shot, the details of these high-brightness areas are lost, and the related technology is not effective. handle the situation.

For example, Fig. 1 is an image captured by using related technologies. As shown in Figure 1, there are three people in the scene to be shot waiting for the user to take a photo in the sun. Since the sun shines on the face area, and the sunlight is very strong, the face area produces high light reflection, and the face area is high Brightness area. At this time, when the user uses the relevant technology to shoot the three people, the captured image loses the details of the face area, resulting in poor image quality and the content of the face area cannot be seen clearly, which affects the user experience.

In view of this, the embodiment of the present application provides an image processing method, by collecting the first image and the second image with different resolutions, using the content corresponding to the high-brightness area in the clearer second image, and the low-definition first image The high-brightness area in the first image is fused, so that the missing details in the high-brightness area in the first image can be recovered, and then a higher-quality captured image can be obtained through multiple fusions to improve user experience.

Firstly, the application scenarios of the embodiments of the present application are briefly described.

Fig. 2 is a schematic diagram of an application scenario provided by an embodiment of the present application. The image processing method provided in this application can be applied to restore the details of the high-brightness area in the image.

Exemplarily, as shown in (a) in FIG. 2, it is a graphical user interface (graphical user interface, GUI) of the electronic device. After the electronic device detects that the user clicks the icon of the camera application on the interface, the camera application can be started, and another GUI as shown in (b) in FIG. 2 is displayed, which can be called a preview interface.

The preview interface may include a viewfinder window 21 . In the preview state, the preview image can be displayed in the viewfinder window 21 in real time. The preview interface may also include a variety of shooting mode options and a first control, that is, the shooting key 11 . The multiple shooting mode options include, for example: a shooting mode, a video recording mode, etc., and the shooting key 11 is used to indicate that the current shooting mode is a shooting mode, a video recording mode or other modes. Among them, when the camera application is opened, it is generally in the camera mode by default.

Exemplarily, as shown in (b) of FIG. 2 , when the electronic device starts the camera application, the electronic device runs the program corresponding to the image processing method, and acquires and stores the captured image in response to the user's click operation on the shooting key 11 .

It should be understood that there are 3 people in the scene to be photographed, and since the sun shines on the faces of the 3 people, and the sunlight is very strong, the facial areas of the 3 people produce high light reflections, so when taking pictures of the 3 people, use related technologies Facial features of 3 people are usually not available. However, the image processing method of the present application can detect the highlighted face area, and then restore the details of the face area to obtain a high-quality captured image.

It should be understood that the foregoing is an illustration of an application scenario, and does not impose any limitation on the application scenario of the present application.

The image processing method provided by the embodiment of the present application will be described in detail below with reference to the drawings in the description.

FIG. 3 is a schematic flowchart of an image processing method provided by an embodiment of the present application. As shown in FIG. 3 , the image processing method includes the following S10-S60.

The electronic device starts the camera, and displays a preview interface as shown in (b) in FIG.

S10. After the electronic device detects the first operation of the camera key 11 by the user, in response to the first operation, the first camera captures the first image, and the second camera captures the second image.

Wherein, the first image and the second image are images captured for the same scene to be captured. The definition of the first image is lower than that of the second image, and the first image includes a first area, where the first area is an area in the first image whose resolution is less than a preset threshold.

Wherein, the preset threshold can be set and modified according to needs, which is not limited in this embodiment of the present application.

It should be understood that both the first image and the second image are Bayer format images, and may also be referred to as images in the RAW domain.

It should be understood that the first area is used to represent an area in the first image that is unclear and lacks details. Exemplarily, the first area may refer to a high-brightness area where details are missing due to strong illumination when the first image is acquired, or may refer to a key area where details are missing when the first image is acquired, for example, a human face, Human body, facial features, etc.

S20. According to the first image, determine whether a mask (mask) block is obtained, and the mask block corresponds to the first area.

It should be understood that the mask block refers to a mask image corresponding to the first region in the first image. The processing of the first region in the first image where details need to be restored is controlled by replacing or merging the first region in the first image where details are missing.

S30. Regardless of whether the mask block is obtained, fuse the first image and the second image to obtain a first fused image.

It should be understood that since the definition of the second image is higher than that of the first image, the definition of the overall image can be improved after the fusion of the first image and the second image, and a first fused image with higher definition can be obtained.

S40. If so, then according to the mask block, determine the first image block in the first image, determine the second image block in the second image, the first image block corresponds to the mask block, and the second image block corresponds to the mask block correspond.

For example, when a user uses an electronic device to take pictures of three colleagues, the first area may refer to the face areas where the three colleagues are illuminated by strong light so that facial features cannot be seen clearly, and the generated mask block corresponds to the first area , used to represent the face area. The first image block is the human face area determined from the first image, and the second image block is the human face area determined from the second image.

S50. Fusion the first image block and the second image block to obtain a fused image block.

It should be understood that since the definition of the first image is lower than that of the second image, the definition of the first image block is also lower than that of the second image block, and even there is no detail in the first image block, Thus, the first image block that is not clear and lacks details is fused with the second image block that is clear and rich in details, and a fused image block with higher definition can be obtained.

S60. Fusion the first fused image and the fused image block to obtain a third image.

It should be understood that the overall definition of the first fused image is improved relative to the first image, the fused image block is partially improved relative to the first image block in the first image, and the first fused image By merging with the fused image block, the part in the first fused image can be further repaired to obtain a third image with higher definition.

An embodiment of the present application provides an image processing method, by determining the mask block corresponding to the first area missing details from the low-resolution first image, and then obtaining the first mask block corresponding to the mask block from the first image. An image block, and obtaining a second image block corresponding to the mask block from a second image with high definition and rich details, and fusing the first image block and the second image block to obtain a clear fused image block; then The first fused image obtained by fusing the first image and the second image is further fused with the fused image block to restore missing details and obtain a high-definition third image.

FIG. 4 is a schematic flowchart of another image processing method provided by an embodiment of the present application.

As shown in FIG. 4 , the image processing method 10 includes: S110 to S190.

S110. Acquire a first image and a second image. The first image and the second image are images captured for the same scene to be captured. The resolution of the first image is lower than that of the second image.

It should be understood that the first image and the second image are images captured by the electronic device through a camera, or the first image and the second image may also be images obtained from inside the electronic device, for example, images stored in the electronic device, or , the image obtained by the electronic device from the cloud. Wherein, both the first image and the second image are Bayer format images.

It should be understood that when the camera is used to acquire the first image and the second image, generally, the corresponding low-resolution image of the two images is called the first image; and the corresponding high-resolution image is called the second image. Since the definition is relative, the first image and the second image are also relative.

For example, when the resolution corresponding to image a is lower than that corresponding to image b, when the image processing method provided by the embodiment of the present application is used to perform image processing on image a and image b, image a is the first image, and image b That is the second image.

When the definition corresponding to image b is lower than that corresponding to image c, when using the image processing method provided by the embodiment of the present application to perform image processing on image b and image c, image b is the first image, and image c is second image.

For example, the first image is an image collected by a wide-angle camera, the second image is an image collected by a telephoto camera, and the wide-angle camera and the telephoto camera collect images at the same time; similarly, the first image is an image collected by a wide-angle camera , the second image is an image collected by the ultra-wide-angle camera, and the wide-angle camera and the ultra-wide-angle camera collect images at the same time.

It should be understood that the first image may be an image with a region of missing details, and the missing details in the first image may be restored through the image processing method of the embodiment of the present application.

S120. Input the first image into the segmentation model for segmentation. If the first image is segmentable, generate a corresponding mask block according to the segmented first region. If it is not segmentable, it means that the first image does not include the first region. By Therefore, mask blocks cannot be obtained.

Wherein, the segmentation model is used to segment the first region in the first image, and generate a mask block corresponding to the first region. The first area is used to represent an area in the first image whose sharpness is less than a preset threshold, that is, an area lacking certain details.

Optionally, the segmentation model may be: a fully convolutional neural network (fully convolutional networks, FCN) and the like.

It should be understood that the segmentation model may segment the first image to obtain a plurality of segmented image regions, where the plurality of image regions include some regions containing details, and may also include some regions lacking details. When the first image contains one or more regions with missing details, the segmentation model can segment the one or more regions with missing details, and generate corresponding one or more mask blocks. However, when the first image does not contain areas with missing details, the segmentation model will not segment out areas with missing details, let alone generate corresponding mask blocks.

It should be understood that the first area may refer to a high-brightness area where details are missing due to strong illumination when the first image is acquired, for example, in an HDR scene, or may also refer to a key area where details are missing when the first image is acquired, For example, human face, human body, facial features, etc.

It should also be understood that the number of first regions is the same as the number of mask blocks, and the range of viewing angles of the mask blocks corresponding to each first region is the same as the range of viewing angles corresponding to the first region.

Exemplarily, FIG. 5 shows a schematic diagram of processing an image by a segmentation model provided in an embodiment of the present application. As shown in Figure 5, the first image is input into the segmentation model. Since the first image contains three face areas that are illuminated by strong light and lack details, the segmentation model can segment three first areas and generate corresponding The 3 mask blocks. For example, the three mask blocks corresponding to the face area in Fig. 1 .

Exemplarily, the pixel value of each pixel corresponding to the mask block is 0.

S130. When the mask block is obtained or not obtained, register the first image and the second image.

It should be understood that when no mask blocks are obtained, it means that although the first image has low resolution, it does not include the first region, that is, there is no region below the preset threshold and seriously missing details.

Optionally, as an example, the registration may be global registration.

Global registration is used to register all the content in multiple images, that is to say, here you can register all the content in the first image and the second image, so that the first image and the second image are in In the subsequent fusion, it can correspond more accurately.

Optionally, as an example, the registration may include global registration and local registration.

Local registration is used to refer to the registration of local content in multiple images. Exemplarily, although the first region is not segmented from the first image by using the segmentation model, some other regions can be segmented, for example, the human body region and the background region other than the human body region, thus, the first The body region in the image is locally registered with the body region in the second image without registering the background region in the first image with the background region in the second image.

Exemplarily, the global registration can be performed first, and then the local registration can be performed, or the local registration can be performed first, and then the global registration can be performed, and the registration order can be set and adjusted as required. No restrictions are imposed.

It should be understood that after the first image and the second image are registered, the accuracy of the subsequent fusion can be improved, and the fusion effect is better.

S140. Fusion the registered first image and the second image by using the first fusion model to obtain a first fusion image.

It should be understood that, after registration, the definition of the first image and the second image are still different, and the first fusion model can fuse images with different resolutions. Wherein, the first fusion model may be a VGG net model.

It should be understood that since the second image has a higher definition than the first image, after the registration of the first image and the second image are fused, the field angle range corresponding to the second image in the first image can be improved The sharpness of the content in the image, so as to obtain the first fused image with higher definition.

Wherein, the field angle range of the first fused image is the same as the field angle range of the first image.

Of course, the first image and the second image may not be registered, and the acquired first image and the second image may be fused using the first fusion model to obtain the first fused image.

S150. When the mask block is obtained, determine a first image block in the first image and determine a second image block in the second image according to the mask block.

It should be understood that when one mask block is determined from the first image by using the segmentation model, one first image block corresponding to the mask block can be determined from the first image according to the mask block. Wherein, the field angle ranges of the first image block and the mask block are the same.

Similarly, according to the mask block, a corresponding second image block can be determined from the second image. Wherein, the field angle range of the second image block is the same as that of the mask block.

Based on this, when using the segmentation model to determine a plurality of mask blocks from the first image, according to the plurality of mask blocks, it can be determined from the first image that each mask block in the plurality of mask blocks Corresponding to one first image block, that is to say, the same number of multiple first image blocks can be determined from the first image, and the first image block corresponds to the mask block one by one.

Similarly, according to the plurality of mask blocks, a second image block corresponding to each mask block in the plurality of mask blocks can also be determined from the second image, that is, it can be obtained from the second image A plurality of second image blocks of the same number are determined, and the second image blocks are in one-to-one correspondence with the mask block and the first image block.

S160. Register the first image block and the second image block.

It should be understood that, according to the mask block, there is a one-to-one correspondence between the acquired first image block and the second image block, and the corresponding first image block and the second image block may form a group of image blocks. Therefore, registering the first image block and the second image block refers to registering the first image block and the second image block in each group of image blocks.

Optionally, as an example, the registration may be global registration.

The first image block and the second image block in each group of image blocks are globally registered. Here, it refers to registering all contents of the first image block and all contents of the second image block in each group of image blocks.

Optionally, as another example, the registration may include global registration and local registration.

Exemplarily, the first image block and the second image block in each group of image blocks are firstly registered globally, and then locally registered. Here, local registration refers to registering the local content of the first image block in each group of image blocks with the local content of the second image block. For example, if both the first image block and the second image block include a human face, the corresponding regions of the eyes in the human face in the first image block and the second image block are respectively registered, and the eyes in the human face are The mouth is registered in the regions corresponding to the first image block and the second image block respectively.

It should be understood that global registration and local registration are relative, and when the image area is reduced, the accuracy and effect of global registration and local registration are higher.

In addition, it should be noted that, when performing local registration on the first image and the second image, for example, when performing local registration on the human body area in the first image and the second image, correspondingly, the adjacent background area will be affected by Influence, but become inaccurate, resulting in errors. Therefore, in order to avoid unnecessary influence on the background area, the present application extracts the first image block from the first image, extracts the second image block from the second image, and then extracts the first image block Global registration is performed with the second image block, so that the background area is isolated without affecting the surrounding background area. Moreover, local registration may continue to be performed on the first image block and the second image block, so as to improve the registration accuracy, and obtain the first image block and the second image block with higher registration accuracy.

S170. Use the second fusion model to fuse the first image block and the second image block in each group of image blocks after registration to obtain a fused image block.

It should be understood that the first image block and the second image block after registration still have different resolutions, and the second fusion model can fuse image blocks with different resolutions.

It should be understood that since the definition of the second image is higher than that of the first image, the definition of the second image block is higher than that of the first image block, thus, the registered first image block and the second image After the blocks are fused, a fused image block with higher definition can be obtained.

Wherein, the field angle range of the fused image block is the same as the field angle ranges of the first image block and the second image block.

In the embodiment of the present application, the second fusion model is a pre-trained fusion model. The training image set may include an original image and a manually marked mask block, the mask block is used to identify a first region of missing details in the original image. For example, original images refer to images in various HDR scenarios. On each original image, there are 1 or more mask blocks that indicate high-brightness regions (ie, the first region where details are missing) are manually marked.

In one example, the second fusion model is trained from the first fusion model.

In another example, the second fusion model is trained by adding random highlight noise to the first fusion model.

Due to the addition of random highlight noise during training, when the first image block and the second image block are fused by using the trained second fusion model, the second image block with higher definition is larger than that of the first image block. The proportion of the weight is larger, so that the fused image block obtained by fusion can obtain more details from the second image block.

S180. Use the third fusion model to fuse the first fusion image and the fusion image block to obtain a third image.

Optionally, the third blending model may be a Laplacian blending model.

When using the Laplacian fusion model for fusion, the Laplacian fusion model can first decompose the first fused image and the fused image block into different spatial frequency bands, and then perform fusion on each spatial frequency band layer, thus , through the frequency division processing, the fusion of the first fused image and the fused image block can be made more natural, the joint is more delicate, and the obtained third image has higher quality.

It should be understood that, since the fused image block has higher definition than the corresponding region in the first fused image, a third image with higher definition can be obtained after fusion by using the third fusion model.

S190. When no mask block is obtained, output the first fused image as a captured image. When the mask block is obtained, the third image is output as a captured image.

Exemplarily, FIG. 6 shows a schematic diagram of processing an image when obtaining a mask block according to an embodiment of the present application. As shown in Figure 6, the first image is input into the segmentation model. Since the first image contains three face areas that are illuminated by strong light and lack details, the segmentation model can segment three first areas and generate corresponding The 3 mask blocks.

At this time, the first image and the second image are registered, and the registered first image and the second image are fused using the first fusion model to obtain the first fusion image.

At the same time, according to the 3 mask blocks, the corresponding 3 first image blocks in the first image are obtained, and the corresponding 3 second image blocks in the second image are obtained, and then, the first image corresponding to the same mask block The block and the second image block are registered and fused by using the second fusion model to obtain a fused image block, thus, three fused image blocks can be obtained.

Then the first fused image and the 3 fused image blocks are fused using the third fused model to obtain the third image.

In the embodiment of the present application, when the mask block is not obtained from the first image by using the segmentation model, only the first fusion model is used to register and fuse the first image and the second image, and the obtained first The fused images are taken as captured images. However, when the segmentation model is used to obtain the mask block from the first image, it means that there are areas with missing details in the first image. At this time, the first image and the second image are first fused to obtain a large-scale definition improvement. Then, according to the mask block, the first image block is obtained from the first image, the second image block is obtained from the second image, and the first image block and the second image block are registered and Fusion, thereby obtaining a fused image block that effectively restores clarity and details, and then further fusing the first fused image with the fused image block to repair missing details and obtain a high-definition, high-quality captured image.

It should be understood that the above illustrations are intended to help those skilled in the art understand the embodiments of the present application, rather than to limit the embodiments of the present application to the illustrated specific values or specific scenarios. Those skilled in the art can obviously make various equivalent modifications or changes based on the above illustrations given, and such modifications or changes also fall within the scope of the embodiments of the present application.

The image processing method of the embodiment of the present application has been described in detail above with reference to FIGS. 2 to 6. In combination with the above content, the first image and the second image are captured by two cameras. However, current electronic devices usually include 3 or more The camera, therefore, needs to trigger two different cameras at different focal lengths to acquire the first image and the second image.

Taking an electronic device including a wide-angle camera, a main camera, and a telephoto camera as an example, the trigger method provided by this application will be described in detail below.

Exemplarily, the zoom factor range corresponding to the electronic device is set to [0.4, 100]. The zoom factor range is divided into three zoom factor ranges, and the three zoom factor ranges are respectively the first zoom factor range, the second zoom factor range, and the third zoom factor range, and the zoom factors included in the three zoom factor ranges are sequentially increase.

Exemplarily, it is assumed that the first zoom factor range F1 is [0.4, 0.9), the second zoom factor range F2 is [0.9, 3.5), and the third zoom factor range F3 is [3.5, 100]. It should be understood that, here, each number is only for illustration, which can be set and changed as required, and is not limited in this embodiment of the present application.

Exemplarily, the applicable zoom factor range of the wide-angle camera itself is [0.4,1], the applicable zoom factor range of the main camera itself is [0.6,3.5], and the applicable zoom factor range of the telephoto camera itself is [2.0,100] ].

Based on this, the target camera corresponding to the first zoom range is set as the wide-angle camera, the target camera corresponding to the second zoom range is the main camera, and the target camera corresponding to the third zoom range is set as the telephoto camera.

FIG. 7 is a schematic diagram of an interface for zooming during photo preview provided by an embodiment of the present application. FIG. 8 shows a schematic diagram of a process of multi-camera zooming during photo preview provided by an embodiment of the present application.

Exemplarily, in response to a user's touch operation, when the electronic device 100 runs the camera application, the electronic device 100 displays a preview interface as shown in (a) in FIG. 7 . On the preview interface, the shooting key 11 indicates that the current shooting mode is the shooting mode. The preview interface also includes a viewfinder window 21, and the viewfinder window 21 can be used to display a preview image before taking pictures in real time. In addition, a zoom option 22 is also displayed on the preview screen. The user can select the zoom factor of the current photo taking in the zoom option 22, for example, 0.4 times, 2 times or 50 times. As shown in (b) in Figure 7, in response to the user's zoom operation, the preview image can be enlarged or reduced according to the currently selected zoom factor, and as the zoom factor is enlarged or reduced, the preview image in the viewfinder window 21 also becomes larger or smaller. zoom out. When zooming to a certain switching point, two different cameras are invoked to acquire captured images by using the image processing method provided by the embodiment of the present application.

As shown in Figure 8, when taking pictures and previewing, when zooming from small to large within the first zoom multiple range F1, the wide-angle camera corresponding to the first zoom multiple range is in the foreground sending display state, and the acquired image is sent to the display screen show.

When zooming to the first zoom switching point (for example, 0.6X), the wide-angle camera continues to be in the foreground display state, and the main camera corresponding to the second zoom range F2 starts to enter the background operation state.

Since the wide-angle camera has a larger field of view and lower definition than the main camera, therefore, within the zoom range F11 of [0.6,0.9], in response to the user's operation on the shooting key 11, the wide-angle camera captures The image is used as the first image, and the image obtained by the main camera is used as the second image. Then, based on the first image obtained by the wide-angle camera and the second image obtained by the main camera, using the image processing method provided in the embodiment of the present application, a clear Capture images with high resolution and rich details.

When zooming to 0.9X, the wide-angle camera is turned off, and the main camera switches to the foreground sending display state, that is, the main camera sends the acquired image to the display screen for display.

When zooming to the second zoom switching point (for example, 2.0X), the wide-angle camera continues to be in the foreground display state, and the telephoto camera corresponding to the third zoom multiple range F3 starts to enter the background operation state.

Compared with the telephoto camera, the image captured by the main camera has low resolution and a large field of view. Therefore, within the zoom range F21 of [2.0, 3.5], in response to the user's operation on the shooting key 11, the main camera The acquired image is used as the first image, and the image acquired by the telephoto camera is used as the second image. Then, based on the first image acquired by the main camera and the second image acquired by the telephoto camera, the image processing method provided by the embodiment of the present application is used. , to obtain images with high definition and rich details.

When zooming to 3.5X, the main camera is turned off, and the telephoto camera switches to the foreground sending display state, that is, the telephoto camera sends the acquired image to the display screen for display.

The image processing method of the embodiment of the present application and the trigger conditions of different cameras are described in detail above in conjunction with FIG. 2 to FIG. 8 . The software system and hardware of the electronic device applicable to this application will be described in detail below in conjunction with FIG. 9 to FIG. 12 Systems, devices and chips. It should be understood that the software systems, hardware systems, devices, and chips in the embodiments of the present application can execute various image processing methods in the foregoing embodiments of the present application, that is, the specific working processes of the following various products can refer to the above-mentioned method embodiments corresponding process.

The image processing method provided in the embodiment of the present application may be applicable to various electronic devices, and correspondingly, the image processing apparatus provided in the embodiment of the present application may be electronic devices in various forms.

In some embodiments of the present application, the electronic device may be various camera devices such as SLR cameras and card players, mobile phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (augmented reality, AR)/virtual reality (virtual reality) reality, VR) equipment, notebook computer, ultra-mobile personal computer (ultra-mobile personal computer, UMPC), netbook, personal digital assistant (personal digital assistant, PDA), etc., or other equipment or devices capable of image processing, The embodiment of the present application does not set any limitation on the specific type of the electronic device.

Hereinafter, the electronic device is taken as an example of a mobile phone, and FIG. 9 shows a schematic structural diagram of an electronic device 100 provided in an embodiment of the present application.

The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (universal serial bus, USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, and an antenna 2 , mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, earphone jack 170D, sensor module 180, button 190, motor 191, indicator 192, camera 193, display screen 194, and A subscriber identification module (subscriber identification module, SIM) card interface 195 and the like. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, bone conduction sensor 180M, etc.

It should be noted that the structure shown in FIG. 1 does not constitute a specific limitation on the electronic device 100 . In other embodiments of the present application, the electronic device 100 may include more or fewer components than those shown in FIG. 1 , or the electronic device 100 may include a combination of some of the components shown in FIG. 1 , or , the electronic device 100 may include subcomponents of some of the components shown in FIG. 1 . The components shown in FIG. 1 can be realized in hardware, software, or a combination of software and hardware.

The processor 110 may include one or more processing units, for example: the processor 110 may include an application processor (application processor, AP), a modem processor, a graphics processing unit (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), controller, video codec, digital signal processor (digital signal processor, DSP), baseband processor, and/or neural network processor (neural-network processing unit, NPU), etc. Wherein, different processing units may be independent devices, or may be integrated in one or more processors.

Wherein, the controller may be the nerve center and command center of the electronic device 100 . The controller can generate an operation control signal according to the instruction opcode and timing signal, and complete the control of fetching and executing the instruction.

A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. The memory may hold instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to use the instruction or data again, it can be called directly from the memory. Repeated access is avoided, and the waiting time of the processor 110 is reduced, thereby improving the efficiency of the system.

The processor 110 may run the software code of the image processing method provided in the embodiment of the present application to capture an image with higher definition.

In some embodiments, processor 110 may include one or more interfaces. The interface may include an integrated circuit (inter-integrated circuit, I2C) interface, an integrated circuit built-in audio (inter-integrated circuit sound, I2S) interface, a pulse code modulation (pulse code modulation, PCM) interface, a universal asynchronous transmitter (universal asynchronous receiver/transmitter, UART) interface, mobile industry processor interface (mobile industry processor interface, MIPI), general-purpose input and output (general-purpose input/output, GPIO) interface, subscriber identity module (subscriber identity module, SIM) interface, and /or universal serial bus (universal serial bus, USB) interface, etc.

The MIPI interface can be used to connect the processor 110 with peripheral devices such as the display screen 194 and the camera 193 . MIPI interface includes camera serial interface (camera serial interface, CSI), display serial interface (display serial interface, DSI), etc. In some embodiments, the processor 110 communicates with the camera 193 through the CSI interface to realize the shooting function of the electronic device 100 . The processor 110 communicates with the display screen 194 through the DSI interface to realize the display function of the electronic device 100 .

The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 with the camera 193 , the display screen 194 , the wireless communication module 160 , the audio module 170 , the sensor module 180 and so on. The GPIO interface can also be configured as an I2C interface, I2S interface, UART interface, MIPI interface, etc.

The USB interface 130 is an interface conforming to the USB standard specification, specifically, it can be a Mini USB interface, a Micro USB interface, a USB Type C interface, and the like. The USB interface 130 can be used to connect a charger to charge the electronic device 100 , and can also be used to transmit data between the electronic device 100 and peripheral devices. It can also be used to connect headphones and play audio through them. This interface can also be used to connect other electronic devices, such as AR devices.

It can be understood that the interface connection relationship between the modules shown in the embodiment of the present application is only a schematic illustration, and does not constitute a structural limitation of the electronic device 100 . In other embodiments of the present application, the electronic device 100 may also adopt different interface connection manners in the foregoing embodiments, or a combination of multiple interface connection manners.

The charging management module 140 is configured to receive a charging input from a charger. Wherein, the charger may be a wireless charger or a wired charger. In some embodiments of wired charging, the charging management module 140 can receive the current of the wired charger through the USB interface 130 . In some wireless charging embodiments, the charging management module 140 can receive electromagnetic waves through the wireless charging coil of the electronic device 100 (the current path is shown as a dotted line). While the charging management module 140 is charging the battery 142 , it can also supply power to the electronic device 100 through the power management module 141 .

The power management module 141 is used for connecting the battery 142 , the charging management module 140 and the processor 110 . The power management module 141 receives the input from the battery 142 and/or the charging management module 140 to provide power for the processor 110 , the internal memory 121 , the display screen 194 , the camera 193 , and the wireless communication module 160 .

The wireless communication function of the electronic device 100 can be realized by the antenna 1 , the antenna 2 , the mobile communication module 150 , the wireless communication module 160 , a modem processor, a baseband processor, and the like.

Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 may be used to cover single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization of the antennas. For example: Antenna 1 can be multiplexed as a diversity antenna of a wireless local area network. In other embodiments, the antenna may be used in conjunction with a tuning switch.

The mobile communication module 150 may provide a wireless communication solution applied to the electronic device 100, such as at least one of the following solutions: a second generation (2th generation, 2G) mobile communication solution, a third generation (3th generation, 3G) Mobile communication solutions, fourth generation (4th generation, 5G) mobile communication solutions, fifth generation (5th generation, 5G), sixth generation (6th generation, 6G) mobile communication solutions. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (low noise amplifier, LNA) and the like. The mobile communication module 150 can receive electromagnetic waves through the antenna 1, filter and amplify the received electromagnetic waves, and send them to the modem processor for demodulation. The mobile communication module 150 can also amplify the signals modulated by the modem processor, and convert them into electromagnetic waves through the antenna 1 for radiation. In some embodiments, at least part of the functional modules of the mobile communication module 150 may be set in the processor 110 . In some embodiments, at least part of the functional modules of the mobile communication module 150 and at least part of the modules of the processor 110 may be set in the same device.

The wireless communication module 160 can provide wireless local area networks (wireless local area networks, WLAN) (such as wireless fidelity (Wireless Fidelity, Wi-Fi) network), bluetooth (bluetooth, BT), global navigation satellite, etc. applied on the electronic device 100. System (global navigation satellite system, GNSS), frequency modulation (frequency modulation, FM), near field communication technology (near field communication, NFC), infrared technology (infrared, IR) and other wireless communication solutions. The wireless communication module 160 may be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2 , frequency-modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110 . The wireless communication module 160 can also receive the signal to be sent from the processor 110 , frequency-modulate it, amplify it, and convert it into electromagnetic waves through the antenna 2 for radiation.

In some embodiments, the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (general packet radio service, GPRS), code division multiple access (code division multiple access, CDMA), broadband Code division multiple access (wideband code division multiple access, WCDMA), time division code division multiple access (time-division code division multiple access, TD-SCDMA), long term evolution (long term evolution, LTE), BT, GNSS, WLAN, NFC , FM, and/or IR techniques, etc. The GNSS may include a global positioning system (global positioning system, GPS), a global navigation satellite system (global navigation satellite system, GLONASS), a Beidou navigation satellite system (beidou navigation satellite system, BDS), a quasi-zenith satellite system (quasi -zenith satellite system (QZSS) and/or satellite based augmentation systems (SBAS).

The electronic device 100 realizes the display function through the GPU, the display screen 194 , and the application processor. The GPU is a microprocessor for image processing, and is connected to the display screen 194 and the application processor. GPUs are used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

The display screen 194 is used to display images, videos and the like. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active matrix organic light emitting diode or an active matrix organic light emitting diode (active-matrix organic light emitting diode, AMOLED), flexible light-emitting diode (flex light-emitting diode, FLED), Miniled, MicroLed, Micro-oLed, quantum dot light emitting diodes (quantum dot light emitting diodes, QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194 , where N is a positive integer greater than 1.

Camera 193 is used to capture images or videos. It can be triggered by an application command to realize the camera function, such as capturing images of any scene. A camera may include components such as an imaging lens, an optical filter, and an image sensor. The light emitted or reflected by the object enters the imaging lens, passes through the filter, and finally converges on the image sensor. The image sensor is mainly used for converging and imaging the light emitted or reflected by all objects in the camera perspective (also called the scene to be shot, the target scene, or the scene image that the user expects to shoot); the filter is mainly used to It is used to filter out redundant light waves (such as light waves other than visible light, such as infrared) in the light; the image sensor is mainly used to perform photoelectric conversion on the received light signal, convert it into an electrical signal, and input it into the processor 130 for subsequent processing . Wherein, the camera 193 may be located at the front of the electronic device 100, or at the back of the electronic device 100, and the specific number and arrangement of the cameras may be set according to requirements, which are not limited in this application.

Exemplarily, the electronic device 100 includes a front camera and a rear camera. For example, a front camera or a rear camera may include one or more cameras. Taking the electronic device 100 with three rear cameras as an example, in this way, when the electronic device 100 starts shooting with two of the three rear cameras, the image processing method provided in the embodiment of the present application can be used. Alternatively, the camera is arranged on an external accessory of the electronic device 100, the external accessory is rotatably connected to the frame of the mobile phone, and the angle formed between the external accessory and the display screen 194 of the electronic device 100 is 0-360 degrees any angle between. For example, when the electronic device 100 takes a selfie, the external accessory drives the camera to rotate to a position facing the user. Certainly, when the mobile phone has multiple cameras, only some of the cameras may be set on the external accessories, and the rest of the cameras may be set on the electronic device 100 body, which is not limited in this embodiment of the present application.

The internal memory 121 may be used to store computer-executable program codes including instructions. The internal memory 121 may include an area for storing programs and an area for storing data. Wherein, the stored program area can store an operating system, at least one application program required by a function (such as a sound playing function, an image playing function, etc.) and the like. The storage data area can store data created during the use of the electronic device 100 (such as audio data, phonebook, etc.) and the like. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, flash memory device, universal flash storage (universal flash storage, UFS) and the like. The processor 110 executes various functional applications and data processing of the electronic device 100 by executing instructions stored in the internal memory 121 and/or instructions stored in a memory provided in the processor.

The internal memory 121 can also store the software code of the image processing method provided by the embodiment of the present application. When the processor 110 runs the software code, it executes the process steps of the image processing method to obtain an image with higher definition.

The internal memory 121 can also store captured images.

The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, so as to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function. Such as saving files such as music in an external memory card.

Certainly, the software code of the image processing method provided in the embodiment of the present application can also be stored in an external memory, and the processor 110 can run the software code through the external memory interface 120 to execute the process steps of the image processing method to obtain a high-definition image. Image. Images captured by the electronic device 100 may also be stored in an external memory.

It should be understood that the user can designate whether to store the image in the internal memory 121 or the external memory. For example, when the electronic device 100 is currently connected to the external memory, if the electronic device 100 captures one frame of image, a prompt message may pop up to remind the user whether to store the image in the external memory or the internal memory; of course, there may be other specified ways , the embodiment of the present application does not impose any limitation on this; alternatively, when the electronic device 100 detects that the memory capacity of the internal memory 121 is less than a preset amount, it may automatically store the image in the external memory.

The electronic device 100 can implement audio functions through the audio module 170 , the speaker 170A, the receiver 170B, the microphone 170C, the earphone interface 170D, and the application processor. Such as music playback, recording, etc.

The pressure sensor 180A is used to sense the pressure signal and convert the pressure signal into an electrical signal. In some embodiments, pressure sensor 180A may be disposed on display screen 194 .

The gyro sensor 180B can be used to determine the motion posture of the electronic device 100 . In some embodiments, the angular velocity of the electronic device 100 around three axes (ie, x, y and z axes) may be determined by the gyro sensor 180B. The gyro sensor 180B can be used for image stabilization.

The air pressure sensor 180C is used to measure air pressure. In some embodiments, the electronic device 100 calculates the altitude based on the air pressure value measured by the air pressure sensor 180C to assist positioning and navigation.

The magnetic sensor 180D includes a Hall sensor. The electronic device 100 may use the magnetic sensor 180D to detect the opening and closing of the flip leather case. In some embodiments, when the electronic device 100 is a clamshell machine, the electronic device 100 can detect opening and closing of the clamshell according to the magnetic sensor 180D. Furthermore, according to the detected opening and closing state of the leather case or the opening and closing state of the flip cover, features such as automatic unlocking of the flip cover are set.

The acceleration sensor 180E can detect the acceleration of the electronic device 100 in various directions (generally three axes). When the electronic device 100 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of electronic devices, and can be used in applications such as horizontal and vertical screen switching, pedometers, etc.

The distance sensor 180F is used to measure the distance. The electronic device 100 may measure the distance by infrared or laser. In some embodiments, when shooting a scene, the electronic device 100 may use the distance sensor 180F for distance measurement to achieve fast focusing.

Proximity light sensor 180G may include, for example, light emitting diodes (LEDs) and light detectors, such as photodiodes. The light emitting diodes may be infrared light emitting diodes. The electronic device 100 emits infrared light through the light emitting diode. Electronic device 100 uses photodiodes to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it may be determined that there is an object near the electronic device 100 . When insufficient reflected light is detected, the electronic device 100 may determine that there is no object near the electronic device 100 . The electronic device 100 can use the proximity light sensor 180G to detect that the user is holding the electronic device 100 close to the ear to make a call, so as to automatically turn off the screen to save power. The proximity light sensor 180G can also be used in leather case mode, automatic unlock and lock screen in pocket mode.

The ambient light sensor 180L is used for sensing ambient light brightness. The electronic device 100 can adaptively adjust the brightness of the display screen 194 according to the perceived ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180L can also cooperate with the proximity light sensor 180G to detect whether the electronic device 100 is in the pocket, so as to prevent accidental touch.

The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to implement fingerprint unlocking, access to application locks, take pictures with fingerprints, answer incoming calls with fingerprints, and the like.

The temperature sensor 180J is used to detect temperature. In some embodiments, the electronic device 100 uses the temperature detected by the temperature sensor 180J to implement a temperature treatment strategy. For example, when the temperature reported by the temperature sensor 180J exceeds the threshold, the electronic device 100 may reduce the performance of the processor located near the temperature sensor 180J, so as to reduce power consumption and implement thermal protection. In some other embodiments, when the temperature is lower than another threshold, the electronic device 100 heats the battery 142 to avoid abnormal shutdown of the electronic device 100 caused by the low temperature. In some other embodiments, when the temperature is lower than another threshold, the electronic device 100 boosts the output voltage of the battery 142 to avoid abnormal shutdown caused by low temperature.

The touch sensor 180K is also called "touch device". The touch sensor 180K can be disposed on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also called a “touch screen”. The touch sensor 180K is used to detect a touch operation on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194 . In other embodiments, the touch sensor 180K may also be disposed on the surface of the electronic device 100 , which is different from the position of the display screen 194 .

The bone conduction sensor 180M can acquire vibration signals. In some embodiments, the bone conduction sensor 180M can acquire the vibration signal of the vibrating bone mass of the human voice. The bone conduction sensor 180M can also contact the human pulse and receive the blood pressure beating signal. In some embodiments, the bone conduction sensor 180M can also be disposed in the earphone, combined into a bone conduction earphone. The audio module 170 can analyze the voice signal based on the vibration signal of the vibrating bone mass of the voice part acquired by the bone conduction sensor 180M, so as to realize the voice function. The application processor can analyze the heart rate information based on the blood pressure beating signal acquired by the bone conduction sensor 180M, so as to realize the heart rate detection function.

The keys 190 include a power key, a volume key and the like. The key 190 may be a mechanical key. It can also be a touch button. The electronic device 100 can receive key input and generate key signal input related to user settings and function control of the electronic device 100 .

The motor 191 can generate a vibrating reminder. The motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations applied to different applications (such as taking pictures, playing audio, etc.) may correspond to different vibration feedback effects.

The indicator 192 can be an indicator light, and can be used to indicate charging status, power change, and can also be used to indicate messages, missed calls, notifications, and the like.

The SIM card interface 195 is used for connecting a SIM card. The SIM card can be connected and separated from the electronic device 100 by inserting it into the SIM card interface 195 or pulling it out from the SIM card interface 195 .

The hardware system of the electronic device 100 is described in detail above, and the software system of the electronic device 100 is introduced below. The software system may adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application uses a layered architecture as an example to exemplarily describe the software system of the electronic device 100 .

As shown in Figure 10, a software system adopting a layered architecture is divided into several layers, and each layer has a clear role and division of labor. Layers communicate through software interfaces. In some embodiments, the software system can be divided into five layers, which are application layer 210 , application framework layer 220 , hardware abstraction layer 230 , driver layer 240 and hardware layer 250 from top to bottom.

The application layer 210 may include application programs such as camera and gallery, and may also include application programs such as calendar, call, map, navigation, WLAN, Bluetooth, music, video, and short message.

The application framework layer 220 provides application program access interfaces and programming frameworks for the applications of the application layer 210 .

For example, the application framework layer includes a camera access interface, and the camera access interface is used to provide camera shooting services through camera management and camera equipment.

Camera management in the application framework layer is used to manage cameras. Camera management can obtain camera parameters, such as judging the working status of the camera.

The camera device in the application framework layer is used to provide a data access interface between different camera devices and camera management.

The hardware abstraction layer 230 is used to abstract hardware. For example, the hardware abstraction layer can include the camera hardware abstraction layer and other hardware device abstraction layers; the camera hardware abstraction layer can include camera device 1, camera device 2, etc.; the camera hardware abstraction layer can be connected with the camera algorithm library, and the camera hardware abstraction layer Algorithms in the camera algorithm library can be called.

The driver layer 240 is used to provide drivers for different hardware devices. For example, the driver layer may include camera drivers; digital signal processor drivers and graphics processor drivers.

The hardware layer 250 may include sensors, image signal processors, digital signal processors, graphics processors, and other hardware devices. Wherein, the sensor may include sensor 1, sensor 2, etc., and may also include a depth sensor (time of flight, TOF) and a multispectral sensor.

In the following, the workflow of the software system of the electronic device 100 is exemplarily described in conjunction with displaying a photographing scene.

When the user performs a click operation on the touch sensor 180K, after the camera APP is awakened by the click operation, it calls each camera device of the camera hardware abstraction layer through the camera access interface. Exemplarily, the camera hardware abstraction layer judges that the current zoom factor is between the range of [0.6, 0.9] zoom factor, so it can send instructions to the camera device driver to call the wide-angle camera and the main camera, and the camera algorithm library Start to load the algorithm in the network model used in the embodiment of this application.

When the sensor at the hardware layer is invoked, for example, sensor 1 in the wide-angle camera is invoked to obtain the first image, and sensor 2 in the main camera captures the second image, and the first image and the second image are sent to image signal processing for processing Preliminary processing such as registration, after processing, the camera device driver returns to the hardware abstraction layer, and then uses the algorithm in the loaded camera algorithm library for processing, such as using the segmentation model, the first fusion model, the second fusion model and the third fusion model according to The relevant processing steps provided in the embodiment of the present application are processed to obtain the captured image. Wherein, the segmentation model, the first fusion model, the second fusion model and the third fusion model can be processed by calling the digital signal processor through the driver of the digital signal processor, and calling the graphics processor through the driver of the graphics processor.

The captured images are sent back to the camera application via the camera hardware abstraction layer and the camera access interface for display and storage.

FIG. 11 is a schematic structural diagram of an image processing device provided by an embodiment of the present application. As shown in FIG. 11 , the image processing device 300 includes an acquisition module 310 and a processing module 320 .

The image processing device 300 can perform the following schemes:

The acquisition module 310 is configured to acquire a first image and a second image, the resolution of the first image is lower than that of the second image, the first image includes a first area, and the first area is that the resolution of the first image is smaller than a preset Threshold area.

A processing module 320, configured to input the first image into a segmentation model to determine whether a mask block is obtained, wherein the segmentation model is used to segment the first region in the first image and generate a mask block corresponding to the first region , the first region is used to represent the region of missing detail in the first image.

The processing module 320 is further configured to fuse the first image and the second image using the first fusion model to obtain a first fusion image.

When the mask block is obtained, the processing module 320 is further configured to determine the first image block in the first image according to the mask block, determine the second image block in the second image, and combine the first image block and the second image block The blocks are fused using the second fusion model to obtain fused image blocks.

The processing module 320 is further configured to fuse the first fusion image and the fusion image block by using the third fusion model to obtain a captured image.

Optionally, as an embodiment, when the mask block is not obtained, the processing module 320 will fuse the first image and the second image using the first fusion model to obtain the first fusion image.

Optionally, as an embodiment, the processing module 320 is further configured to register the first image and the second image.

Optionally, as an embodiment, the processing module 320 is further configured to register the first image block and the second image block.

Registration includes: global registration and/or local registration. Global registration is used to register all content in multiple images, and local registration is used to register local content in multiple images.

Optionally, as an embodiment, the processing module 320 is further configured to use the training image set and add random highlight noise to train the first fusion model to obtain the second fusion model, wherein the training image set includes the original image, the original Images are annotated with mask blocks.

Optionally, as an embodiment, the third fusion model is a Laplace fusion model.

It should be noted that, the image processing apparatus 300 described above is embodied in the form of functional modules. The term "module" here may be implemented in the form of software and/or hardware, which is not specifically limited.

For example, a "module" may be a software program, a hardware circuit or a combination of both to realize the above functions. The hardware circuitry may include application specific integrated circuits (ASICs), electronic circuits, processors (such as shared processors, dedicated processors, or group processors) for executing one or more software or firmware programs. etc.) and memory, incorporating logic, and/or other suitable components to support the described functionality.

Therefore, the modules of each example described in the embodiments of the present application can be realized by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed by hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be regarded as exceeding the scope of the present application.

The embodiment of the present application also provides another electronic device, including a camera module, a processor, and a memory.

The camera module is used to acquire a first image and a second image, the first image and the second image are images taken for the same scene to be photographed, and the definition of the first image is lower than that of the second image.

Memory, which stores computer programs that run on the processor.

A processor, configured to execute the processing steps in the above-mentioned image processing method.

Optionally, the camera module includes a wide-angle camera, a main camera and a telephoto camera; the wide-angle camera is used to obtain the first image after the processor obtains the camera instruction; the main camera is used to obtain the first image after the processor obtains the camera instruction , to acquire the second image; or, the main camera is used to acquire the first image after the processor acquires the photographing instruction; the telephoto camera is configured to acquire the second image after the processor acquires the photographing instruction.

Strictly speaking, the image is obtained by the image processor in the color camera and the black and white camera. Wherein, the image sensor may be, for example, a charge-coupled device (charge-coupled device, CCD), a complementary metal oxide semiconductor (complementary metal oxide semiconductor, CMOS) and the like.

The embodiment of the present application also provides a computer-readable storage medium, where computer instructions are stored in the computer-readable storage medium; when the computer-readable storage medium is run on an image processing device, the image processing device executes the following steps: The method shown in Figure 3 and/or Figure 4. The computer instructions may be stored in or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website, computer, server, or data center Transmission to another website site, computer, server or data center by wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.). The computer-readable storage medium may be any available medium that can be accessed by a computer, or may be a data storage device including one or more servers, data centers, etc. that can be integrated with the medium. The available medium may be a magnetic medium (for example, a floppy disk, a hard disk, or a magnetic tape), an optical medium, or a semiconductor medium (for example, a solid state disk (solid state disk, SSD)) and the like.

The embodiment of the present application also provides a computer program product including computer instructions, which, when run on an image processing device, enables the image processing device to execute the method shown in FIG. 3 and/or FIG. 4 .

FIG. 12 is a schematic structural diagram of a chip provided by an embodiment of the present application. The chip shown in FIG. 12 may be a general-purpose processor or a special-purpose processor. The chip includes a processor 401 . Wherein, the processor 401 is configured to support the image processing apparatus to execute the technical solutions shown in FIG. 3 and/or FIG. 4 .

Optionally, the chip further includes a transceiver 402, and the transceiver 402 is configured to be controlled by the processor 401, and configured to support the communication device to execute the technical solution shown in FIG. 3 and/or FIG. 4 .

Optionally, the chip shown in FIG. 12 may further include: a storage medium 403 .

It should be noted that the chip shown in Figure 12 can be implemented using the following circuits or devices: one or more field programmable gate arrays (field programmable gate array, FPGA), programmable logic device (programmable logic device, PLD) , controllers, state machines, gate logic, discrete hardware components, any other suitable circuitry, or any combination of circuitry capable of performing the various functions described throughout this application.

The electronic equipment, image processing device, computer storage medium, computer program product, and chip provided by the above-mentioned embodiments of the present application are all used to execute the method provided above. Therefore, the beneficial effects that it can achieve can refer to the above-mentioned The beneficial effects corresponding to the method will not be repeated here.

It should be understood that the foregoing is only to help those skilled in the art better understand the embodiments of the present application, rather than limiting the scope of the embodiments of the present application. Those skilled in the art can obviously make various equivalent modifications or changes based on the above examples given, for example, some steps in the various embodiments of the above detection method may be unnecessary, or some steps may be newly added wait. Or a combination of any two or more of the above-mentioned embodiments. Such modifications, changes or combined solutions also fall within the scope of the embodiments of the present application.

It should also be understood that the above description of the embodiments of the present application focuses on emphasizing the differences between the various embodiments, and the same or similar points that are not mentioned can be referred to each other, and for the sake of brevity, details are not repeated here.

It should also be understood that the sequence numbers of the above processes do not mean the order of execution, and the execution order of each process should be determined by its functions and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

It should also be understood that in this embodiment of the present application, "presetting" and "predefining" can be realized by pre-saving corresponding codes, tables or other methods that can be used to indicate related information in devices (for example, including electronic devices) , the present application does not limit its specific implementation.

It should also be understood that the division of methods, situations, categories and embodiments in the embodiments of the present application is only for the convenience of description, and should not constitute a special limitation, and the features in various methods, categories, situations and embodiments are not contradictory cases can be combined.

It should also be understood that in each embodiment of the present application, if there is no special explanation and logical conflict, the terms and/or descriptions between different embodiments are consistent and can be referred to each other, and the technical features in different embodiments New embodiments can be formed by combining them according to their inherent logical relationships.

Finally, it should be noted that: the above is only a specific implementation of the application, but the scope of protection of the application is not limited thereto, and any changes or replacements within the technical scope disclosed in the application shall be covered by this application. within the scope of the application. Therefore, the protection scope of the present application should be determined by the protection scope of the claims.

Claims (15)

An image processing method, characterized in that it is applied to an electronic device including a first camera and a second camera, the method comprising: the electronic device activates a camera; displaying a preview interface, where the preview interface includes a first control; detecting a first operation on the first control; In response to the first operation, the first camera captures a first image and the second camera captures a second image, the resolution of the first image is lower than that of the second image, and the second image An image includes a first area, where the first area is an area in the first image whose resolution is less than a preset threshold; Obtaining a mask block according to the first image, the mask block corresponding to the first region; fusing the first image and the second image to obtain a first fused image; According to the mask block, determining a first image block in the first image and determining a second image block in the second image, the first image block corresponds to the mask block, and the first image block corresponds to the mask block, and the second image block in the second image is determined. The second image block corresponds to the mask block; fusing the first image block and the second image block to obtain a fused image block; The first fused image is fused with the fused image block to obtain a third image. The image processing method according to claim 1, wherein obtaining a mask block according to the first image comprises: The first image is input into a segmentation model for segmentation, and the mask block is generated; the segmentation model is used to segment the first region in the first image, and generate a block corresponding to the first region corresponding to the mask block. The image processing method according to claim 1 or 2, wherein merging the first image and the second image to obtain a first fused image comprises: The first image and the second image are fused using a first fusion model to obtain the first fused image. The image processing method according to any one of claims 1-3, wherein merging the first image block and the second image block to obtain a fused image block includes: The first image block and the second image block are fused using a second fusion model to obtain the fused image block. The image processing method according to any one of claims 1-4, wherein merging the first fused image and the fused image block to obtain a third image comprises: The first fusion image and the fusion image block are fused using a third fusion model to obtain the third image. The image processing method according to any one of claims 1-5, wherein the method further comprises: When the mask block is not obtained according to the first image, the first image and the second image are fused using the first fusion model to obtain the first fused image. The image processing method according to claim 3 or 6, wherein the method further comprises: The first image and the second image are registered. The image processing method according to claim 4, further comprising: registering the first image block and the second image block. The image processing method according to claim 7 or 8, wherein the registration includes: global registration and/or local registration, and the global registration is used to indicate that all contents in multiple images are Registration, the local registration refers to registering local contents in multiple images. The image processing method according to claim 4, wherein the method further comprises: Using a training image set and adding random highlight noise to train the first fusion model to obtain the second fusion model, wherein the training image set includes an original image, and the original image is marked with a mask block. The image processing method according to claim 5, wherein the third fusion model is a Laplace fusion model. An electronic device, characterized in that it includes a camera module, a processor and a memory; The camera module is configured to collect a first image and a second image, the definition of the first image is lower than that of the second image, the first image includes a first area, and the first The area is an area in the first image whose resolution is less than a preset threshold; said memory for storing a computer program executable on said processor; The processor is configured to execute the processing steps in the image processing method according to any one of claims 1 to 11. The electronic device according to claim 12, wherein the camera module includes a wide-angle camera, a main camera and a telephoto camera; The wide-angle camera is used to acquire the first image after the processor acquires a photographing instruction; The main camera is configured to acquire the second image after the processor acquires the photographing instruction, or; The main camera is configured to acquire the first image after the processor acquires the photographing instruction; The telephoto camera is configured to acquire the second image after the processor acquires the photographing instruction. A chip, characterized by comprising: a processor for calling and running a computer program from a memory, so that a device installed with the chip executes the image processing method according to any one of claims 1 to 11. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program, the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes the following steps: The image processing method according to any one of claims 1 to 11.

Images