CN114125318B - Image sensor, camera module, electronic device, image generation method and device - Google Patents

Abstract

The present application relates to an image sensor and an image generation method. The image sensor 21 includes a filter array 23 and a pixel array 24. The filter array 23 includes a minimum repeating unit 230. The minimum repeating unit 230 includes multiple filter groups. Each filter group includes a full-color filter 233 and a color filter 234. The amount of light transmitted by the full-color filter 233 is greater than the amount of light transmitted by the color filter 234. The color filter 234 and the full-color filter 233 each include four sub-filters. The multiple filter groups include at least a first filter group 231. At least one color filter 234 in the first filter group 231 includes a first color sub-filter 2341 and a second color sub-filter 2342. The pixel array 24 includes multiple pixels. The pixels of the pixel array 24 are arranged corresponding to the sub-filters of the filter array 23. The pixel array 24 is configured to receive light passing through the filter array 23 to generate an electrical signal. The images generated by this image sensor have higher clarity and better quality.

Description

Image sensor, camera module, electronic device, image generation method and device

Technical Field

The present application relates to the field of imaging technology, and in particular to an image sensor, a camera module, an electronic device, an image generating method, an apparatus, a computer-readable storage medium, and a computer program product.

Background technique

With the development of computer technology, most electronic devices such as mobile phones are equipped with cameras to realize the photo function through the camera. The camera is provided with an image sensor, through which a color image is collected. In order to realize the collection of color images, a filter array arranged in the form of a Bayer array is usually provided in the image sensor, so that multiple pixels in the image sensor can receive light passing through the corresponding filter, thereby generating pixel signals with different color channels, and then generating an image.

However, conventional image sensors produce images with low definition.

Summary of the invention

The embodiments of the present application provide an image sensor, an image generating method, an apparatus, an electronic device, and a computer-readable storage medium, which can improve the clarity of imaging.

An image sensor, the image sensor comprising a filter array and a pixel array, the filter array comprising a minimum repeating unit, the minimum repeating unit comprising a plurality of filter groups, each filter group comprising a color filter and a full-color filter, the amount of light transmitted by the full-color filter being greater than the amount of light transmitted by the color filter, the color filter and the full-color filter each comprising four sub-filters, the plurality of filter groups comprising at least a first filter group, at least one of the color filters in the first filter group comprising a first color sub-filter and a second color sub-filter; the pixel array comprising a plurality of pixels, the pixels of the pixel array being arranged corresponding to the sub-filters of the filter array, the pixel array being configured to receive light passing through the filter array to generate an electrical signal.

A camera module, the camera module comprising a lens and an image sensor; the image sensor is used to receive light passing through the lens, and the pixel generates an electrical signal according to the light; the image sensor comprises a filter array and a pixel array, the filter array comprises a minimum repeating unit, the minimum repeating unit comprises a plurality of filter groups, each filter group comprises a color filter and a full-color filter, the amount of light transmitted by the full-color filter is greater than the amount of light transmitted by the color filter, the color filter and the full-color filter each comprise four sub-filters, the plurality of filter groups comprise at least a first filter group, at least one of the color filters in the first filter group comprises a first color sub-filter and a second color sub-filter; the pixel array comprises a plurality of pixels, the pixels of the pixel array are arranged corresponding to the sub-filters of the filter array, and the pixel array is configured to receive light passing through the filter array to generate an electrical signal.

An electronic device, comprising a camera module and a housing, wherein the camera module is arranged on the housing; the camera module comprises a lens and an image sensor; the image sensor is used to receive light passing through the lens, and the pixel generates an electrical signal according to the light; the image sensor comprises a filter array and a pixel array, the filter array comprises a minimum repeating unit, the minimum repeating unit comprises a plurality of filter groups, each filter group comprises a color filter and a full-color filter, the amount of light transmitted by the full-color filter is greater than the amount of light transmitted by the color filter, the color filter and the full-color filter each comprise four sub-filters, the plurality of filter groups comprise at least a first filter group, at least one of the color filters in the first filter group comprises a first color sub-filter and a second color sub-filter; the pixel array comprises a plurality of pixels, the pixels of the pixel array are arranged corresponding to the sub-filters of the filter array, and the pixel array is configured to receive light passing through the filter array to generate an electrical signal.

The above-mentioned image sensor includes a filter array and a pixel array. The filter array includes a minimum repeating unit. The minimum repeating unit includes multiple filter groups. The filter group includes a full-color filter and a color filter. The amount of light transmitted through the full-color filter is greater than the amount of light transmitted through the color filter. More light can be obtained during shooting, so there is no need to adjust the shooting parameters. The clarity of the image in the dark is improved without affecting the stability of the shooting. When imaging in the dark, both stability and clarity can be taken into account. The stability and clarity of the image in the dark are both high. Furthermore, each full-color filter includes 4 sub-filters, each color filter includes 4 sub-filters, the multiple filter groups include at least a first filter group, at least one color filter in the first filter group includes a first color sub-filter and a second color sub-filter, the pixel array includes multiple full-color pixels and multiple color pixels, each full-color pixel corresponds to a sub-filter of the full-color filter, each first color photosensitive pixel corresponds to a first color sub-filter of the color filter, and each second color photosensitive pixel corresponds to a second color sub-filter of the color filter. Different color photosensitive pixels can be mixed and arranged in the same color filter, thereby effectively improving the resolution of the color channel and reducing the problem of false color in the generated image.

An image generation method is applied to an image sensor, wherein the image sensor comprises a filter array and a pixel array, the filter array comprises a minimum repeating unit, the minimum repeating unit comprises a plurality of filter groups, each filter group comprises a color filter and a full-color filter, the amount of light transmitted by the full-color filter is greater than the amount of light transmitted by the color filter, the color filter and the full-color filter each comprise four sub-filters, the plurality of filter groups comprise a first filter group and a second filter group, at least one of the color filters in the first filter group comprises a first sub-filter, a color sub-filter and a second color sub-filter; the color filter in the second filter group includes at least a third color sub-filter; the pixel array includes a plurality of pixels, the first color photosensitive pixel of the pixel array is arranged correspondingly to the first color sub-filter of the filter array, the second color photosensitive pixel of the pixel array is arranged correspondingly to the second color sub-filter of the filter array, the third color photosensitive pixel of the pixel array is arranged correspondingly to the third color sub-filter of the filter array, and the pixel array is configured to receive light passing through the filter array to generate an electrical signal;

The method comprises:

In the first resolution mode, a first merged image is obtained according to a first pixel value obtained by merging and reading a plurality of panchromatic pixels corresponding to the panchromatic filter in the filter group, a second pixel value obtained by merging and reading a plurality of first color photosensitive pixels corresponding to the color filter, and a third pixel value obtained by merging and reading a plurality of third color photosensitive pixels corresponding to the color filter;

Obtain a second merged image by merging and reading out fourth pixel values of a plurality of second color photosensitive pixels corresponding to the color filter;

A first target image is obtained based on the first merged image and the second merged image.

An image generating device is applied to an image sensor, the image sensor comprising a filter array and a pixel array, the filter array comprising a minimum repeating unit, the minimum repeating unit comprising a plurality of filter groups, each of the filter groups comprising a color filter and a full-color filter, the amount of light transmitted by the full-color filter is greater than the amount of light transmitted by the color filter, the color filter and the full-color filter each comprising four sub-filters, the plurality of filter groups comprising a first filter group and a second filter group, at least one of the color filters in the first filter group comprising a first sub-filter; a color sub-filter and a second color sub-filter; the color filter in the second filter group includes at least a third color sub-filter; the pixel array includes a plurality of pixels, the first color photosensitive pixel of the pixel array is arranged correspondingly to the first color sub-filter of the filter array, the second color photosensitive pixel of the pixel array is arranged correspondingly to the second color sub-filter of the filter array, the third color photosensitive pixel of the pixel array is arranged correspondingly to the third color sub-filter of the filter array, and the pixel array is configured to receive light passing through the filter array to generate an electrical signal;

The device comprises:

A first merging module is used to obtain a first merged image in a first resolution mode according to a first pixel value read out by merging a plurality of panchromatic pixels corresponding to the panchromatic filter in the filter group, a second pixel value read out by merging a plurality of first color photosensitive pixels corresponding to the color filter, and a third pixel value read out by merging a plurality of third color photosensitive pixels corresponding to the color filter;

A second merging module, configured to obtain a second merged image by merging the fourth pixel values read out according to the plurality of second color photosensitive pixels corresponding to the color filter;

An image generation module is used to obtain a first target image based on the first merged image and the second merged image.

An electronic device includes a memory, a processor, and an image sensor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the following steps:

In the first resolution mode, a first merged image is obtained based on a first pixel value obtained by merging and reading out a plurality of panchromatic pixels corresponding to the panchromatic filter in the filter group, a second pixel value obtained by merging and reading out a plurality of first color photosensitive pixels corresponding to the color filter, and a third pixel value obtained by merging and reading out a plurality of third color photosensitive pixels corresponding to the color filter; a second merged image is obtained based on a fourth pixel value obtained by merging and reading out a plurality of second color photosensitive pixels corresponding to the color filter; and a first target image is obtained based on the first merged image and the second merged image.

A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

In the first resolution mode, a first merged image is obtained based on a first pixel value obtained by merging and reading out a plurality of panchromatic pixels corresponding to the panchromatic filter in the filter group, a second pixel value obtained by merging and reading out a plurality of first color photosensitive pixels corresponding to the color filter, and a third pixel value obtained by merging and reading out a plurality of third color photosensitive pixels corresponding to the color filter; a second merged image is obtained based on a fourth pixel value obtained by merging and reading out a plurality of second color photosensitive pixels corresponding to the color filter; and a first target image is obtained based on the first merged image and the second merged image.

A computer program product, comprising a computer program, which, when executed by a processor, implements the following steps:

In the first resolution mode, a first merged image is obtained based on a first pixel value obtained by merging and reading out a plurality of panchromatic pixels corresponding to the panchromatic filter in the filter group, a second pixel value obtained by merging and reading out a plurality of first color photosensitive pixels corresponding to the color filter, and a third pixel value obtained by merging and reading out a plurality of third color photosensitive pixels corresponding to the color filter; a second merged image is obtained based on a fourth pixel value obtained by merging and reading out a plurality of second color photosensitive pixels corresponding to the color filter; and a first target image is obtained based on the first merged image and the second merged image.

The above-mentioned image generation method, device, electronic device, computer-readable storage medium and computer program product, in the first resolution mode, can fuse the full-color channel information into the image by merging the first pixel value read out by merging the multiple full-color pixels corresponding to the full-color filter in the filter group, the second pixel value read out by merging the multiple first-color photosensitive pixels corresponding to the color filter, and the third pixel value read out by merging the multiple third-color photosensitive pixels corresponding to the color filter, so as to increase the overall light input, so as to generate a first merged image with more information and clearer detail resolution. According to the fourth pixel value read out by merging the multiple second-color photosensitive pixels corresponding to the color filter, the second-color photosensitive pixels can be separated to form a second merged image, so that the positions of the second-color photosensitive pixels in the second merged image and the first-color photosensitive pixels in the first merged image are consistent. Based on the first merged image and the second merged image, the first target image is obtained, and the first-color photosensitive pixels in the first merged image and the second-color photosensitive pixels in the second merged image can be mixed and arranged, so that the color of the generated first target image is clearer. And the pixel merging readout reduces the size of the generated first target image and the power consumption required to generate the image is low.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

FIG1 is a schematic diagram of the structure of an electronic device in one embodiment;

FIG2 is an exploded schematic diagram of an image sensor in one embodiment;

FIG3 is a schematic diagram showing the connection between a pixel array and a readout circuit in one embodiment;

FIG4a is a schematic diagram of a first diagonal direction and a second diagonal direction in one embodiment;

FIG4 b is a schematic diagram of a third diagonal direction and a fourth diagonal direction in one embodiment;

FIG5 is a schematic diagram of the arrangement of the minimum repeating unit in one embodiment;

FIG6 is a schematic diagram of the arrangement of the minimum repeating unit in one embodiment;

FIG7 is a schematic diagram of the arrangement of the minimum repeating unit in another embodiment;

FIG8 is a schematic diagram of the arrangement of the minimum repeating unit in another embodiment;

FIG9 is a flow chart of an image generation method in one embodiment;

FIG10 is a schematic diagram of obtaining a first target image in a first resolution mode in one embodiment;

FIG11 is a schematic diagram of obtaining a first target image in a first resolution mode in another embodiment;

FIG12 is a schematic diagram of obtaining a second target image in a second resolution mode in one embodiment;

FIG13 is a schematic diagram of obtaining a second target image in a second resolution mode in another embodiment;

FIG14 is a schematic diagram of an image generating method in a second resolution mode according to another embodiment;

FIG15 is a block diagram of a structure of an image generating device in one embodiment;

FIG. 16 is a block diagram of the internal structure of an electronic device in one embodiment.

Detailed ways

In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

It is understood that the terms "first", "second", etc. used in this application may be used herein to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish a first element from another element. For example, without departing from the scope of this application, a first target image may be referred to as a second target image, and similarly, a second target image may be referred to as a first target image. Both the first and the second are target images, but they are not the same target image. A plurality refers to at least two, such as 2, 4, but is not limited thereto.

In one embodiment, the electronic device 100 includes a mobile phone, a tablet computer, a laptop computer, a teller machine, a gate machine, a smart watch, a head-mounted display device, etc. It can be understood that the electronic device 100 can also be a device with any other image processing function. The electronic device 100 includes a camera module 20, a processor 30 and a housing 40. The camera module 20 and the processor 30 are both arranged in the housing 40, and the housing 40 can also be used to install functional modules such as a power supply device and a communication device of the electronic device 100, so that the housing 40 provides dustproof, drop-proof, waterproof and other protection for the functional modules.

The camera module 20 can be a front camera module, a rear camera module, a side camera module, an under-screen camera module, etc., and is not limited here. The camera module 20 includes a lens and an image sensor 21. When the camera module 20 captures an image, light passes through the lens and reaches the image sensor 21. The image sensor 21 is used to convert the light signal irradiated on the image sensor 21 into an electrical signal.

In one embodiment, as shown in FIG. 2 , the image sensor 21 includes a microlens array 22 , a filter array 23 , and a pixel array 24 .

The microlens array 22 includes a plurality of microlenses 221. The microlenses 221, the sub-filters in the filter array 23, and the pixels in the pixel array 24 are arranged in a one-to-one correspondence. The microlenses 221 are used to gather the incident light. The gathered light passes through the corresponding sub-filter and is then projected onto the pixel and received by the corresponding pixel. The pixel then converts the received light into an electrical signal.

The filter array 23 includes a plurality of minimal repeating units 230. The minimal repeating unit 230 may include a plurality of filter groups. Each filter group includes a full-color filter 233 and a color filter 234, and the amount of light transmitted through the full-color filter 233 is greater than the amount of light transmitted through the color filter 234. Each full-color filter 233 includes four sub-filters 2331, and each color filter 234 includes four sub-filters. The plurality of filter groups include at least a first filter group 231, and at least one color filter 234 in the first filter group 231 includes a first color sub-filter 2341 and a second color sub-filter 2342. Different color filters 234 are also included in different filter groups.

In one embodiment, the plurality of filter groups further include a second filter group 232 , and the color filter 234 in the second filter group 232 includes at least a third color sub-filter 2343 .

The colors corresponding to the wavelength band of the light transmitted by the color filter 234 of the filter group in the minimum repeating unit 230 include color a, color b and/or color c. The colors corresponding to the wavelength band of the light transmitted by the color filter 234 of the filter group 232 include color a, color b and color c, or color a, color b or color c, or color a and color b, or color b and color c, or color a and color c. Among them, color a is red, color b is green, and color c is blue, or for example, color a is magenta, color b is cyan, and color c is yellow, etc., which is not limited here.

In one embodiment, the width of the wavelength band of the light transmitted by the color filter 234 is smaller than the width of the wavelength band of the light transmitted by the full-color filter 233. For example, the wavelength band of the light transmitted by the color filter 234 may correspond to the wavelength band of red light, the wavelength band of green light, or the wavelength band of blue light, and the wavelength band of the light transmitted by the full-color filter 233 is the wavelength band of all visible light. In other words, the color filter 234 only allows light of a specific color to be transmitted, while the full-color filter 233 can pass light of all colors. Of course, the wavelength band of the light transmitted by the color filter 234 may also correspond to the wavelength band of other colored light, such as magenta light, purple light, cyan light, yellow light, etc., which is not limited here.

In one embodiment, the ratio of the number of color filters 234 to the number of panchromatic filters 233 in the filter group can be 1:3, 1:1 or 3:1. For example, if the ratio of the number of color filters 234 to the number of panchromatic filters 233 is 1:3, then the number of color filters 234 is 1 and the number of panchromatic filters 233 is 3. In this case, the number of panchromatic filters 233 is large and the imaging quality in dim light is better; or, if the ratio of the number of color filters 234 to the number of panchromatic filters 233 is 1:1, then the number of color filters 234 is 2 and the number of panchromatic filters 233 is 2. In this case, both good color performance and good imaging quality in dim light can be obtained; or, if the ratio of the number of color filters 234 to the number of panchromatic filters 233 is 3:1, then the number of color filters 234 is 3 and the number of panchromatic filters 233 is 1. In this case, better color performance can be obtained and the imaging quality in dim light can be improved.

The pixel array 24 includes a plurality of pixels, and the pixels of the pixel array 24 are arranged corresponding to the sub-filters of the filter array 23. The pixel array 24 is configured to receive light passing through the filter array 23 to generate an electrical signal.

The pixel array 24 is configured to receive the light passing through the filter array 23 to generate an electrical signal, which means that the pixel array 24 is used to perform photoelectric conversion on the light of a given set of scenes of the photographed object passing through the filter array 23 to generate an electrical signal. The light of a given set of scenes of the photographed object is used to generate image data. For example, if the photographed object is a building, the given set of scenes of the photographed object refers to the scene where the building is located, and the scene may also contain other objects.

In one embodiment, the pixel array 24 includes a plurality of minimal repeating units 240, the minimal repeating unit 240 includes a plurality of pixel groups, the plurality of pixel groups include a panchromatic pixel group 243 and a color pixel group 244. Each panchromatic pixel group 243 includes four panchromatic pixels 2431, each color pixel group 244 includes four color pixels, the plurality of pixel groups include at least a first pixel group 241, at least one color pixel group 244 in the first pixel group 241 includes a first color photosensitive pixel 2441 and a second color photosensitive pixel 2442. Each panchromatic pixel 2431 corresponds to a sub-filter 2331 in the panchromatic filter 233, and the panchromatic pixel 2431 receives light passing through the corresponding sub-filter 2331 to generate an electrical signal. Each first color photosensitive pixel 2441 corresponds to a first color sub-filter 2341 of the color filter 234, and the first color photosensitive pixel 2441 receives light passing through the corresponding first color sub-filter 2341 to generate an electrical signal. Each second color photosensitive pixel 2442 corresponds to a second color sub-filter 2342 of the color filter 234 , and the second color photosensitive pixel 2442 receives light passing through the corresponding second color sub-filter 2342 to generate an electrical signal.

In one embodiment, the plurality of pixel groups further include a second pixel group 242, and the color pixel group 244 in the second pixel group 242 includes at least a third color photosensitive pixel 2443. Each third color photosensitive pixel 2443 corresponds to a third color sub-filter 2343 of the color filter 234, and the third color photosensitive pixel 2443 receives light passing through the corresponding third color sub-filter 2343 to generate an electrical signal.

The image sensor 21 in this embodiment includes a filter array 23 and a pixel array 24. The filter array 23 includes a minimum repeating unit 230. The minimum repeating unit 230 includes a plurality of filter groups. Each filter group includes a full-color filter 233 and a color filter 234. The color filter 234 has a narrower spectral response than the full-color filter 233. The amount of light transmitted by the full-color filter 233 is greater than the amount of light transmitted by the color filter 234. More light can be obtained during shooting, so there is no need to adjust the shooting parameters. The clarity of the image under low light is improved without affecting the stability of the shooting. When imaging under low light, both stability and clarity can be taken into account. The stability and clarity of the image under low light are both high. Furthermore, each full-color filter 233 includes four sub-filters 2331, each color filter 234 includes four sub-filters, the multiple filter groups include at least a first filter group 231, at least one color filter 234 in the first filter group 231 includes a first color sub-filter 2341 and a second color sub-filter 2342, the pixel array 24 includes a plurality of full-color pixels 2431 and a plurality of color pixels 2441, each full-color pixel 2431 corresponds to a sub-filter 2331 of the full-color filter 233, each first color photosensitive pixel 2441 corresponds to a first color sub-filter 2341 of the color filter 234, and each second color photosensitive pixel 2442 corresponds to a second color sub-filter 2342 of the color filter 234, so that different color photosensitive pixels can be mixed and arranged in the same color filter, thereby effectively improving the resolution of the color channel and reducing the problem of false color in the generated image.

In one embodiment, as shown in FIG. 2 , the minimum repeating unit 230 in the filter array 23 includes four filter groups, and the four filter groups include two first filter groups 231 and two second filter groups 232 , and the two first filter groups 231 and the two second filter groups 232 are arranged in a matrix.

Each first filter group 231 includes a full-color filter 233 and a color filter 234, each full-color filter 233 and each color filter 234 has 4 sub-filters, and at least one color filter 234 includes a first color sub-filter 2341 and a second color sub-filter 2342, then the second filter group 232 includes a total of 16 sub-filters.

Each second filter group 232 includes a full-color filter 233 and a color filter 234. Each full-color filter 233 and each color filter 234 has 4 sub-filters, and the color filter 234 includes at least a third color sub-filter 2343. The second filter group 232 includes a total of 16 sub-filters.

Similarly, the pixel array 24 includes a plurality of minimum repeating units 240, which correspond one to one with the plurality of minimum repeating units 230 of the filter array 23. Each minimum repeating unit 240 includes four pixel groups, and the four pixel groups include two first pixel groups 241 and two second pixel groups 242, and the two first pixel groups 241 and the two second pixel groups 242 are arranged in a matrix. Each first pixel group 241 corresponds to a first filter group 231, and each second pixel group 242 corresponds to a second filter group 232.

As shown in FIG3 , the readout circuit 25 is electrically connected to the pixel array 24 for controlling the exposure of the pixel array 24 and the reading and output of the pixel values of the pixel points. The readout circuit 25 includes a vertical drive unit 251, a control unit 252, a column processing unit 253, and a horizontal drive unit 254. The vertical drive unit 251 includes a shift register and an address decoder. The vertical drive unit 251 includes a readout scan and a reset scan function. The control unit 252 configures the timing signal according to the operation mode, and uses a variety of timing signals to control the vertical drive unit 251, the column processing unit 253, and the horizontal drive unit 254 to work together. The column processing unit 253 may have an analog-to-digital (A/D) conversion function for converting an analog pixel signal into a digital format. The horizontal drive unit 254 includes a shift register and an address decoder. The horizontal drive unit 254 sequentially scans the pixel array 24 column by column.

In one embodiment, as shown in FIG4a , the first filter group 231 in the minimum repeating unit 230 is arranged in the direction of the first diagonal line D1, and the second filter group 232 is arranged in the direction of the second diagonal line D2, and the direction of the first diagonal line D1 is different from the direction of the second diagonal line D2. The direction of the first diagonal line D1 is different from the direction of the second diagonal line D2, which can take into account both color performance and low-light imaging quality.

The direction of the first diagonal line D1 is different from the direction of the second diagonal line D2. Specifically, the direction of the first diagonal line D1 is not parallel to the direction of the second diagonal line D2, or the direction of the first diagonal line D1 is perpendicular to the direction of the second diagonal line D2.

In one embodiment, as shown in FIG4b, each filter group includes a color filter 234 and a full-color filter 233, and the color filter 234 and the full-color filter 233 of each filter group are arranged in a matrix. Each full-color filter 233 in the filter group is arranged in the third diagonal direction D3, and each color filter 234 in the filter group is arranged in the fourth diagonal direction D4. The third diagonal direction D3 and the fourth diagonal direction D4 are different, which can take into account both color performance and low-light imaging quality.

The direction of the third diagonal line D3 is different from the direction of the fourth diagonal line D4. Specifically, the direction of the third diagonal line D3 is not parallel to the direction of the fourth diagonal line D4, or the direction of the third diagonal line D3 is perpendicular to the direction of the fourth diagonal line D4.

In other embodiments, one color filter 234 and one panchromatic filter 233 may be located on the third diagonal line D3 , and another color filter 234 and another panchromatic filter 233 may be located on the fourth diagonal line D4 .

In one embodiment, the color filter 234 may be disposed in a direction parallel to the third diagonal line D3 , or the color filter 234 may be disposed in a direction parallel to the fourth diagonal line D4 .

In one embodiment, at least one color filter 234 in the first filter group 231 includes a first color sub-filter 2341 and a second color sub-filter 2342. The first color sub-filter 2341 is arranged in one direction of the fifth diagonal direction or the sixth diagonal direction, and the second color sub-filter 2342 is arranged in the other direction of the fifth diagonal direction or the sixth diagonal direction. The sub-filter 2331 of the panchromatic filter 233 in the first filter group 231 is arranged in the fifth diagonal direction and the sixth diagonal direction.

In one embodiment, the minimum repeating unit 230 includes two first filter groups 231 and two second filter groups 232 , and the two first filter groups 231 and the two second filter groups 232 are arranged in a matrix.

Among the two first filter groups 231, the first color sub-filter 2341 in the color filter 234 of one of the first filter groups 231 is arranged in the fifth diagonal direction, and the second color sub-filter 2342 is arranged in the sixth diagonal direction; and, the first color sub-filter 2341 in the color filter 234 of the other first filter group 231 is arranged in the sixth diagonal direction, and the second color sub-filter 2342 is arranged in the fifth diagonal direction.

It can be understood that the first color sub-filter 2341 is set in the fifth diagonal direction. The first color sub-filter 2341 can be on the fifth diagonal line or in a direction parallel to the fifth diagonal line, and other diagonal directions are similar.

In one embodiment, as shown in FIG5 , the plurality of filter groups include a first filter group 231 and a second filter group 232, and at least one color filter 234 in the first filter group 231 includes a first color sub-filter 2341 and a second color sub-filter 2342. The color filter 234 in the second filter group 232 includes at least a third color sub-filter 2343. The third color sub-filter 2343 in the second filter group 232 is arranged in the seventh diagonal line D7 direction and the eighth diagonal line D8 direction, and the seventh diagonal line D7 direction is different from the eighth diagonal line D8 direction.

The direction of the seventh diagonal line D7 is different from the direction of the eighth diagonal line D8. Specifically, the direction of the seventh diagonal line D7 is not parallel to the direction of the eighth diagonal line D8, or the direction of the seventh diagonal line D7 is perpendicular to the direction of the eighth diagonal line D8.

In one embodiment, as shown in FIG5 , the minimum repeating unit 230 in the filter array 23 includes four filter groups, and the four filter groups are arranged in a matrix. The four filter groups include a first filter group 231 and a second filter group 232. Each first filter group 231 includes two full-color filters 233 and two color filters 234, and each second filter group 232 includes two full-color filters 233 and two color filters 234. The full-color filter 233 includes four sub-filters 2331, the same color filter 234 of the first filter group 231 includes two first color sub-filters 2341 and two second color sub-filters 2342, and the same color filter 234 of the second filter group 232 includes four third color sub-filters 2343. The minimum repeating unit is 8 rows and 8 columns of 64 sub-filters, and the arrangement is as follows:

Wherein, w represents the full-color sub-filter 2331 , and a, b and c all represent color sub-filters. The color sub-filters include a first color sub-filter 2341 , a second color sub-filter 2342 and a third color sub-filter 2343 .

a, b and c respectively represent one of the first color sub-filter 2341, the second color sub-filter 2342 and the third color sub-filter 2343. The full-color sub-filter 2331 refers to a sub-filter that can filter out all light outside the visible light band, and the first color sub-filter 2341, the second color sub-filter 2342 and the third color sub-filter 2343 can be a red sub-filter, a green sub-filter, a blue sub-filter, a magenta sub-filter, a cyan sub-filter and a yellow sub-filter. The red sub-filter is a sub-filter that filters out all light except red light, the green sub-filter is a sub-filter that filters out all light except green light, the blue sub-filter is a sub-filter that filters out all light except blue light, the magenta sub-filter is a sub-filter that filters out all light except magenta light, the cyan sub-filter is a sub-filter that filters out all light except cyan light, and the yellow sub-filter is a sub-filter that filters out all light except yellow light.

a can be a red sub-filter, a green sub-filter, a blue sub-filter, a magenta sub-filter, a cyan sub-filter or a yellow sub-filter, b can be a red sub-filter, a green sub-filter, a blue sub-filter, a magenta sub-filter, a cyan sub-filter or a yellow sub-filter, and c can be a red sub-filter, a green sub-filter, a blue sub-filter, a magenta sub-filter, a cyan sub-filter or a yellow sub-filter. For example, b is a red sub-filter, a is a green sub-filter, and c is a blue sub-filter; or, c is a red sub-filter, a is a green sub-filter, and b is a blue sub-filter; for another example, c is a red sub-filter, a is a green sub-filter, and b is a blue sub-filter; or, a is a red sub-filter, b is a blue sub-filter, and c is a green sub-filter, etc., which are not limited here; for another example, b is a magenta sub-filter, a is a cyan sub-filter, and b is a yellow sub-filter, etc. In other embodiments, the color filter may further include sub-filters of other colors, such as an orange sub-filter, a purple sub-filter, etc., which are not limited here.

In one embodiment, as shown in FIG6 , the minimum repeating unit 230 in the filter array 23 includes four filter groups, and the four filter groups are arranged in a matrix. The four filter groups include a first filter group 231 and a second filter group 232. Each first filter group 231 includes two full-color filters 233 and two color filters 234, and each second filter group 232 includes two full-color filters 233 and two color filters 234. The full-color filter 233 includes four sub-filters 2331, the same color filter 234 of the first filter group 231 includes two first color sub-filters 2341 and two second color sub-filters 2342, and the same color filter 234 of the second filter group 232 includes four third color sub-filters 2343. The minimum repeating unit is 8 rows and 8 columns of 64 sub-filters, and the arrangement is as follows:

Wherein, w represents the full-color sub-filter 2331 , and a, b and c all represent color sub-filters. The color sub-filters include a first color sub-filter 2341 , a second color sub-filter 2342 and a third color sub-filter 2343 .

In this embodiment, photosensitive pixels of different colors are mixed and arranged, such as red pixels and blue pixels are mixed and arranged, so that each row and column of the minimum repeating unit has RGB pixels, which can effectively improve the resolution of the color channel and reduce the problem of false colors in the generated image. In addition, full-color pixels are introduced to effectively increase the amount of light entering, thereby improving the clarity of the image. At the same time, the image sensor in this embodiment also has the advantage of secondary pixel merging reading. Each row and column of the minimum repeating unit of the image sensor has RGB pixels, and there is no need to merge the same pixels across different minimum repeating units, which can effectively reduce the power consumption of processing.

In one embodiment, as shown in FIG7 , the plurality of filter groups include a first filter group 231 and a second filter group 232, and at least one color filter 234 in the first filter group 231 includes a first color sub-filter 2341 and a second color sub-filter 2342. The color filter 234 in the second filter group 232 includes a third color sub-filter 2343, and the color filter 234 in the second filter group 232 also includes the first color sub-filter 2341 or the second color sub-filter 2342.

Specifically, the same color filter 234 in the second filter group 232 includes a third color sub-filter 2343 and a first color sub-filter 2341 , or the same color filter 234 in the second filter group 232 includes a third color sub-filter 2343 and a second color sub-filter 2342 .

In one embodiment, as shown in FIG. 7 , the second filter group 232 includes two color filters 234 , and the sub-filters of the same color in each color filter 234 are arranged diagonally, and the sub-filters of the same color in the two color filters 234 are arranged in opposite directions.

Specifically, the second filter group 232 includes two color filters 234, and the same color filter 234 includes a third color sub-filter 2343 and a first color sub-filter 2341. The third color sub-filters 2343 in the same color filter 234 are arranged diagonally, and the first color sub-filters 2341 in the same color filter 234 are arranged diagonally. The arrangement directions of the first color sub-filters 2341 in the two color filters 234 are opposite, and the arrangement directions of the third color sub-filters 2343 in the two color filters 234 are opposite. For example, the third color sub-filters 2343 are respectively arranged in the diagonal A1 direction and the diagonal A2 direction in the two color filters 234.

Alternatively, the second filter group 232 includes two color filters 234, and the same color filter 234 includes a third color sub-filter 2343 and a second color sub-filter 2342. The third color sub-filters 2343 in the same color filter 234 are arranged diagonally, and the second color sub-filters 2342 in the same color filter 234 are arranged diagonally. The second color sub-filters 2342 are arranged in opposite directions in the two color filters 234, and the third color sub-filters 2343 are arranged in opposite directions in the two color filters 234.

In one embodiment, as shown in FIG7 , the minimum repeating unit 230 in the filter array 23 includes four filter groups, and the four filter groups are arranged in a matrix. Each filter group includes two full-color filters 233 and two color filters 234 . The full-color filter 233 includes four sub-filters 2331, the same color filter 234 of the first filter group 231 includes two first color sub-filters 2341 and two second color sub-filters 2342, the same color filter 234 in the second filter group 232 includes two first color sub-filters 2341 and two third color sub-filters 2343, and the same color filter 234 in the second filter group 232 includes two second color sub-filters 2342 and two third color sub-filters 2343, then the minimum repeating unit is 8 rows and 8 columns of 64 sub-filters, and the arrangement is as follows:

Wherein, w represents the full-color sub-filter 2331 , and a, b and c all represent color sub-filters. The color sub-filters include a first color sub-filter 2341 , a second color sub-filter 2342 and a third color sub-filter 2343 .

In one embodiment, as shown in FIG8 , the minimum repeating unit 230 in the filter array 23 includes four filter groups, and the four filter groups are arranged in a matrix. Each filter group includes two full-color filters 233 and two color filters 234 . The full-color filter 233 includes four sub-filters 2331, the same color filter 234 of the first filter group 231 includes two first color sub-filters 2341 and two second color sub-filters 2342, the same color filter 234 of the second filter group 232 includes two first color sub-filters 2341 and two third color sub-filters 2343, and the same color filter 234 of the second filter group 232 includes two second color sub-filters 2342 and two third color sub-filters 2343, then the minimum repeating unit is 8 rows and 8 columns of 64 sub-filters, and the arrangement is as follows:

Wherein, w represents the full-color sub-filter 2331 , and a, b and c all represent color sub-filters. The color sub-filters include a first color sub-filter 2341 , a second color sub-filter 2342 and a third color sub-filter 2343 .

In one embodiment, an image generation method is provided, which is applied to an image sensor 21 as shown in FIG. 2 . The image sensor 21 includes a filter array 23 and a pixel array 24. The filter array 23 includes a minimum repeating unit 230. The minimum repeating unit 230 includes a plurality of filter groups. Each filter group includes a color filter 234 and a panchromatic filter 233. The amount of light transmitted by the panchromatic filter 233 is greater than the amount of light transmitted by the color filter 234. The color filter 234 has a narrower spectral response than the panchromatic filter 233. Each panchromatic filter 233 includes four sub-filters 2331. Each color filter 234 includes four sub-filters. The plurality of filter groups include a first filter group 231 and a second filter group 232. At least one color filter 234 in the first filter group 231 includes a first color sub-filter 2341 and a second color sub-filter 2342. The color filter 234 of the second filter group 232 includes at least a third color sub-filter 2343 .

The pixel array 24 includes a plurality of pixels. The first color photosensitive pixel 2441 of the pixel array 24 is arranged corresponding to the first color sub-filter 2341 of the filter array 23. The second color photosensitive pixel 2442 of the pixel array 24 is arranged corresponding to the second color sub-filter 2342 of the filter array 23. The third color photosensitive pixel 2443 of the pixel array 24 is arranged corresponding to the third color sub-filter 2343 of the filter array 23. The pixel array 24 is configured to receive light passing through the filter array 23 to generate an electrical signal.

In one embodiment, the pixel array 24 includes a plurality of panchromatic pixels 2431 and a plurality of color pixels 2441, each panchromatic pixel 2431 corresponds to a sub-filter 2331 of the panchromatic filter 233, each first color photosensitive pixel 2441 corresponds to a first color sub-filter 2341 of the color filter 234, each second color photosensitive pixel 2442 corresponds to a second color sub-filter 2342 of the color filter 234, and each third color photosensitive pixel 2443 corresponds to a third color sub-filter 2343 of the color filter 234;

As shown in FIG9 , the image generation method includes:

Step 902, in the first resolution mode, obtain a first merged image based on the first pixel value read out by merging a plurality of panchromatic pixels corresponding to the panchromatic filters in the filter group, the second pixel value read out by merging a plurality of first color photosensitive pixels corresponding to the color filters, and the third pixel value read out by merging a plurality of third color photosensitive pixels corresponding to the color pixels.

The first resolution mode refers to a first-level pixel merging readout mode in which the resolution, power consumption, signal-to-noise ratio, and frame rate are relatively balanced. The first resolution mode can specifically be the default mode for image and video shooting. Merged readout refers to summing the pixel values of multiple pixels, or calculating the average of the pixel values of multiple pixels. For example, the first pixel value of a plurality of full-color pixels merged and read out can be the sum of the pixel values of a plurality of full-color pixels to obtain the first pixel value; or, the average of the pixel values of a plurality of full-color pixels is calculated as the first pixel value.

The color filter 234 has a narrower spectral response than the full-color filter 233, so the amount of light transmitted by the full-color filter 233 is greater than the amount of light transmitted by the color filter 234, that is, the band width of the light transmitted by the color filter 234 is smaller than the band width of the light transmitted by the full-color filter 233. The full-color filter 233 transmits more light, and the corresponding full-color pixel 2431 obtained through the full-color filter 233 has a higher signal-to-noise ratio. The full-color pixel contains more information and can resolve more texture details. Among them, the signal-to-noise ratio refers to the ratio between the normal signal and the noise signal. The higher the signal-to-noise ratio of a pixel, the higher the proportion of normal signals contained in the pixel, and the more information is resolved from the pixel.

The color pixels may be G (Green) pixels, R (Red) pixels, B (Blue) pixels, etc., but are not limited thereto.

When a shooting instruction is received, it is detected whether the user selects the required resolution mode. When it is detected that the user selects to use the first resolution mode, or when the user does not select the required resolution mode, preview shooting is not used, and the current environment is not night scene mode, the first resolution mode is used to respond to the shooting instruction.

In the first resolution mode, the light transmitted through the sub-filter 2331 in the full-color filter 233 is projected onto the corresponding full-color pixel 2431, and the full-color pixel 2431 receives the light transmitted through the sub-filter 2331 to generate an electrical signal. The light transmitted through the first color sub-filter 2341 in the color filter 234 is projected onto the corresponding first color photosensitive pixel 2441, and the first color photosensitive pixel 2441 generates an electrical signal by the light transmitted through the corresponding first color sub-filter 2341. The light transmitted through the second color sub-filter 2342 in the color filter 234 is projected onto the corresponding second color photosensitive pixel 2442, and the second color photosensitive pixel 2442 generates an electrical signal by the light transmitted through the corresponding second color sub-filter 2342. The light transmitted through the third color sub-filter 2343 in the color filter 234 is projected onto the corresponding third color photosensitive pixel 2443 , and the third color photosensitive pixel 2443 generates an electrical signal by the light transmitted through the corresponding third color sub-filter 2343 .

At least one color filter 234 in the first filter group 231 includes a first color sub-filter 2341 and a second color sub-filter 2342 , and at least one color filter 234 in the first filter group 231 corresponds to a first color photosensitive pixel 2441 and a second color photosensitive pixel 2442 .

The electronic device combines the multiple full-color pixels 2431 corresponding to the same full-color filter 233 to read out the first pixel value. The same color filter 234 may include multiple first color sub-filters 2341 and multiple second color sub-filters 2342 at the same time, and the electronic device combines the first color photosensitive pixels 2441 corresponding to the multiple first color sub-filters 2341 in the same color filter 234 to read out the second pixel value. The same color filter 234 may include multiple third color sub-filters 2343, and the electronic device combines the third color photosensitive pixels 2443 corresponding to the multiple third color sub-filters 2343 in the same color filter 234 to read out the third pixel value.

The electronic device obtains a first combined image according to each first pixel value, each second pixel value and each third pixel value.

Step 904, obtaining a second merged image according to the fourth pixel values read out by merging the plurality of second color photosensitive pixels corresponding to the color filter.

At least one color filter 234 in the first filter group 231 includes a first color sub-filter 2341 and a second color sub-filter 2342 , and at least one color filter 234 in the first filter group 231 corresponds to a first color photosensitive pixel 2441 and a second color photosensitive pixel 2442 .

For the color filter 234 including the second color sub-filter 2342, the electronic device reads the second color photosensitive pixels 2442 corresponding to each second color sub-filter 2342 in the same color filter 234, combines and reads out the fourth pixel value, and generates a second combined image based on each fourth pixel value.

In one embodiment, the first merged image is an image composed of the first pixel value, the second pixel value and the third pixel value, and the second merged image is an image composed of the fourth pixel value and empty pixels, wherein the empty pixels are pixels without any information.

In one embodiment, the same full-color filter 233 corresponds to 4 full-color pixels 2431. In different filter groups, the same color filter 234 can simultaneously correspond to 2 first color photosensitive pixels 2441 and 2 second color photosensitive pixels 2442. The color filter 234 can correspond to 4 third color photosensitive pixels 2443.

Step 906: Obtain a first target image based on the first merged image and the second merged image.

The electronic device can read pixel values from the first merged image and the second merged image according to a preset pixel reading method to generate a first target image. The preset pixel reading method is a pre-set pixel reading method.

In this embodiment, in the first resolution mode, the full-color channel information can be integrated into the image by merging the first pixel value read out by merging the multiple full-color pixels corresponding to the full-color filter in the filter group, the second pixel value read out by merging the multiple first-color photosensitive pixels corresponding to the color filter, and the third pixel value read out by merging the multiple third-color photosensitive pixels corresponding to the color filter, so as to increase the overall light input, so as to generate a first merged image with more information and clearer detail resolution. According to the fourth pixel value read out by merging the multiple second-color photosensitive pixels corresponding to the color filter, the second-color photosensitive pixels can be separated to form a second merged image, so that the positions of the second-color photosensitive pixels in the second merged image and the first-color photosensitive pixels in the first merged image are consistent. Based on the first merged image and the second merged image, the first target image is obtained, and the first-color photosensitive pixels in the first merged image and the second-color photosensitive pixels in the second merged image can be mixed and arranged, so that the color of the generated first target image is clearer. In addition, the pixel merging readout reduces the size of the generated first target image and the power consumption required to generate the image is low.

In one embodiment, obtaining a first target image based on the first merged image and the second merged image includes:

The pixel values in the first merged image are traversed, and when the traversed pixel value is the second pixel value, a fourth pixel value at the same position as the second pixel value is read from the second merged image, and the read fourth pixel value is adjusted to be adjacent to the second pixel value, until all the pixel values in the first merged image are traversed, so as to obtain the first target image.

Specifically, the position of each second pixel value in the first merged image is the same as the position of each fourth pixel value in the second merged image. The electronic device can traverse each pixel value in the first merged image, and in each traversal, determine whether the currently traversed pixel value is the second pixel value. In the case that the currently traversed pixel value is not the second pixel value, continue to traverse the next pixel value. In the case that the currently traversed pixel value is the second pixel value, determine the position of the currently traversed second pixel value, and read the fourth pixel value with the same position as the currently traversed second pixel value from the second merged image, and adjust the read fourth pixel value to be adjacent to the second pixel value, and then continue to traverse the next pixel value until all pixel values in the first merged image are traversed, and generate the first target image.

In this embodiment, in the first target image, the second pixel value and the fourth pixel value are adjacent, which means that the second pixel value and the fourth pixel value have the same positions in the first target image, that is, the second pixel value and the fourth pixel value have the same coordinates in the first target image.

In one embodiment, when the currently traversed pixel value is not the second pixel value, continue to traverse the next pixel value. When the currently traversed pixel value is the second pixel value, determine the position of the currently traversed second pixel value, read a fourth pixel value that is the same as the position of the currently traversed second pixel value from the second merged image, and adjust the read fourth pixel value to be horizontally adjacent to the second pixel value. The fourth pixel value is horizontally adjacent to the second pixel value, which can be to use the fourth pixel value as the adjacent previous pixel value of the second pixel value in the horizontal direction, or to use the fourth pixel value as the adjacent next pixel value of the second pixel value in the horizontal direction.

In this embodiment, the pixel values in the first merged image are traversed, and when the traversed pixel value is the second pixel value, the fourth pixel value at the same position as the second pixel value is read from the second merged image, and the read fourth pixel value is adjusted to be adjacent to the second pixel value, so that the first color photosensitive pixels and the second color photosensitive pixels are mixed and arranged, thereby improving the color resolution capability, so that each row and each column of the generated first target image has the first color photosensitive pixels, the second color photosensitive pixels and the third color photosensitive pixels, that is, each row and each column of the target image has RGB pixels, which can effectively reduce the risk of false color.

As shown in FIG. 10 , in the first resolution mode, an original image 1002 is obtained by an image sensor, and locally identical pixels in the original image 1002 are averaged and binned, that is, locally identical four panchromatic pixels are combined to read out a first pixel value, locally identical two first color light-sensitive pixels are combined to read out a second pixel value, and locally identical four third color light-sensitive pixels are combined to read out a third pixel value, to obtain a first combined image 1004. In addition, locally identical two second color light-sensitive pixels are combined to read out a fourth pixel value, to obtain a second combined image 1006. In order to keep the second combined image 1006 and the first combined image 1004 the same size, the second combined image 1006 may be generated by the fourth pixel value and empty pixels.

For example, the full-color pixel is a w pixel, the first color light-sensitive pixel is an R pixel, the third color light-sensitive pixel is a G pixel, and the second color light-sensitive pixel is a B pixel.

Pixel values are read from the first merged image 1004 and the second merged image 1006 so that each fourth pixel value is a subsequent pixel value adjacent to the second pixel value, thereby obtaining a first target image 1008 .

In one embodiment, the color filter of the second filter group further includes a first color sub-filter or a second color sub-filter; in the first resolution mode, a first merged image is obtained according to a first pixel value read out by merging a plurality of panchromatic pixels corresponding to the panchromatic filters in the filter group, a second pixel value read out by merging a plurality of first color photosensitive pixels corresponding to the color filters, and a third pixel value read out by merging a plurality of third color photosensitive pixels corresponding to the color filters, including:

In the first resolution mode, a first merged image is obtained according to a first pixel value read out by merging a plurality of full-color pixels corresponding to the full-color filters in the filter group, a second pixel value read out by merging a first color photosensitive pixels corresponding to a plurality of first color sub-filters in the first filter group, a third pixel value read out by merging a third color photosensitive pixels corresponding to a plurality of third color sub-filters in the second filter group including the first color sub-filters, and a fourth pixel value read out by merging a second color photosensitive pixels corresponding to a plurality of second color sub-filters in the second filter group;

A second merged image is obtained based on a fourth pixel value obtained by merging and reading out multiple second color photosensitive pixels corresponding to the color filters, including: a fourth pixel value obtained by merging and reading out second color photosensitive pixels corresponding to multiple second color sub-filters in the first filter group, a second pixel value obtained by merging and reading out first color photosensitive pixels corresponding to multiple first color sub-filters in the second filter group, and a third pixel value obtained by merging and reading out third color photosensitive pixels corresponding to multiple third color sub-filters in the second filter group including the second color sub-filters.

Specifically, the plurality of filter groups include a first filter group 231 and a second filter group 232. At least one color filter 234 in the first filter group 231 includes a first color sub-filter 2341 and a second color sub-filter 2342. The color filter 234 of the second filter group 232 includes a third color sub-filter 2343 and also includes the first color sub-filter 2341 or the second color sub-filter 2342.

In the first resolution mode, the electronic device combines multiple full-color pixels 2431 corresponding to the same full-color filter 233 to read out a first pixel value.

For the color filter 234 in the first filter group 231, the electronic device combines the first color photosensitive pixels 2441 corresponding to the multiple first color sub-filters 2341 of the same color filter 234 to read out the second pixel value, and combines the second color photosensitive pixels 2442 corresponding to the multiple second color sub-filters 2342 of the same color filter 234 to read out the fourth pixel value, then the same color filter 234 corresponds to the second pixel value and the fourth pixel value.

When the color filter 234 in the second filter group 232 includes a first color sub-filter 2341 and a third color sub-filter 2343, the electronic device merges the first color photosensitive pixels 2441 corresponding to the multiple first color sub-filters 2341 of the same color filter 234 to read out the second pixel value, and merges the third color photosensitive pixels 2443 corresponding to the multiple third color sub-filters 2343 to read out the third pixel value, then the same color filter 234 corresponds to the second pixel value and the third pixel value.

When the color filter 234 in the second filter group 232 includes a second color sub-filter 2342 and a third color sub-filter 2343, the electronic device merges the second color photosensitive pixels 2442 corresponding to the multiple second color sub-filters 2342 of the same color filter 234 to read out a fourth pixel value, and merges the third color photosensitive pixels 2443 corresponding to the multiple third color sub-filters 2343 of the same color filter 234 to read out a third pixel value, then the same color filter 234 corresponds to the fourth pixel value and the third pixel value.

For the color filter 234 corresponding to the second pixel value and the fourth pixel value, the electronic device can read the second pixel value; for the color filter 234 corresponding to the second pixel value and the third pixel value, the electronic device can read the third pixel value; for the color filter 234 corresponding to the fourth pixel value and the third pixel value, the electronic device can read the fourth pixel value; the electronic device generates a first merged image based on the read second pixel value, third pixel value, fourth pixel value and all the first pixel values.

For the color filter 234 corresponding to the second pixel value and the fourth pixel value, the electronic device can obtain the fourth pixel value; for the color filter 234 corresponding to the second pixel value and the third pixel value, the electronic device can obtain the second pixel value; for the color filter 234 corresponding to the fourth pixel value and the third pixel value, the electronic device can obtain the third pixel value; the electronic device generates a second merged image based on the obtained fourth pixel value, second pixel value, third pixel value and empty pixels.

The electronic device reads pixel values from the first merged image and the second merged image to generate a first target image.

In this embodiment, the color filter 234 of the second filter group 232 includes a third color sub-filter 2343 and also includes a first color sub-filter 2341 or a second color sub-filter 2342. In the first resolution mode, a first pixel value is read out by merging a plurality of full-color pixels corresponding to the full-color filters in the filter group, a second pixel value is read out by merging a plurality of first color photosensitive pixels corresponding to the plurality of first color sub-filters in the first filter group, a third pixel value is read out by merging a plurality of third color photosensitive pixels corresponding to the plurality of third color sub-filters in the second filter group including the first color sub-filter, and a fourth pixel value is read out by merging a plurality of second color photosensitive pixels corresponding to the plurality of second color sub-filters in the second filter group, so that the first merged image obtained by the merged reading contains pixels of all colors, thereby improving the resolution capability of the color channel. A second merged image is obtained based on the second pixel value read out by merging multiple first color photosensitive pixels corresponding to the color filter, the third pixel value read out by merging multiple third color photosensitive pixels corresponding to the color pixel, and the fourth pixel value read out by merging multiple second color photosensitive pixels corresponding to the color filter, so that the pixel distribution of each color in the first target image generated according to the first merged image and the second merged image is more uniform, effectively improving the clarity of the image.

As shown in FIG. 11 , in the first resolution mode, an original image 1102 is obtained by an image sensor, four adjacent full-color pixels in the original image 1102 are merged to read out a first pixel value, two first-color light-sensitive pixels on a diagonal are merged to read out a second pixel value, two third-color light-sensitive pixels on a diagonal are merged to read out a third pixel value, and two second-color light-sensitive pixels on a diagonal are merged to read out a fourth pixel value; the second pixel value, the third pixel value, the fourth pixel value, and all the first pixel values are read to generate a first merged image 1104; the remaining fourth pixel value, the second pixel value, and the third pixel value are read, and combined with empty pixels to generate a second merged image 1106. The electronic device can generate a first target image based on the first merged image 1104 and the second merged image 1106.

In one embodiment, as shown in FIG12 , the method further includes:

Step 1202, in the second resolution mode, a first merged image is obtained based on a first pixel value read out by merging a plurality of panchromatic pixels corresponding to the panchromatic filters in the filter group, a second pixel value read out by merging a plurality of first color photosensitive pixels corresponding to the color filters, and a third pixel value read out by merging a plurality of third color photosensitive pixels corresponding to the color filters; the resolution corresponding to the second resolution mode is smaller than the resolution corresponding to the first resolution mode.

The second resolution mode refers to the mode used in scenarios with relatively low resolution requirements. It is a secondary pixel merging readout mode with low resolution, low power consumption, high signal-to-noise ratio, and high frame rate. The resolution and power consumption corresponding to the second resolution mode are lower than those corresponding to the first resolution mode. The signal-to-noise ratio and frame rate corresponding to the second resolution mode are higher than those corresponding to the first resolution mode.

The second resolution mode may be a preview mode for image capture, a preview mode for video capture, or a scene with lower resolution requirements such as image capture in night scenes or night scene mode for video capture, but is not limited thereto. The preview mode for video capture may be, for example, 1080p video preview, application video preview, etc.

When receiving a shooting instruction, determining whether the shooting instruction is a preview shooting. When the shooting instruction is a preview shooting, triggering the second resolution mode. Alternatively, the electronic device detects whether the current environment is a night scene, and when the current environment is a night scene, triggering the second resolution mode. Alternatively, when the user selects the second resolution mode, triggering the readout mode corresponding to the second resolution mode.

In the second resolution mode, the light transmitted through the sub-filter 2331 in the full-color filter 233 is projected onto the corresponding full-color pixel 2431, and the full-color pixel 2431 receives the light transmitted through the sub-filter 2331 to generate an electrical signal. The light transmitted through the first color sub-filter 2341 in the color filter 234 is projected onto the corresponding first color photosensitive pixel 2441, and the first color photosensitive pixel 2441 generates an electrical signal by the light transmitted through the corresponding first color sub-filter 2341. The light transmitted through the second color sub-filter 2342 in the color filter 234 is projected onto the corresponding second color photosensitive pixel 2442, and the second color photosensitive pixel 2442 generates an electrical signal by the light transmitted through the corresponding second color sub-filter 2342. The light transmitted through the third color sub-filter 2343 in the color filter 234 is projected onto the corresponding third color photosensitive pixel 2443 , and the third color photosensitive pixel 2443 generates an electrical signal by the light transmitted through the corresponding third color sub-filter 2343 .

At least one color filter 234 in the first filter group 231 includes a first color sub-filter 2341 and a second color sub-filter 2342 , and at least one color filter 234 in the first filter group 231 corresponds to a first color photosensitive pixel 2441 and a second color photosensitive pixel 2442 .

The electronic device combines the multiple full-color pixels 2431 corresponding to the same full-color filter 233 to read out the first pixel value. The same color filter 234 may include multiple first color sub-filters 2341 and multiple second color sub-filters 2342 at the same time, and the electronic device combines the first color photosensitive pixels 2441 corresponding to the multiple first color sub-filters 2341 in the same color filter 234 to read out the second pixel value. The same color filter 234 may include multiple third color sub-filters 2343, and the electronic device combines the third color photosensitive pixels 2443 corresponding to the multiple third color sub-filters 2343 in the same color filter 234 to read out the third pixel value.

The electronic device obtains a first combined image according to each first pixel value, each second pixel value and each third pixel value.

Step 1204, obtaining a second merged image according to the fourth pixel values read out by merging the plurality of second color photosensitive pixels corresponding to the color filter.

For the color filter 234 including the second color sub-filter 2342, the electronic device reads the second color photosensitive pixels 2442 corresponding to each second color sub-filter 2342 in the same color filter 234, combines and reads out the fourth pixel value, and generates a second combined image based on each fourth pixel value.

Step 1206 , obtaining a full-color combined image based on the fifth pixel value read out by combining a plurality of full-color pixels in the ninth diagonal direction in the first combined image.

The electronic device combines a plurality of full-color pixels in the ninth diagonal direction in the first combined image to read out fifth pixel values, and generates a full-color combined image according to each fifth pixel value. The full-color combined image contains all full-color pixels.

Step 1208, based on the sixth pixel value read out by merging multiple first color photosensitivity pixels in the tenth diagonal direction in the first merged image, and the seventh pixel value read out by merging multiple third color photosensitivity pixels in the tenth diagonal direction, a third merged image is obtained, and the ninth diagonal direction is different from the tenth diagonal direction.

The electronic device combines a plurality of first color light-sensitive pixels in the tenth diagonal direction in the first combined image to read out a sixth pixel value, and combines a plurality of third color light-sensitive pixels in the tenth diagonal direction to read out a seventh pixel value, and generates a third combined image according to the sixth pixel value and the seventh pixel value. The third combined image includes first color light-sensitive pixels and third color light-sensitive pixels. The ninth diagonal direction is different from the tenth diagonal direction, and specifically, the seventh diagonal direction is not parallel to the tenth diagonal direction, or the ninth diagonal direction is perpendicular to the tenth diagonal direction, etc.

Step 1210, obtaining a fourth merged image based on the eighth pixel value read out by merging a plurality of second color light-sensitive pixels in the tenth diagonal direction in the second merged image.

The electronic device combines a plurality of second color light-sensitive pixels in the tenth diagonal direction in the second combined image to read out eighth pixel values, and generates a fourth combined image based on the eighth pixel values.

In one embodiment, the electronic device combines two second color photosensitive pixels in the tenth diagonal direction in the second combined image to read out the eighth pixel value, and generates a fourth combined image based on the eighth pixel values and empty pixels.

Step 1212: Obtain a second target image based on the full-color merged image, the third merged image, and the fourth merged image.

The electronic device may generate a second target image based on reading pixel values from the full-color combined image, the third combined image, and the fourth combined image.

In this embodiment, in the second resolution mode, a first merged image is obtained according to the first pixel value read out by merging multiple panchromatic pixels corresponding to the panchromatic filters in the filter group, the second pixel value read out by merging multiple color pixels corresponding to the color filters, and the third pixel value read out by merging multiple third color photosensitive pixels corresponding to the color filters, and a second merged image is obtained according to the fourth pixel value read out by merging multiple second color photosensitive pixels corresponding to the color filters, so that the second color photosensitive pixels can be separated as a separate image and the resolution of the image can be reduced. A panchromatic merged image is obtained based on the fifth pixel value read out by merging multiple panchromatic pixels in the ninth diagonal direction in the first merged image, so that the panchromatic pixels can be separated from the first merged image while reducing the size of the image to reduce power consumption. A third merged image including the first color photosensitive pixels and the third color photosensitive pixels is obtained based on the sixth pixel value read out by merging multiple first color photosensitive pixels in the tenth diagonal direction in the first merged image, and the seventh pixel value read out by merging multiple third color photosensitive pixels in the tenth diagonal direction, so that the generated image has less noise. Based on the eighth pixel value read out by merging multiple second color photosensitive pixels in the tenth diagonal direction in the second merged image, the size of the fourth merged image is kept consistent with the size of the third merged image and the full-color merged image. Based on the full-color merged image, the third merged image and the fourth merged image, different first color photosensitive pixels and second color photosensitive pixels can be mixed and arranged, so that the RGB pixels in the generated second target image are more evenly distributed and the image quality is higher. In addition, the resolution of the obtained second target image is further reduced, and the full-color pixels have a higher signal-to-noise ratio, and the frame rate of the image is high, thereby achieving an image processing effect with lower power consumption and better signal-to-noise ratio of the secondary pixel merging output.

In one embodiment, obtaining a second target image based on the full-color merged image, the third merged image, and the fourth merged image includes:

Traverse the pixels in the third merged image; when the traversed pixel value is the sixth pixel value or the seventh pixel value, read the fifth pixel value corresponding to the traversed pixel value from the full-color merged image, and adjust the read fifth pixel value to be adjacent to the traversed pixel value; when the traversed pixel value is the sixth pixel value, read the eighth pixel value at the same position as the sixth pixel from the fourth merged image, and adjust the read eighth pixel value to be adjacent to the sixth pixel value, until the pixel values in the full-color merged image, the third merged image and the fourth merged image are all read, and the second target image is obtained.

Specifically, the position of each sixth pixel value in the third merged image is the same as the position of each eighth pixel value in the fourth merged image. The position of each panchromatic pixel in the panchromatic merged image is the same as the position of each sixth pixel value in the third merged image and the position of each eighth pixel value in the fourth merged image.

The electronic device may traverse each pixel value in the third merged image, and in each traversal, determine whether the currently traversed pixel value is the sixth pixel value or the seventh pixel value. If the currently traversed pixel value is neither the sixth pixel value nor the seventh pixel value, continue to traverse the next pixel value.

When the pixel value currently traversed is the sixth pixel value or the seventh pixel value, the position of the pixel value currently traversed is determined, the fifth pixel value corresponding to the traversed pixel value is read from the full-color merged image, and the read fifth pixel value is adjusted to be adjacent to the pixel value currently traversed. Furthermore, when the pixel value currently traversed is the sixth pixel value, the eighth pixel value having the same position as the sixth pixel is read from the fourth merged image, the read eighth pixel value is adjusted to be adjacent to the sixth pixel value, and then the next pixel value is traversed continuously until the pixel values in the full-color merged image, the third merged image, and the fourth merged image are all read, thereby obtaining the second target image.

In this embodiment, if the sixth pixel value and the eighth pixel value are adjacent in the second target image, it means that the sixth pixel value and the eighth pixel value have the same position in the second target image, that is, it means that the sixth pixel value and the eighth pixel value have the same coordinates in the second target image.

In one embodiment, the fifth pixel value is adjacent to the currently traversed pixel value, which may be that the fifth pixel value is adjacent to the currently traversed pixel value in the horizontal direction. Specifically, the fifth pixel value may be taken as the adjacent previous pixel value or the adjacent next pixel value of the currently traversed pixel value in the horizontal direction.

The eighth pixel value is adjacent to the sixth pixel value, which may be that the eighth pixel value is horizontally adjacent to the sixth pixel value. The eighth pixel value is horizontally adjacent to the sixth pixel value, which may be that the eighth pixel value is used as the adjacent previous pixel value of the sixth pixel value in the horizontal direction, or that the eighth pixel value is used as the adjacent subsequent pixel value of the sixth pixel value in the horizontal direction.

In one embodiment, when the fifth pixel value is used as the adjacent previous pixel value of the sixth pixel value in the horizontal direction, the eighth pixel value is used as the adjacent subsequent pixel value of the sixth pixel value in the horizontal direction.

In this embodiment, the pixels in the third merged image are traversed, and when the traversed pixel value is the sixth pixel value or the seventh pixel value, the fifth pixel value corresponding to the traversed pixel value is read from the full-color merged image, and the read fifth pixel value is adjusted to be adjacent to the traversed pixel value, so that full-color pixels are introduced into the image, and the amount of light entering is increased. When the traversed pixel value is the sixth pixel value, the eighth pixel value at the same position as the sixth pixel is read from the fourth merged image, and the read eighth pixel value is adjusted to be adjacent to the sixth pixel value, so that the first color photosensitive pixels and the second color photosensitive pixels are mixed and arranged, and the color resolution ability is improved, so that each row and each column of the generated second target image has the first color photosensitive pixels, the second color photosensitive pixels, and the second color photosensitive pixels, that is, each row and each column of the target image has RGB pixels, which can effectively reduce the risk of false colors.

As shown in FIG. 13 , in the second resolution mode, an original image 1302 is obtained by an image sensor, and the four partially identical full-color pixels in the original image 1302 are merged to read out a first pixel value, the two partially identical first-color light-sensitive pixels are merged to read out a second pixel value, and the four partially identical third-color light-sensitive pixels are merged to read out a third pixel value, thereby obtaining a first merged image 1304. Furthermore, the two partially identical second-color light-sensitive pixels are merged to read out a fourth pixel value, thereby obtaining a second merged image 1306. In order to keep the second merged image 1306 and the first merged image 1304 the same size, the second merged image 1306 may be generated by the fourth pixel value and the empty pixel.

The two full-color pixels in the direction of the ninth diagonal line D9 in the first merged image 1304 are merged to read out a fifth pixel value, thereby obtaining a full-color merged image 1308. The two first-color light-sensitive pixels in the direction of the tenth diagonal line D10 in the first merged image 1304 are merged to read out a sixth pixel value, and the two third-color light-sensitive pixels in the direction of the tenth diagonal line D10 are merged to read out a seventh pixel value, thereby obtaining a third merged image 1310.

The fourth merged image 1312 is obtained based on the eighth pixel value read out by merging the two second color light-sensitive pixels in the direction of the tenth diagonal line D10 in the second merged image 1306 .

The pixels in the third merged image 1310 are traversed. When the traversed pixel value is the sixth pixel value or the seventh pixel value, the fifth pixel value corresponding to the traversed pixel value is read from the full-color merged image 1308, and the read fifth pixel value is used as the previous pixel value adjacent to the traversed pixel value. Furthermore, when the traversed pixel value is the sixth pixel value, the eighth pixel value at the same position as the sixth pixel is read from the fourth merged image 1312, and the read eighth pixel value is used as the next pixel value adjacent to the sixth pixel value, until all the pixel values in the third merged image are traversed, and the second target image 1314 is obtained.

In one implementation, in the second resolution mode, the color filter of the second filter group further includes a first color sub-filter or a second color sub-filter; a first merged image is obtained according to a first pixel value obtained by merging and reading a plurality of panchromatic pixels corresponding to the panchromatic filters in the filter group, a second pixel value obtained by merging and reading a plurality of first color photosensitive pixels corresponding to the color filters, and a third pixel value obtained by merging and reading a plurality of third color photosensitive pixels corresponding to the color filters, including:

In the second resolution mode, a first merged image is obtained according to a first pixel value obtained by merging and reading a plurality of full-color pixels corresponding to the full-color filters in the filter group, a second pixel value obtained by merging and reading a first color photosensitive pixel corresponding to a plurality of first color sub-filters in the first filter group, a third pixel value obtained by merging and reading a third color photosensitive pixel corresponding to a plurality of third color sub-filters in the second filter group including the first color sub-filters, and a fourth pixel value obtained by merging and reading a second color photosensitive pixel corresponding to a plurality of second color sub-filters in the second filter group;

The second merged image is obtained by merging and reading out fourth pixel values of a plurality of second color photosensitive pixels corresponding to the color filter, including:

A second merged image is obtained by combining and reading a fourth pixel value of second color photosensitive pixels corresponding to a plurality of second color sub-filters in the first filter group, combining and reading a second pixel value of first color photosensitive pixels corresponding to a plurality of first color sub-filters in the second filter group, and combining and reading a third pixel value of third color photosensitive pixels corresponding to a plurality of third color sub-filters in the second filter group including the second color sub-filter;

Based on the sixth pixel value obtained by merging and reading out multiple first color photosensitive pixels in the tenth diagonal direction in the first merged image, and the seventh pixel value obtained by merging and reading out multiple third color photosensitive pixels in the tenth diagonal direction, a third merged image is obtained, including: based on the fifth pixel value obtained by merging and reading out multiple full-color pixels in the ninth diagonal direction in the first merged image, the sixth pixel value obtained by merging and reading out multiple first color photosensitive pixels in the tenth diagonal direction, the seventh pixel value obtained by merging and reading out multiple third color photosensitive pixels in the tenth diagonal direction, and the eighth pixel value obtained by merging and reading out multiple second color photosensitive pixels in the tenth diagonal direction, respectively obtaining a full-color merged image and a third merged image;

A fourth merged image is obtained based on the eighth pixel value read out by merging multiple second color photosensitive pixels in the tenth diagonal direction in the second merged image, including: a fourth merged image is obtained based on the sixth pixel value read out by merging multiple first color photosensitive pixels in the tenth diagonal direction in the second merged image, a seventh pixel value read out by merging multiple third color photosensitive pixels in the tenth diagonal direction, and an eighth pixel value read out by merging multiple second color photosensitive pixels in the tenth diagonal direction.

Specifically, the plurality of filter groups include a first filter group 231 and a second filter group 232. The color filter 234 in the first filter group 231 includes a first color sub-filter 2341 and a second color sub-filter 2342. The color filter 234 in the second filter group 232 includes a third color sub-filter 2343 and further includes the first color sub-filter 2341 or the second color sub-filter 2342.

In the second resolution mode, the electronic device combines multiple full-color pixels 2431 corresponding to the same full-color filter 233 to read out a first pixel value.

For the color filter 234 in the first filter group 231, the electronic device combines the first color photosensitive pixels 2441 corresponding to the multiple first color sub-filters 2341 of the same color filter 234 to read out the second pixel value, and combines the second color photosensitive pixels 2442 corresponding to the multiple second color sub-filters 2342 of the same color filter 234 to read out the fourth pixel value, then the same color filter 234 corresponds to the second pixel value and the fourth pixel value.

When the color filter 234 in the second filter group 232 includes a first color sub-filter 2341 and a third color sub-filter 2343, the electronic device merges the first color photosensitive pixels 2441 corresponding to the multiple first color sub-filters 2341 of the same color filter 234 to read out the second pixel value, and merges the third color photosensitive pixels 2443 corresponding to the multiple third color sub-filters 2343 to read out the third pixel value, then the same color filter 234 corresponds to the second pixel value and the third pixel value.

When the color filter 234 in the second filter group 232 includes a second color sub-filter 2342 and a third color sub-filter 2343, the electronic device merges the second color photosensitive pixels 2442 corresponding to the multiple second color sub-filters 2342 of the same color filter 234 to read out a fourth pixel value, and merges the third color photosensitive pixels 2443 corresponding to the multiple third color sub-filters 2343 of the same color filter 234 to read out a third pixel value, then the same color filter 234 corresponds to the fourth pixel value and the third pixel value.

For the color filter 234 corresponding to the second pixel value and the fourth pixel value, the electronic device can read the second pixel value; for the color filter 234 corresponding to the second pixel value and the third pixel value, the electronic device can read the third pixel value; for the color filter 234 corresponding to the fourth pixel value and the third pixel value, the electronic device can read the fourth pixel value; the electronic device generates a first merged image based on the read second pixel values, third pixel values, fourth pixel values and all first pixel values. The first merged image includes full-color pixels, first color photosensitive pixels, second color photosensitive pixels and third color photosensitive pixels.

For the first combined image, the electronic device combines a plurality of full-color pixels in the ninth diagonal direction to read out fifth pixel values, and generates a full-color combined image according to each fifth pixel value. The full-color combined image contains all full-color pixels.

For the first merged image, the electronic device combines the plurality of first color photosensitive pixels in the tenth diagonal direction in the first merged image to read out the sixth pixel value, combines the plurality of third color photosensitive pixels in the tenth diagonal direction to read out the seventh pixel value, and combines the plurality of second color photosensitive pixels in the tenth diagonal direction to read out the eighth pixel value. A third merged image is generated based on the read out sixth pixel value, seventh pixel value, and eighth pixel value. The third merged image includes the first color photosensitive pixels, the second color photosensitive pixels, and the third color photosensitive pixels.

For the color filter 234 corresponding to the second pixel value and the fourth pixel value, the electronic device can obtain the fourth pixel value; for the color filter 234 corresponding to the second pixel value and the third pixel value, the electronic device can obtain the second pixel value; for the color filter 234 corresponding to the fourth pixel value and the third pixel value, the electronic device can obtain the third pixel value; the electronic device generates a second merged image based on the obtained fourth pixel value, second pixel value, third pixel value and empty pixels.

For the second merged image, the electronic device merges and reads out a sixth pixel value from a plurality of first color light-sensitive pixels in the tenth diagonal direction, merges and reads out a seventh pixel value from a plurality of third color light-sensitive pixels in the tenth diagonal direction, and merges and reads out an eighth pixel value from a plurality of second color light-sensitive pixels in the tenth diagonal direction. A fourth merged image is generated based on the read out sixth pixel value, seventh pixel value, and eighth pixel value, as well as empty pixels. The fourth merged image includes first color light-sensitive pixels, second color light-sensitive pixels, third color light-sensitive pixels, and empty pixels.

The electronic device may generate a second target image according to the full-color combined image, the third combined image and the fourth combined image.

In this embodiment, in the second resolution mode, a first pixel value is read out according to a combination of a plurality of full-color pixels corresponding to the full-color filters in the filter group, a second pixel value is read out according to a combination of a first color photosensitive pixel corresponding to a plurality of first color sub-filters in the first filter group, a third pixel value is read out according to a combination of a third color photosensitive pixel in the second filter group including the first color sub-filter, and a fourth pixel value is read out according to a combination of a third color photosensitive pixel corresponding to a plurality of second color sub-filters in the second filter group to obtain a first merged image, and a plurality of second color sub-filters in the first filter group are combined to form a first merged image. The second merged image is obtained by combining and reading the fourth pixel value of the second color photosensitive pixels corresponding to the multiple first color sub-filters in the second filter group, and combining and reading the third pixel value of the third color photosensitive pixels corresponding to the multiple third color sub-filters in the second filter group including the second color sub-filter. The first color photosensitive pixels, the second color photosensitive pixels and the third color photosensitive pixels corresponding to each color filter can be separated to form two merged images, so that the photosensitive pixels of each color in the two merged images are in the same position in each merged image.

The full-color merged image is obtained based on the fifth pixel value read out by merging multiple full-color pixels in the ninth diagonal direction in the first merged image, and the full-color pixels can be separated from the first merged image while reducing the size of the image to reduce power consumption.

Based on the sixth pixel value read out by merging multiple first color photosensitive pixels in the tenth diagonal direction in the first merged image, the seventh pixel value read out by merging multiple third color photosensitive pixels in the tenth diagonal direction, and the eighth pixel value read out by merging multiple second color photosensitive pixels in the tenth diagonal direction, a third merged image including first color photosensitive pixels, second color photosensitive pixels and third color photosensitive pixels is obtained, so that the distribution of RGB pixels in the image is more uniform. Based on the sixth pixel value read out by merging multiple first color photosensitive pixels in the tenth diagonal direction, the seventh pixel value read out by merging multiple third color photosensitive pixels in the tenth diagonal direction, and the eighth pixel value read out by merging multiple second color photosensitive pixels in the tenth diagonal direction in the second merged image, a fourth merged image is obtained, so that the size of the fourth merged image is consistent with that of the third merged image and the full-color merged image. Based on the full-color merged image, the third merged image and the fourth merged image, different first color photosensitive pixels, second color photosensitive pixels and third color photosensitive pixels can be mixed and arranged, so that the distribution of RGB pixels in the generated second target image is more uniform and the image quality is higher. Moreover, the resolution of the obtained second target image is further reduced, and the full-color pixels have a higher signal-to-noise ratio, and the image frame rate is high, thereby achieving an image processing effect with lower power consumption and better signal-to-noise ratio of the secondary pixel merging output.

As shown in FIG14 , in the second resolution mode, an original image 1402 is obtained through an image sensor, four adjacent full-color pixels in the original image 1402 are locally merged to read out a first pixel value, two first-color light-sensitive pixels on the diagonal are merged to read out a second pixel value, two third-color light-sensitive pixels on the diagonal are merged to read out a third pixel value, and two second-color light-sensitive pixels on the diagonal are merged to read out a fourth pixel value; the second pixel value, the third pixel value, the fourth pixel value and all the first pixel values are read to generate a first merged image 1404; the remaining fourth pixel values, second pixel values and third pixel values are read and combined with empty pixels to generate a second merged image 1406.

The two full-color pixels in the direction of the ninth diagonal line D9 in the first merged image 1404 are merged to read out a fifth pixel value, thereby obtaining a full-color merged image 1408. The two first-color light-sensitive pixels in the direction of the tenth diagonal line D10 in the first merged image 1404 are merged to read out a sixth pixel value, and the two third-color light-sensitive pixels in the direction of the tenth diagonal line D10 are merged to read out a seventh pixel value, and the two second-color light-sensitive pixels in the direction of the tenth diagonal line D10 are merged to read out an eighth pixel value. Based on the read out sixth pixel value, seventh pixel value, and eighth pixel value, a third merged image 1410 is generated.

For the second merged image 1406, the two first color light-sensitive pixels in the direction of the tenth diagonal line D10 in the second merged image 1406 are merged to read out the sixth pixel value, the two third color light-sensitive pixels in the direction of the tenth diagonal line D10 are merged to read out the seventh pixel value, and the two second color light-sensitive pixels in the direction of the tenth diagonal line D10 are merged to read out the eighth pixel value. A fourth merged image 1412 is generated based on the read out sixth pixel value, seventh pixel value, and eighth pixel value.

The electronic device may generate a second target image based on the full-color merged image 1408 , the third merged image 1410 , and the fourth merged image 1412 .

In one embodiment, the method also includes: in the second resolution mode, obtaining a full-color image based on a first pixel value read out by merging a plurality of full-color pixels corresponding to the filter group; obtaining a fifth merged image based on a second pixel value read out by merging a plurality of first-color photosensitive pixels corresponding to the filter group, and a third pixel value read out by merging a plurality of third-color photosensitive pixels corresponding to the filter group; obtaining a sixth merged image based on a fourth pixel value read out by merging a plurality of second-color photosensitive pixels corresponding to the filter group; and obtaining a third target image based on the full-color image, the fifth merged image and the sixth merged image.

The plurality of filter groups include a first filter group 231 and a second filter group 232. At least one color filter 234 in the first filter group 231 includes a first color sub-filter 2341 and a second color sub-filter 2342. Then, at least one color filter 234 in the first filter group 231 corresponds to a first color photosensitive pixel 2441 and a second color photosensitive pixel 2442. The color filter 234 in the second filter group 232 includes at least a third color sub-filter 2343. Then, the color filter 234 in the second filter group 232 corresponds to at least a third color photosensitive pixel 2443.

In the second resolution mode, the electronic device merges the panchromatic pixels 2431 corresponding to all panchromatic filters 233 in the same filter group and reads out the first pixel value, and obtains a panchromatic image according to each first pixel value. Further, the electronic device merges the panchromatic pixels 2431 corresponding to all panchromatic filters 233 in each first filter group 231 and reads out the first pixel value, so that one first filter group 231 corresponds to one first pixel value, so as to obtain the first pixel value corresponding to each first filter group 231. The electronic device merges the panchromatic pixels 2431 corresponding to all panchromatic filters 233 in each second filter group 232 and reads out the first pixel value, so that one second filter group 232 corresponds to one first pixel value, so as to obtain the first pixel value corresponding to each second filter group 232. A panchromatic image is obtained according to the first pixel value corresponding to each first filter group 231 and the first pixel value corresponding to each second filter group 232.

The electronic device combines the first color photosensitive pixels 2441 corresponding to the multiple first color sub-filters 2341 of all color filters 234 in the same filter group to read out the second pixel value. The electronic device combines the third color photosensitive pixels 2443 corresponding to the multiple third color sub-filters 2343 of all color filters 234 in the same filter group to read out the third pixel value. The electronic device obtains a fifth combined image based on the second pixel values and the third pixel values.

Further, the electronic device respectively combines and reads out the second pixel value of each first color photosensitive pixel 2441 corresponding to the multiple first color sub-filters 2341 of all color filters 234 in each first filter group 231, so that one first filter group 231 corresponds to one second pixel value, so as to obtain the second pixel value corresponding to each first filter group 231. The electronic device respectively combines and reads out the third pixel value of each third color photosensitive pixel 2443 corresponding to the multiple third color sub-filters 2343 of all color filters 234 in each second filter group 232, so that one second filter group 232 corresponds to one third pixel value, so as to obtain the third pixel value corresponding to each second filter group 232. The fifth combined image is obtained according to the second pixel value corresponding to each first filter group 231 and the third pixel value corresponding to each second filter group 232.

The electronic device combines the second color photosensitive pixels 2442 corresponding to the multiple second color sub-filters 2342 of all color filters 234 in the same filter group and reads out a fourth pixel value. Further, the electronic device combines the second color photosensitive pixels 2442 corresponding to the multiple second color sub-filters 2342 of all color filters 234 in each first filter group 231 and reads out a fourth pixel value, so that one first filter group 231 corresponds to one fourth pixel value, so as to obtain the fourth pixel values corresponding to each first filter group 231. The electronic device obtains a sixth combined image according to the fourth pixel values corresponding to each first filter group 231.

The electronic device may read pixel values from the full-color image, the fifth combined image, and the sixth combined image according to a preset pixel reading method to generate a third target image.

In this embodiment, in the second resolution mode, a full-color image can be separated by merging the first pixel value read out by the plurality of full-color pixels corresponding to the filter group; according to the second pixel value read out by merging the plurality of first-color photosensitive pixels corresponding to the filter group, and the third pixel value read out by merging the plurality of third-color photosensitive pixels corresponding to the filter group, the first-color photosensitive pixels and the third-color photosensitive pixels can be separated to form a fifth merged image, which can quickly and effectively reduce the image size. According to the fourth pixel value read out by merging the plurality of second-color photosensitive pixels corresponding to the filter group, the second-color photosensitive pixels can be separated to form a sixth merged image, so that the coordinate positions of the second-color photosensitive pixels in the sixth merged image and the first-color photosensitive pixels in the fifth merged image are consistent. The third target image is obtained based on the full-color image, the fifth merged image, and the sixth merged image, so that the full-color channel information can be integrated into the image, the overall amount of light input can be increased, and the third target image with more information and clearer detail resolution can be generated. And the size of the generated third target image is reduced by merging and reading the plurality of pixels corresponding to the filter group, and the power consumption required to generate the image is low.

In one embodiment, the color filter of the second filter group further includes a first color sub-filter or a second color sub-filter; a fifth merged image is obtained according to a second pixel value read out by merging a plurality of first color photosensitive pixels corresponding to the filter group and a third pixel value read out by merging a plurality of third color photosensitive pixels corresponding to the filter group, including:

Obtain a fifth merged image according to a second pixel value obtained by merging and reading out first color photosensitive pixels corresponding to a plurality of first color sub-filters in the first filter group, a third pixel value obtained by merging and reading out third color photosensitive pixels corresponding to a plurality of third color sub-filters in the second filter group including the first color sub-filter, and a fourth pixel value obtained by merging and reading out second color photosensitive pixels corresponding to a plurality of second color sub-filters in the second filter group;

A sixth merged image is obtained by merging and reading out fourth pixel values of a plurality of second color light-sensitive pixels corresponding to the filter group, including:

A sixth merged image is obtained based on the fourth pixel value read out by merging the second color photosensitive pixels corresponding to multiple second color sub-filters in the first filter group, the second pixel value read out by merging the first color photosensitive pixels corresponding to multiple first color sub-filters in the second filter group, and the third pixel value read out by merging the third color photosensitive pixels corresponding to multiple third color sub-filters in the second filter group including the second color sub-filters.

The plurality of filter groups include a first filter group 231 and a second filter group 232. At least one color filter 234 in the first filter group 231 includes a first color sub-filter 2341 and a second color sub-filter 2342. Then, at least one color filter 234 in the first filter group 231 corresponds to a first color photosensitive pixel 2441 and a second color photosensitive pixel 2442. The color filter 234 in the second filter group 232 includes a third color sub-filter 2343 and also includes the first color sub-filter 2341 or the second color sub-filter 2342.

In the second resolution mode, the electronic device merges the panchromatic pixels 2431 corresponding to all panchromatic filters 233 in the same filter group and reads out the first pixel value, and obtains a panchromatic image according to each first pixel value. Further, the electronic device merges the panchromatic pixels 2431 corresponding to all panchromatic filters 233 in each first filter group 231 and reads out the first pixel value, so that one first filter group 231 corresponds to one first pixel value, so as to obtain the first pixel value corresponding to each first filter group 231. The electronic device merges the panchromatic pixels 2431 corresponding to all panchromatic filters 233 in each second filter group 232 and reads out the first pixel value, so that one second filter group 232 corresponds to one first pixel value, so as to obtain the first pixel value corresponding to each second filter group 232. A panchromatic image is obtained according to the first pixel value corresponding to each first filter group 231 and the first pixel value corresponding to each second filter group 232.

The electronic device respectively combines the first color photosensitive pixels 2441 corresponding to multiple first color sub-filters 2341 of all color filters 234 in each first filter group 231 and reads out the second pixel value, so that one first filter group 231 corresponds to one second pixel value, to obtain the second pixel values corresponding to each first filter group 231.

For the second filter group 232 including the first color sub-filter 2341 and the third color sub-filter 2343, the electronic device respectively merges the third color photosensitive pixels 2443 corresponding to the multiple third color sub-filters 2343 of all the color filters 234 in each second filter group 232 and reads out the third pixel value, so that a second filter group 232 including the first color sub-filter 2341 corresponds to a third pixel value, so as to obtain the third pixel value corresponding to each second filter group 232 including the first color sub-filter 2341.

For the second filter group 232 including the second color sub-filter 2342 and the third color sub-filter 2343, the electronic device respectively merges the second color photosensitive pixels 2442 corresponding to the multiple second color sub-filters 2342 of all the color filters 234 in each second filter group 232 and reads out the fourth pixel value, so that a second filter group 232 including the second color sub-filter 2342 corresponds to a fourth pixel value, so as to obtain the fourth pixel value corresponding to each second filter group 232 including the second color sub-filter 2342.

The electronic device obtains a fifth merged image based on the second pixel values corresponding to each first filter group 231, the third pixel values corresponding to each second filter group 232 including the first color sub-filter 2341, and the fourth pixel values corresponding to each second filter group 232 including the second color sub-filter 2342.

The electronic device respectively combines the second color photosensitive pixels 2442 corresponding to the multiple second color sub-filters 2342 of all the color filters 234 in each first filter group 231 and reads out the fourth pixel value, so that one first filter group 231 corresponds to one fourth pixel value, to obtain the fourth pixel value corresponding to each first filter group 231.

For the second filter group 232 including the first color sub-filter 2341 and the third color sub-filter 2343, the electronic device respectively merges the first color photosensitive pixels 2441 corresponding to the multiple first color sub-filters 2341 of all the color filters 234 in each second filter group 232 and reads out the second pixel value, so that a second filter group 232 including the first color sub-filter 2341 corresponds to a second pixel value, so as to obtain the second pixel value corresponding to each second filter group 232 including the first color sub-filter 2341.

For the second filter group 232 including the second color sub-filter 2342 and the third color sub-filter 2343, the electronic device respectively merges the third color photosensitive pixels 2443 corresponding to the multiple third color sub-filters 2343 of all the color filters 234 in each second filter group 232 and reads out the third pixel value, so that a second filter group 232 including the second color sub-filter 2342 corresponds to a third pixel value, so as to obtain the third pixel value corresponding to each second filter group 232 including the second color sub-filter 2342.

The electronic device obtains a sixth merged image based on the fourth pixel values corresponding to each first filter group 231, the second pixel values corresponding to each second filter group 232 including the first color sub-filter 2341, and the third pixel values corresponding to each second filter group 232 including the second color sub-filter 2342.

The electronic device may read pixel values from the full-color image, the fifth combined image, and the sixth combined image according to a preset pixel reading method to generate a third target image.

In this embodiment, in the second resolution mode, by combining and reading out the second pixel value of the first color photosensitive pixels corresponding to the multiple first color sub-filters in the first filter group, combining and reading out the third pixel value of the third color photosensitive pixels corresponding to the multiple third color sub-filters in the second filter group including the first color sub-filter, and combining and reading out the fourth pixel value of the second color photosensitive pixels corresponding to the multiple second color sub-filters in the second filter group, some first color photosensitive pixels, third color photosensitive pixels and second color photosensitive pixels can be separated to form a fifth combined image, and the image size can be reduced by pixel merging to reduce processing power consumption. According to the fourth pixel value read out by merging the second color photosensitive pixels corresponding to the plurality of second color sub-filters in the first filter group, the second pixel value read out by merging the first color photosensitive pixels corresponding to the plurality of first color sub-filters in the second filter group, and the third pixel value read out by merging the third color photosensitive pixels corresponding to the plurality of third color sub-filters in the second filter group including the second color sub-filter, part of the first color photosensitive pixels, the third color photosensitive pixels and the second color photosensitive pixels can be separated to form a sixth merged image, so that the coordinate position of the second color photosensitive pixels in the sixth merged image and the first color photosensitive pixels in the fifth merged image are consistent. The third target image is obtained based on the full-color image, the fifth merged image and the sixth merged image, and the full-color channel information can be integrated into the image to increase the overall light input, so that a third target image with more information and clearer detail resolution can be generated. In addition, the size of the generated third target image is reduced and the power consumption required to generate the image is low by merging and reading out the plurality of pixels corresponding to the filter group.

In one embodiment, an image generation method is provided, which is applied to an image sensor of an electronic device, wherein the image sensor includes a filter array and a pixel array, the filter array includes a minimum repeating unit, the minimum repeating unit includes a plurality of filter groups, each filter group includes a color filter and a full-color filter, the amount of light transmitted by the full-color filter is greater than the amount of light transmitted by the color filter, the color filter and the full-color filter each include four sub-filters, the plurality of filter groups include at least a first filter group, and at least one color filter in the first filter group includes a first color sub-filter and a first color sub-filter. two color sub-filters; the color filter in the second filter group includes a third color sub-filter, and the color filter of the second filter group also includes a first color sub-filter or a second color sub-filter; the first color photosensitive pixel of the pixel array is arranged correspondingly to the first color sub-filter of the filter array, the second color photosensitive pixel of the pixel array is arranged correspondingly to the second color sub-filter of the filter array, and the third color photosensitive pixel of the pixel array is arranged correspondingly to the third color sub-filter of the filter array, and the pixel array is configured to receive light passing through the filter array to generate an electrical signal;

The method includes:

In the first resolution mode, a first merged image is obtained based on a first pixel value read out by merging a plurality of panchromatic pixels corresponding to the panchromatic filters in the filter group, a second pixel value read out by merging a plurality of first color photosensitive pixels corresponding to the color filters, and a third pixel value read out by merging a plurality of third color photosensitive pixels corresponding to the color filters; a second merged image is obtained based on a fourth pixel value read out by merging a plurality of second color photosensitive pixels corresponding to the color filters; the pixel values in the first merged image are traversed, and when the traversed pixel value is the second pixel value, a fourth pixel value at the same position as the second pixel value is read from the second merged image, and the read fourth pixel value is adjusted to be adjacent to the second pixel value, until all pixel values in the first merged image are traversed, thereby obtaining a first target image.

Optionally, in the first resolution mode, a first merged image is obtained according to a first pixel value merged and read out from a plurality of full-color pixels corresponding to the full-color filters in the filter group, a second pixel value merged and read out from a plurality of first color photosensitive pixels corresponding to a plurality of first color sub-filters in the first filter group, a third pixel value merged and read out from a plurality of third color photosensitive pixels corresponding to a plurality of third color sub-filters in the second filter group including the first color sub-filters, and a fourth pixel value merged and read out from a plurality of second color photosensitive pixels corresponding to a plurality of second color sub-filters in the second filter group; a second merged image is obtained according to a fourth pixel value merged and read out from a plurality of second color photosensitive pixels corresponding to a plurality of second color sub-filters in the first filter group, a second pixel value merged and read out from a plurality of first color photosensitive pixels corresponding to a plurality of first color sub-filters in the second filter group, and a third pixel value merged and read out from a plurality of third color photosensitive pixels corresponding to a plurality of third color sub-filters in the second filter group including the second color sub-filters; and a first target image is obtained based on the first merged image and the second merged image.

Optionally, in the second resolution mode, a first merged image is obtained based on a first pixel value obtained by merging and reading a plurality of panchromatic pixels corresponding to the panchromatic filters in the filter group, a second pixel value obtained by merging and reading a plurality of first color photosensitive pixels corresponding to the color filters, and a third pixel value obtained by merging and reading a plurality of third color photosensitive pixels corresponding to the color filters; the resolution corresponding to the second resolution mode is smaller than the resolution corresponding to the first resolution mode; a second merged image is obtained based on a fourth pixel value obtained by merging and reading a plurality of second color photosensitive pixels corresponding to the color filters; a full-color merged image is obtained based on a fifth pixel value obtained by merging and reading a plurality of panchromatic pixels in the ninth diagonal direction in the first merged image; a sixth pixel value obtained by merging and reading a plurality of first color photosensitive pixels in the tenth diagonal direction in the first merged image, and a plurality of third color photosensitive pixels in the tenth diagonal direction; The seventh pixel value read out by merging the photosensitive pixels is obtained to obtain a third merged image, and the ninth diagonal direction is different from the tenth diagonal direction; based on the eighth pixel value read out by merging multiple second color photosensitive pixels in the tenth diagonal direction in the second merged image, a fourth merged image is obtained; the pixels in the third merged image are traversed; when the traversed pixel value is the sixth pixel value or the seventh pixel value, the fifth pixel value corresponding to the traversed pixel value is read from the full-color merged image, and the read fifth pixel value is adjusted to be adjacent to the traversed pixel value; when the traversed pixel value is the sixth pixel value, the eighth pixel value with the same sixth pixel position is read from the fourth merged image, and the read eighth pixel value is adjusted to be adjacent to the sixth pixel value, until the pixel values in the full-color merged image, the third merged image and the fourth merged image are all read, and the second target image is obtained.

Optionally, in the second resolution mode, a first merged image is obtained according to a first pixel value read out by merging a plurality of full-color pixels corresponding to the full-color filters in the filter group, a second pixel value read out by merging a first color photosensitive pixel corresponding to a plurality of first color sub-filters in the first filter group, a third pixel value read out by merging a third color photosensitive pixel corresponding to a plurality of third color sub-filters in the second filter group including the first color sub-filters, and a fourth pixel value read out by merging a second color photosensitive pixel corresponding to a plurality of second color sub-filters in the second filter group; a first merged image is obtained according to a fourth pixel value read out by merging a second color photosensitive pixel corresponding to a plurality of second color sub-filters in the first filter group, a second pixel value read out by merging a first color photosensitive pixel corresponding to a plurality of first color sub-filters in the second filter group, and a third pixel value read out by merging a third color photosensitive pixel corresponding to a plurality of third color sub-filters in the second filter group including the second color sub-filters. Based on the third pixel value read out by merging the pixels, a second merged image is obtained; based on the fifth pixel value read out by merging the multiple full-color pixels in the ninth diagonal direction in the first merged image, the sixth pixel value read out by merging the multiple first-color photosensitive pixels in the tenth diagonal direction, the seventh pixel value read out by merging the multiple third-color photosensitive pixels in the tenth diagonal direction, and the eighth pixel value read out by merging the multiple second-color photosensitive pixels in the tenth diagonal direction, a full-color merged image and a third merged image are obtained respectively; based on the sixth pixel value read out by merging the multiple first-color photosensitive pixels in the tenth diagonal direction, the seventh pixel value read out by merging the multiple third-color photosensitive pixels in the tenth diagonal direction, and the eighth pixel value read out by merging the multiple second-color photosensitive pixels in the tenth diagonal direction in the second merged image, a fourth merged image is obtained; based on the full-color merged image, the third merged image and the fourth merged image, a second target image is obtained.

Optionally, in the second resolution mode, a full-color image is obtained based on a first pixel value read out by merging a plurality of full-color pixels corresponding to the filter group; a fifth merged image is obtained based on a second pixel value read out by merging a plurality of first-color photosensitive pixels corresponding to the filter group and a third pixel value read out by merging a plurality of third-color photosensitive pixels corresponding to the filter group; a sixth merged image is obtained based on a fourth pixel value read out by merging a plurality of second-color photosensitive pixels corresponding to the filter group; and a third target image is obtained based on the full-color image, the fifth merged image and the sixth merged image.

Optionally, in the second resolution mode, a full-color image is obtained according to a first pixel value read out by merging multiple full-color pixels corresponding to the filter group; a fifth merged image is obtained according to a second pixel value read out by merging first color photosensitive pixels corresponding to multiple first color sub-filters in the first filter group, a third pixel value read out by merging third color photosensitive pixels corresponding to multiple third color sub-filters in the second filter group including the first color sub-filters, and a fourth pixel value read out by merging second color photosensitive pixels corresponding to multiple second color sub-filters in the second filter group; a sixth merged image is obtained according to a fourth pixel value read out by merging second color photosensitive pixels corresponding to multiple second color sub-filters in the first filter group, a second pixel value read out by merging first color photosensitive pixels corresponding to multiple first color sub-filters in the second filter group, and a third pixel value read out by merging third color photosensitive pixels corresponding to multiple third color sub-filters in the second filter group including the second color sub-filters; and a third target image is obtained based on the full-color image, the fifth merged image and the sixth merged image.

In this embodiment, an image sensor structure supporting outputs of two resolutions is provided, thereby providing image output modes of two resolutions, which can be adapted to different application scenarios.

The first resolution mode is used in scenarios with relatively high resolution requirements. The full-color channel information can be integrated into the image by merging the first pixel value read out by merging the multiple full-color pixels corresponding to the full-color filters in the filter group, the second pixel value read out by merging the multiple first-color photosensitive pixels corresponding to the color filters, and the third pixel value read out by merging the multiple third-color photosensitive pixels corresponding to the color filters, thereby increasing the overall amount of light input, so that a first merged image with more information and clearer detail resolution can be generated. The second merged image is obtained according to the fourth pixel value read out by merging the multiple second-color photosensitive pixels corresponding to the color filters, so that the size of the second merged image is consistent with that of the first merged image, thereby ensuring that the position of each second-color photosensitive pixel in the second merged image is the same as the position of each first-color photosensitive pixel in the first merged image, that is, ensuring that the position of each fourth pixel value in the second merged image is the same as the position of each second pixel value in the first merged image.

The pixel values in the first merged image are traversed to adjust the fourth pixel value in the second merged image to be adjacent to the second pixel value at the same position, so that the first color photosensitive pixels and the second color photosensitive pixels are mixedly arranged, thereby improving the color resolution capability, so that each row and each column of the generated first target image has the first color photosensitive pixels, the second color photosensitive pixels and the third color photosensitive pixels, that is, each row and each column of the target image has RGB pixels, which can effectively reduce the risk of false color.

Or in the first resolution mode, according to the first pixel value read out by merging multiple panchromatic pixels corresponding to the panchromatic filters in the filter group, the second pixel value read out by merging multiple first color photosensitive pixels corresponding to the color filters, the third pixel value read out by merging multiple third color photosensitive pixels corresponding to the color filters, and the fourth pixel value read out by merging multiple second color photosensitive pixels corresponding to the color filters, the first merged image obtained by merging and reading includes pixels of all colors, which can improve the resolution of the color channel. According to the second pixel value read out by merging multiple first color photosensitive pixels corresponding to the color filters, the third pixel value read out by merging multiple third color photosensitive pixels corresponding to the color pixels, and the fourth pixel value read out by merging multiple second color photosensitive pixels corresponding to the color filters, the second merged image is obtained, so that the pixels of each color in the first target image generated according to the first merged image and the second merged image are more evenly distributed, effectively improving the clarity of the image.

The second resolution mode is used in scenes with lower resolution requirements such as night scene shooting. The first merged image is obtained based on the first pixel value read out by merging multiple panchromatic pixels corresponding to the panchromatic filter in the filter group, the second pixel value read out by merging multiple color pixels corresponding to the color filter, and the third pixel value read out by merging multiple third color photosensitive pixels corresponding to the color filter. The second merged image is obtained based on the fourth pixel value read out by merging multiple second color photosensitive pixels corresponding to the color filter. The second color photosensitive pixels can be separated as a separate image and the image resolution can be reduced. The panchromatic merged image is obtained based on the fifth pixel value read out by merging multiple panchromatic pixels in the ninth diagonal direction in the first merged image. The panchromatic pixels can be separated from the first merged image while reducing the image size to reduce power consumption. Based on the sixth pixel value read out by merging multiple first color photosensitive pixels in the tenth diagonal direction in the first merged image, and the seventh pixel value read out by merging multiple third color photosensitive pixels in the tenth diagonal direction, a third merged image including first color photosensitive pixels and third color photosensitive pixels is obtained, and the all-in-one pixel readout method makes the generated image less noisy. Based on the eighth pixel value read out by merging multiple second color photosensitive pixels in the eighth diagonal direction in the second merged image, the size of the fourth merged image is kept consistent with the size of the third merged image and the full-color merged image. The pixels in the third merged image are traversed, and when the traversed pixel value is the sixth pixel value or the seventh pixel value, the fifth pixel value corresponding to the traversed pixel value is read from the full-color merged image, and the read fifth pixel value is adjusted to be adjacent to the traversed pixel value, so that full-color pixels are introduced into the image, and the amount of light entering is increased. When the traversed pixel value is the sixth pixel value, the eighth pixel value at the same position as the sixth pixel is read from the fourth merged image, and the read eighth pixel value is adjusted to be adjacent to the sixth pixel value, so that the first color photosensitive pixel and the second color photosensitive pixel are mixed and arranged, and the color resolution capability is improved, so that each row and each column of the generated second target image has the first color photosensitive pixel, the second color photosensitive pixel and the second color photosensitive pixel, that is, each row and each column of the target image has RGB pixels, which can effectively reduce the risk of false color. In addition, the resolution of the obtained second target image is further reduced, and the full-color pixel has a higher signal-to-noise ratio, and the frame rate of the image is high, thereby achieving the image processing effect of lower power consumption and better signal-to-noise ratio of the secondary pixel merging output.

Alternatively, in the second resolution mode, a first merged image is obtained based on a first pixel value merged and read out by a plurality of panchromatic pixels corresponding to the panchromatic filters in the filter group, a second pixel value merged and read out by a plurality of color pixels corresponding to the color filters, a third pixel value merged and read out by a plurality of third color photosensitive pixels corresponding to the color filters, and a fourth pixel value merged and read out by a plurality of third color photosensitive pixels corresponding to the color filters, and a second merged image is obtained based on a second pixel value merged and read out by a plurality of first color photosensitive pixels corresponding to the color filters, a third pixel value merged and read out by a plurality of third color photosensitive pixels corresponding to the color filters, and a fourth pixel value merged and read out by a plurality of second color photosensitive pixels corresponding to the color filters, and the first color photosensitive pixels, second color photosensitive pixels and third color photosensitive pixels corresponding to each color filter can be separated to form two merged images, so that the photosensitive pixels of each color in the two merged images are in the same position in each merged image.

Based on the fifth pixel value read out by merging multiple full-color pixels in the ninth diagonal direction in the first merged image, a full-color merged image is obtained, which can separate the full-color pixels from the first merged image and reduce the size of the image to reduce power consumption. Based on the sixth pixel value read out by merging multiple first-color photosensitive pixels in the tenth diagonal direction in the first merged image, the seventh pixel value read out by merging multiple third-color photosensitive pixels in the tenth diagonal direction, and the eighth pixel value read out by merging multiple second-color photosensitive pixels in the tenth diagonal direction, a third merged image including first-color photosensitive pixels, second-color photosensitive pixels, and third-color photosensitive pixels is obtained, so that the distribution of RGB pixels in the image is more uniform. Based on the sixth pixel value read out by merging multiple first-color photosensitive pixels in the tenth diagonal direction, the seventh pixel value read out by merging multiple third-color photosensitive pixels in the tenth diagonal direction, and the eighth pixel value read out by merging multiple second-color photosensitive pixels in the tenth diagonal direction in the second merged image, a fourth merged image is obtained, so that the size of the fourth merged image is consistent with the size of the third merged image and the full-color merged image. Based on the full-color merged image, the third merged image, and the fourth merged image, different first color photosensitive pixels, second color photosensitive pixels, and third color photosensitive pixels can be mixed and arranged, so that the RGB pixels in the generated second target image are more evenly distributed and the image quality is higher. In addition, the resolution of the obtained second target image is further reduced, and the full-color pixels have a higher signal-to-noise ratio, and the frame rate of the image is high, thereby achieving an image processing effect of lower power consumption and better signal-to-noise ratio of the secondary pixel merging output.

It should be understood that, although the various steps in the flow charts of Figures 10-14 are displayed in sequence according to the indication of the arrows, these steps are not necessarily performed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be performed in other orders. Moreover, at least a portion of the steps in Figures 10-14 may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these sub-steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a portion of other steps or sub-steps or stages of other steps.

FIG15 is a block diagram of an image generating device according to an embodiment. As shown in FIG15 , the image generating device 1500 is applied to an image sensor, the image sensor includes a filter array and a pixel array, the filter array includes a minimum repeating unit, the minimum repeating unit includes a plurality of filter groups, each filter group includes a color filter and a panchromatic filter, the amount of light transmitted by the panchromatic filter is greater than the amount of light transmitted by the color filter, the color filter and the panchromatic filter each include four sub-filters, the plurality of filter groups include a first filter group and a second filter group, at least one color filter in the first filter group includes a sub-filter. a color sub-filter and a second color sub-filter; the color filter in the second filter group includes at least a third color sub-filter; the pixel array includes a plurality of pixels, the first color photosensitive pixel of the pixel array is arranged correspondingly to the first color sub-filter of the filter array, the second color photosensitive pixel of the pixel array is arranged correspondingly to the second color sub-filter of the filter array, the third color photosensitive pixel of the pixel array is arranged correspondingly to the third color sub-filter of the filter array, and the pixel array is configured to receive light passing through the filter array to generate an electrical signal;

The image generating device 1500 comprises:

The first merging module 1502 is used to obtain a first merged image in the first resolution mode based on the first pixel value read out by merging a plurality of panchromatic pixels corresponding to the panchromatic filters in the filter group, the second pixel value read out by merging a plurality of first color photosensitive pixels corresponding to the color filters, and the third pixel value read out by merging a plurality of third color photosensitive pixels corresponding to the color filters.

The second merging module 1504 is used to merge the fourth pixel values read out according to the multiple second color photosensitive pixels corresponding to the color filter to obtain a second merged image.

The image generating module 1506 is configured to obtain a first target image based on the first merged image and the second merged image.

In the present embodiment, in the first resolution mode, the full-color channel information can be integrated into the image by merging the first pixel value read out by merging the multiple full-color pixels corresponding to the full-color filter in the filter group, the second pixel value read out by merging the multiple first-color photosensitive pixels corresponding to the color filter, and the third pixel value read out by merging the multiple third-color photosensitive pixels corresponding to the color filter, so as to increase the overall amount of light input, so as to generate a first merged image with more information and clearer detail resolution. The second merged image is obtained according to the fourth pixel value read out by merging the multiple second-color photosensitive pixels corresponding to the color filter, and the first target image is obtained based on the first merged image and the second merged image, so that the first color photosensitive pixels in the first merged image and the second color photosensitive pixels in the second merged image can be mixed and arranged, so that the color of the generated first target image is clearer. In addition, the pixel merging readout reduces the size of the generated first target image and the power consumption required to generate the image is low.

In one embodiment, the image generation module 1506 is also used to traverse the pixel values in the first merged image, and when the traversed pixel value is the second pixel value, read a fourth pixel value at the same position as the second pixel value from the second merged image, and adjust the read fourth pixel value to be adjacent to the second pixel value, until all the pixel values in the first merged image are traversed, thereby obtaining the first target image.

In this embodiment, the pixel values in the first merged image are traversed, and when the traversed pixel value is the second pixel value, the fourth pixel value at the same position as the second pixel value is read from the second merged image, and the read fourth pixel value is adjusted to be adjacent to the second pixel value, so that the first color photosensitive pixels and the second color photosensitive pixels are mixed and arranged, thereby improving the color resolution capability, so that each row and each column of the generated first target image has the first color photosensitive pixels, the second color photosensitive pixels and the third color photosensitive pixels, that is, each row and each column of the target image has RGB pixels, which can effectively reduce the risk of false color.

In one embodiment, the color filter of the second filter group further includes a first color sub-filter or a second color sub-filter; the first merging module 1502 is further used to obtain a first merged image in the first resolution mode by merging and reading a first pixel value corresponding to a plurality of panchromatic pixels in the filter group, merging and reading a second pixel value corresponding to a plurality of first color sub-filters in the first filter group, merging and reading a third pixel value corresponding to a plurality of third color sub-filters in the second filter group including the first color sub-filter, and merging and reading a fourth pixel value corresponding to a plurality of second color sub-filters in the second filter group;

The second merging module 1504 is also used to obtain a second merged image based on a fourth pixel value read out by merging the second color photosensitive pixels corresponding to multiple second color sub-filters in the first filter group, a second pixel value read out by merging the first color photosensitive pixels corresponding to multiple first color sub-filters in the second filter group, and a third pixel value read out by merging the third color photosensitive pixels corresponding to multiple third color sub-filters in the second filter group including the second color sub-filters.

In this embodiment, the color filter 234 of the second filter group 232 includes a third color sub-filter 2343 and also includes a first color sub-filter 2341 or a second color sub-filter 2342. In the first resolution mode, a first pixel value is read out by merging a plurality of full-color pixels corresponding to the full-color filters in the filter group, a second pixel value is read out by merging a plurality of first color photosensitive pixels corresponding to the plurality of first color sub-filters in the first filter group, a third pixel value is read out by merging a plurality of third color photosensitive pixels corresponding to the plurality of third color sub-filters in the second filter group including the first color sub-filter, and a fourth pixel value is read out by merging a plurality of second color photosensitive pixels corresponding to the plurality of second color sub-filters in the second filter group, so that the first merged image obtained by the merged reading contains pixels of all colors, thereby improving the resolution capability of the color channel. A second merged image is obtained based on the second pixel value read out by merging multiple first color photosensitive pixels corresponding to the color filter, the third pixel value read out by merging multiple third color photosensitive pixels corresponding to the color pixel, and the fourth pixel value read out by merging multiple second color photosensitive pixels corresponding to the color filter, so that the pixel distribution of each color in the first target image generated according to the first merged image and the second merged image is more uniform, effectively improving the clarity of the image.

In one embodiment, the apparatus further comprises a third merging module;

The first merging module 1502 is further configured to obtain a first merged image according to a first pixel value read out by merging a plurality of panchromatic pixels corresponding to the panchromatic filters in the filter group, a second pixel value read out by merging a plurality of first color photosensitive pixels corresponding to the color filters, and a third pixel value read out by merging a plurality of third color photosensitive pixels corresponding to the color filters in a second resolution mode; the resolution corresponding to the second resolution mode is smaller than the resolution corresponding to the first resolution mode;

The second merging module 1504 is further configured to merge the read fourth pixel values according to the plurality of second color photosensitive pixels corresponding to the color filter to obtain a second merged image;

a third merging module, configured to obtain a full-color merged image based on a fifth pixel value obtained by merging and reading a plurality of full-color pixels in a ninth diagonal direction in the first merged image; and obtain a third merged image based on a sixth pixel value obtained by merging and reading a plurality of first-color light-sensitive pixels in a tenth diagonal direction in the first merged image and a seventh pixel value obtained by merging and reading a plurality of third-color light-sensitive pixels in a tenth diagonal direction, wherein the ninth diagonal direction is different from the tenth diagonal direction;

The third merging module is further used to obtain a fourth merged image by merging the eighth pixel value read out based on a plurality of second color light-sensitive pixels in a tenth diagonal direction in the second merged image;

The image generating module 1506 is further configured to obtain a second target image based on the full-color merged image, the third merged image and the fourth merged image.

In this embodiment, in the second resolution mode, a first merged image is obtained according to the first pixel value read out by merging multiple panchromatic pixels corresponding to the panchromatic filters in the filter group, the second pixel value read out by merging multiple color pixels corresponding to the color filters, and the third pixel value read out by merging multiple third color photosensitive pixels corresponding to the color filters, and a second merged image is obtained according to the fourth pixel value read out by merging multiple second color photosensitive pixels corresponding to the color filters, so that the second color photosensitive pixels can be separated as a separate image and the resolution of the image can be reduced. A panchromatic merged image is obtained based on the fifth pixel value read out by merging multiple panchromatic pixels in the ninth diagonal direction in the first merged image, so that the panchromatic pixels can be separated from the first merged image while reducing the size of the image to reduce power consumption. A third merged image including the first color photosensitive pixels and the third color photosensitive pixels is obtained based on the sixth pixel value read out by merging multiple first color photosensitive pixels in the tenth diagonal direction in the first merged image, and the seventh pixel value read out by merging multiple third color photosensitive pixels in the tenth diagonal direction, so that the generated image has less noise. Based on the eighth pixel value read out by merging multiple second color photosensitive pixels in the tenth diagonal direction in the second merged image, the size of the fourth merged image is kept consistent with the size of the third merged image and the full-color merged image. Based on the full-color merged image, the third merged image and the fourth merged image, different first color photosensitive pixels and second color photosensitive pixels can be mixed and arranged, so that the RGB pixels in the generated second target image are more evenly distributed and the image quality is higher. In addition, the resolution of the obtained second target image is further reduced, and the full-color pixels have a higher signal-to-noise ratio, and the frame rate of the image is high, thereby achieving an image processing effect with lower power consumption and better signal-to-noise ratio of the secondary pixel merging output.

In one embodiment, the image generation module 1506 is also used to traverse the pixels in the third merged image; when the traversed pixel value is the sixth pixel value or the seventh pixel value, read the fifth pixel value corresponding to the traversed pixel value from the full-color merged image, and adjust the read fifth pixel value to be adjacent to the traversed pixel value; when the traversed pixel value is the sixth pixel value, read the eighth pixel value at the same position as the sixth pixel from the fourth merged image, and adjust the read eighth pixel value to be adjacent to the sixth pixel value, until the pixel values in the full-color merged image, the third merged image and the fourth merged image are all read, and the second target image is obtained.

In this embodiment, the pixels in the third merged image are traversed, and when the traversed pixel value is the sixth pixel value or the seventh pixel value, the fifth pixel value corresponding to the traversed pixel value is read from the full-color merged image, and the read fifth pixel value is adjusted to be adjacent to the traversed pixel value, so that full-color pixels are introduced into the image, and the amount of light entering is increased. When the traversed pixel value is the sixth pixel value, the eighth pixel value at the same position as the sixth pixel is read from the fourth merged image, and the read eighth pixel value is adjusted to be adjacent to the sixth pixel value, so that the first color photosensitive pixels and the second color photosensitive pixels are mixed and arranged, and the color resolution ability is improved, so that each row and each column of the generated second target image has the first color photosensitive pixels, the second color photosensitive pixels, and the second color photosensitive pixels, that is, each row and each column of the target image has RGB pixels, which can effectively reduce the risk of false colors.

In one embodiment, the color filter of the second filter group further includes a first color sub-filter or a second color sub-filter; the first merging module 1502 is further used to obtain a first merged image in the second resolution mode by merging and reading a first pixel value according to a plurality of panchromatic pixels corresponding to the panchromatic filters in the filter group, a second pixel value according to a first color photosensitive pixel corresponding to a plurality of first color sub-filters in the first filter group, a third pixel value according to a third color photosensitive pixel corresponding to a plurality of third color sub-filters in the second filter group including the first color sub-filter, and a fourth pixel value according to a second color photosensitive pixel corresponding to a plurality of second color sub-filters in the second filter group;

The second merging module 1504 is further configured to obtain a second merged image based on a fourth pixel value read out by merging the second color photosensitive pixels corresponding to the plurality of second color sub-filters in the first filter group, a second pixel value read out by merging the first color photosensitive pixels corresponding to the plurality of first color sub-filters in the second filter group, and a third pixel value read out by merging the third color photosensitive pixels corresponding to the plurality of third color sub-filters in the second filter group including the second color sub-filter;

a third merging module, further configured to generate a third merged image based on a sixth pixel value merged and read out by a plurality of first color light-sensitive pixels in a tenth diagonal direction in the first merged image, a seventh pixel value merged and read out by a plurality of third color light-sensitive pixels in a tenth diagonal direction, and an eighth pixel value merged and read out by a plurality of second color light-sensitive pixels in a tenth diagonal direction;

The third merging module is also used to obtain a fourth merged image based on the sixth pixel value read out by merging multiple first color photosensitive pixels in the tenth diagonal direction, the seventh pixel value read out by merging multiple third color photosensitive pixels in the tenth diagonal direction, and the eighth pixel value read out by merging multiple second color photosensitive pixels in the tenth diagonal direction in the second merged image.

In this embodiment, in the second resolution mode, a first pixel value is read out according to a combination of a plurality of full-color pixels corresponding to the full-color filters in the filter group, a second pixel value is read out according to a combination of a first color photosensitive pixel corresponding to a plurality of first color sub-filters in the first filter group, a third pixel value is read out according to a combination of a third color photosensitive pixel in the second filter group including the first color sub-filter, and a fourth pixel value is read out according to a combination of a third color photosensitive pixel corresponding to a plurality of second color sub-filters in the second filter group to obtain a first merged image, and a plurality of second color sub-filters in the first filter group are combined to form a first merged image. The second merged image is obtained by combining and reading the fourth pixel value of the second color photosensitive pixels corresponding to the multiple first color sub-filters in the second filter group, and combining and reading the third pixel value of the third color photosensitive pixels corresponding to the multiple third color sub-filters in the second filter group including the second color sub-filter. The first color photosensitive pixels, the second color photosensitive pixels and the third color photosensitive pixels corresponding to each color filter can be separated to form two merged images, so that the photosensitive pixels of each color in the two merged images are in the same position in each merged image.

The full-color merged image is obtained based on the fifth pixel value read out by merging multiple full-color pixels in the ninth diagonal direction in the first merged image, and the full-color pixels can be separated from the first merged image while reducing the size of the image to reduce power consumption.

Based on the sixth pixel value read out by merging multiple first color photosensitive pixels in the tenth diagonal direction in the first merged image, the seventh pixel value read out by merging multiple third color photosensitive pixels in the tenth diagonal direction, and the eighth pixel value read out by merging multiple second color photosensitive pixels in the tenth diagonal direction, a third merged image including first color photosensitive pixels, second color photosensitive pixels and third color photosensitive pixels is obtained, so that the distribution of RGB pixels in the image is more uniform. Based on the sixth pixel value read out by merging multiple first color photosensitive pixels in the tenth diagonal direction, the seventh pixel value read out by merging multiple third color photosensitive pixels in the tenth diagonal direction, and the eighth pixel value read out by merging multiple second color photosensitive pixels in the tenth diagonal direction in the second merged image, a fourth merged image is obtained, so that the size of the fourth merged image is consistent with that of the third merged image and the full-color merged image. Based on the full-color merged image, the third merged image and the fourth merged image, different first color photosensitive pixels, second color photosensitive pixels and third color photosensitive pixels can be mixed and arranged, so that the distribution of RGB pixels in the generated second target image is more uniform and the image quality is higher. Moreover, the resolution of the obtained second target image is further reduced, and the full-color pixels have a higher signal-to-noise ratio, and the image frame rate is high, thereby achieving an image processing effect with lower power consumption and better signal-to-noise ratio of the secondary pixel merging output.

In one embodiment, the first merging module 1502 is further configured to obtain a full-color image in the second resolution mode according to a first pixel value read out by merging a plurality of full-color pixels corresponding to the filter group; obtain a fifth merged image according to a second pixel value read out by merging a plurality of first-color light-sensitive pixels corresponding to the filter group and a third pixel value read out by merging a plurality of third-color light-sensitive pixels corresponding to the filter group;

The second merging module 1504 is further configured to merge the read fourth pixel values according to the plurality of second color light-sensitive pixels corresponding to the filter group to obtain a sixth merged image;

The image generating module 1506 is further configured to obtain a third target image based on the full-color image, the fifth combined image and the sixth combined image.

In this embodiment, in the second resolution mode, a full-color image can be separated by merging the first pixel value read out by the plurality of full-color pixels corresponding to the filter group; according to the second pixel value read out by merging the plurality of first-color photosensitive pixels corresponding to the filter group, and the third pixel value read out by merging the plurality of third-color photosensitive pixels corresponding to the filter group, the first-color photosensitive pixels and the third-color photosensitive pixels can be separated to form a fifth merged image, which can quickly and effectively reduce the image size. According to the fourth pixel value read out by merging the plurality of second-color photosensitive pixels corresponding to the filter group, the second-color photosensitive pixels can be separated to form a sixth merged image, so that the coordinate positions of the second-color photosensitive pixels in the sixth merged image and the first-color photosensitive pixels in the fifth merged image are consistent. The third target image is obtained based on the full-color image, the fifth merged image, and the sixth merged image, so that the full-color channel information can be integrated into the image, the overall amount of light input can be increased, and the third target image with more information and clearer detail resolution can be generated. And the size of the generated third target image is reduced by merging and reading the plurality of pixels corresponding to the filter group, and the power consumption required to generate the image is low.

In one embodiment, the color filter of the second filter group further includes a first color sub-filter or a second color sub-filter; the first merging module 1502 is further used to obtain a fifth merged image according to a second pixel value merged and read out by first color photosensitive pixels corresponding to a plurality of first color sub-filters in the first filter group, a third pixel value merged and read out by third color photosensitive pixels corresponding to a plurality of third color sub-filters in the second filter group including the first color sub-filter, and a fourth pixel value merged and read out by second color photosensitive pixels corresponding to a plurality of second color sub-filters in the second filter group;

The second merging module 1504 is also used to obtain a sixth merged image based on a fourth pixel value read out by merging the second color photosensitive pixels corresponding to multiple second color sub-filters in the first filter group, a second pixel value read out by merging the first color photosensitive pixels corresponding to multiple first color sub-filters in the second filter group, and a third pixel value read out by merging the third color photosensitive pixels corresponding to multiple third color sub-filters in the second filter group including the second color sub-filter.

In this embodiment, in the second resolution mode, by combining and reading out the second pixel value of the first color photosensitive pixels corresponding to the multiple first color sub-filters in the first filter group, combining and reading out the third pixel value of the third color photosensitive pixels corresponding to the multiple third color sub-filters in the second filter group including the first color sub-filter, and combining and reading out the fourth pixel value of the second color photosensitive pixels corresponding to the multiple second color sub-filters in the second filter group, some first color photosensitive pixels, third color photosensitive pixels and second color photosensitive pixels can be separated to form a fifth combined image, and the image size can be reduced by pixel merging to reduce processing power consumption. According to the fourth pixel value read out by merging the second color photosensitive pixels corresponding to the plurality of second color sub-filters in the first filter group, the second pixel value read out by merging the first color photosensitive pixels corresponding to the plurality of first color sub-filters in the second filter group, and the third pixel value read out by merging the third color photosensitive pixels corresponding to the plurality of third color sub-filters in the second filter group including the second color sub-filter, part of the first color photosensitive pixels, the third color photosensitive pixels and the second color photosensitive pixels can be separated to form a sixth merged image, so that the coordinate position of the second color photosensitive pixels in the sixth merged image and the first color photosensitive pixels in the fifth merged image are consistent. The third target image is obtained based on the full-color image, the fifth merged image and the sixth merged image, and the full-color channel information can be integrated into the image to increase the overall light input, so that a third target image with more information and clearer detail resolution can be generated. In addition, the size of the generated third target image is reduced and the power consumption required to generate the image is low by merging and reading out the plurality of pixels corresponding to the filter group.

The division of the modules in the above-mentioned image generating device is only for illustration. In other embodiments, the image generating device may be divided into different modules as needed to complete all or part of the functions of the above-mentioned image generating device.

FIG16 is a schematic diagram of the internal structure of an electronic device in an embodiment. As shown in FIG16 , the electronic device includes a processor and a memory connected via a system bus. The processor is used to provide computing and control capabilities to support the operation of the entire electronic device. The memory may include a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The computer program can be executed by the processor to implement an image generation method provided in each of the following embodiments. The internal memory provides a cached operating environment for the operating system computer program in the non-volatile storage medium. The electronic device may be a mobile phone, a tablet computer, a personal digital assistant, a wearable device, etc.

The implementation of each module in the image generation device provided in the embodiment of the present application can be in the form of a computer program. The computer program can be run on a terminal or a server. The program modules constituted by the computer program can be stored in the memory of the terminal or the server. When the computer program is executed by the processor, the steps of the method described in the embodiment of the present application are implemented.

The embodiment of the present application also provides a computer-readable storage medium, one or more non-volatile computer-readable storage media containing computer-executable instructions, which, when executed by one or more processors, enable the processors to perform the steps of the image generation method.

A computer program product comprising instructions, when executed on a computer, causes the computer to perform an image generation method.

Any reference to memory, storage, database or other media used in the embodiments of the present application may include non-volatile and/or volatile memory. Suitable non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM), which is used as an external cache memory. As an illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM) and memory bus dynamic RAM (RDRAM).

The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims (26)

1. An image sensor comprising a filter array and a pixel array, wherein the filter array comprises a minimal repeating unit, the minimal repeating unit comprises a plurality of filter sets, each filter set comprises a color filter and a full-color filter, the amount of light entering transmitted by the full-color filter is greater than the amount of light entering transmitted by the color filter, the color filter and the full-color filter each comprise 4 sub-filters, the plurality of filter sets at least comprise a first filter set, and at least one of the color filters in the first filter set comprises a first color sub-filter and a second color sub-filter; the pixel array comprises a plurality of pixels, the pixels of the pixel array are arranged corresponding to the sub-filters of the filter array, and the pixel array is configured to receive light rays passing through the filter array to generate electric signals;

The image sensor is configured to obtain a first combined image according to a first pixel value that is read out by combining a plurality of full-color pixels corresponding to the full-color filter in the filter set, a second pixel value that is read out by combining a plurality of first color photosensitive pixels corresponding to the color filter, and a third pixel value that is read out by combining a plurality of third color photosensitive pixels corresponding to the color filter in a first resolution mode; combining the read fourth pixel values according to a plurality of second color photosensitive pixels corresponding to the color filters to obtain a second combined image; the position of the second pixel value in the first combined image is the same as the position of the fourth pixel value in the second combined image; and traversing the pixel values in the first combined image, reading a fourth pixel value which is the same as the second pixel value from the second combined image under the condition that the traversed pixel value is the second pixel value, and adjusting the read fourth pixel value to be adjacent to the second pixel value until the pixel values in the first combined image are traversed, so as to obtain a first target image.

2. The image sensor of claim 1, wherein the image sensor comprises a sensor array, the plurality of filter sets further includes a second filter set, the color filters in the second filter set at least comprise a third color sub-filter.

3. The image sensor of claim 2, wherein the minimal repeating unit comprises 4 filter sets, the 4 filter sets comprising 2 of the first filter sets and 2 of the second filter sets, the 2 of the first filter sets and 2 of the second filter sets being arranged in a matrix.

4. The image sensor of claim 3, wherein the first filter set is disposed in a first diagonal direction and the second filter set is disposed in a second diagonal direction in the minimal repeating unit, the first diagonal direction being different from the second diagonal direction.

5. The image sensor of any one of claims 1 to 4, wherein in each of the filter sets, the color filters and the full-color filters are arranged in a matrix, the full-color filters are arranged in a third diagonal direction, the color filters are arranged in a fourth diagonal direction, and the third diagonal direction is different from the fourth diagonal direction.

6. The image sensor of claim 1, wherein the first color sub-filter is disposed in one of a fifth diagonal direction or a sixth diagonal direction, and the second color sub-filter is disposed in the other of the fifth diagonal direction or the sixth diagonal direction; the fifth diagonal direction is different from the sixth diagonal direction.

7. The image sensor of claim 2, wherein the third color sub-filter is disposed in a seventh diagonal direction and an eighth diagonal direction, the seventh diagonal direction being different from the eighth diagonal direction.

8. The image sensor of claim 2, wherein the filter set comprises 2 full color filters and 2 color filters, and the minimum repeating unit is 8 rows and 8 columns of 64 sub-filters, and the arrangement manner is that:

wherein w represents a full-color sub-filter, and a, b and c each represent a color sub-filter.

9. The image sensor of claim 2, wherein the filter set comprises 2 full color filters and 2 color filters, and the minimum repeating unit is 8 rows and 8 columns of 64 sub-filters, and the arrangement manner is that:

wherein w represents a full-color sub-filter, and a, b and c each represent a color sub-filter.

10. The image sensor of claim 2, wherein the color filters in the second filter set further comprise a first color sub-filter or a second color sub-filter.

11. The image sensor of claim 10, wherein the second filter set includes two color filters, the sub-filters of the same color in each color filter are diagonally arranged, and the sub-filters of the same color are oppositely arranged in the two color filters.

12. The image sensor of claim 10, wherein the filter set comprises 2 full color filters and 2 color filters, and the minimum repeating unit is 8 rows and 8 columns of 64 sub-filters, and the arrangement manner is that:

wherein w represents a full-color sub-filter, and a, b and c each represent a color sub-filter.

13. The image sensor of claim 10, wherein the filter set comprises 2 full color filters and 2 color filters, and the minimum repeating unit is 8 rows and 8 columns of 64 sub-filters, and the arrangement manner is that:

wherein w represents a full-color sub-filter, and a, b and c each represent a color sub-filter.

14. A camera module comprising a lens and the image sensor of any one of claims 1-13; the image sensor is used for receiving light rays passing through the lens, and the pixels generate electric signals according to the light rays.

15. An electronic device, comprising the camera module of claim 14 and a housing, wherein the camera module is disposed on the housing.

16. An image generation method is applied to an image sensor, and is characterized in that the image sensor comprises an optical filter array and a pixel array, the optical filter array comprises a minimum repeating unit, the minimum repeating unit comprises a plurality of optical filter sets, each optical filter set comprises a color filter and a full-color filter, the light incoming quantity of the full-color filter transmitted by the full-color filter is larger than the light incoming quantity of the color filter transmitted by the full-color filter, the color filter and the full-color filter both comprise 4 sub-optical filters, the optical filter sets comprise a first optical filter set and a second optical filter set, and at least one of the color filters in the first optical filter set comprises a first color sub-optical filter and a second color sub-optical filter; the color filters in the second filter set at least comprise a third color sub-filter; the pixel array comprises a plurality of pixels, a first color photosensitive pixel of the pixel array is arranged corresponding to a first color sub-filter of the filter array, a second color photosensitive pixel of the pixel array is arranged corresponding to a second color sub-filter of the filter array, a third color photosensitive pixel of the pixel array is arranged corresponding to a third color sub-filter of the filter array, and the pixel array is configured to receive light passing through the filter array to generate an electric signal;

The method comprises the following steps:

In a first resolution mode, according to a first pixel value which is read out in a merging way by a plurality of full-color pixels corresponding to the full-color filter in the filter set, a second pixel value which is read out in a merging way by a plurality of first color photosensitive pixels corresponding to the color filter, and a third pixel value which is read out in a merging way by a plurality of third color photosensitive pixels corresponding to the color filter, a first merged image is obtained;

Combining the read fourth pixel values according to a plurality of second color photosensitive pixels corresponding to the color filters to obtain a second combined image; the position of the second pixel value in the first combined image is the same as the position of the fourth pixel value in the second combined image;

And traversing the pixel values in the first combined image, reading a fourth pixel value which is the same as the second pixel value from the second combined image under the condition that the traversed pixel value is the second pixel value, and adjusting the read fourth pixel value to be adjacent to the second pixel value until the pixel values in the first combined image are traversed, so as to obtain a first target image.

17. The method of claim 16, wherein the color filters of the second filter set further comprise a first color sub-filter or a second color sub-filter; in the first resolution mode, according to the first pixel values read out by merging the plurality of full-color pixels corresponding to the full-color filter in the filter set, the second pixel values read out by merging the plurality of first color photosensitive pixels corresponding to the color filter, and the third pixel values read out by merging the plurality of third color photosensitive pixels corresponding to the color filter, a first merged image is obtained, which includes:

In a first resolution mode, according to first pixel values read out by merging a plurality of full-color pixels corresponding to the full-color filters in the filter set, according to second pixel values read out by merging first color photosensitive pixels corresponding to a plurality of first color sub-filters in the first filter set, according to third pixel values read out by merging third color photosensitive pixels corresponding to a plurality of third color sub-filters in the second filter set containing the first color sub-filters, and according to fourth pixel values read out by merging second color photosensitive pixels corresponding to a plurality of second color sub-filters in the second filter set, obtaining a first merged image;

The step of obtaining a second combined image according to the fourth pixel values read out by combining the plurality of second color photosensitive pixels corresponding to the color filter includes:

And according to the fourth pixel value read out by merging the second color photosensitive pixels corresponding to the plurality of second color sub-filters in the first filter set, according to the second pixel value read out by merging the first color photosensitive pixels corresponding to the plurality of first color sub-filters in the second filter set, and according to the third pixel value read out by merging the third color photosensitive pixels corresponding to the plurality of third color sub-filters in the second filter set containing the second color sub-filters, obtaining a second merged image.

18. The method of claim 16, wherein the method further comprises:

in a second resolution mode, according to the first pixel values read out by merging the plurality of full-color pixels corresponding to the full-color filter in the filter set, the second pixel values read out by merging the plurality of first color photosensitive pixels corresponding to the color filter, and the third pixel values read out by merging the plurality of third color photosensitive pixels corresponding to the color filter, a first merged image is obtained; the resolution corresponding to the second resolution mode is smaller than the resolution corresponding to the first resolution mode;

Combining the read fourth pixel values according to a plurality of second color photosensitive pixels corresponding to the color filters to obtain a second combined image;

Obtaining a full-color combined image based on the fifth pixel value read out by combining a plurality of full-color pixels in a ninth diagonal direction in the first combined image;

Obtaining a third combined image based on a sixth pixel value read by combining a plurality of first-color photosensitive pixels in a tenth diagonal direction in the first combined image and a seventh pixel value read by combining a plurality of third-color photosensitive pixels in the tenth diagonal direction, wherein the ninth diagonal direction is different from the tenth diagonal direction;

Obtaining a fourth combined image based on the read eighth pixel values combined by the plurality of second color photosensitive pixels in the tenth diagonal direction in the second combined image;

And obtaining a second target image based on the full-color combined image, the third combined image and the fourth combined image.

19. The method of claim 18, wherein the deriving a second target image based on the full-color combined image, the third combined image, and the fourth combined image comprises:

traversing pixels in the third combined image;

Reading a fifth pixel value corresponding to the traversed pixel value from the full-color combined image and adjusting the read fifth pixel value to be adjacent to the traversed pixel value in the case that the traversed pixel value is the sixth pixel value or the seventh pixel value;

And under the condition that the traversed pixel value is a sixth pixel value, reading an eighth pixel value which is the same as the sixth pixel value from the fourth combined image, and adjusting the read eighth pixel value to be adjacent to the sixth pixel value until the pixel values in the full-color combined image, the third combined image and the fourth combined image are all read, so as to obtain a second target image.

20. The method of claim 19, wherein the color filters of the second filter set further comprise a first color sub-filter or a second color sub-filter; in the second resolution mode, according to the first pixel values read out by merging the plurality of full-color pixels corresponding to the full-color filter in the filter set, the second pixel values read out by merging the plurality of first color photosensitive pixels corresponding to the color filter, and the third pixel values read out by merging the plurality of third color photosensitive pixels corresponding to the color filter, a first merged image is obtained, which includes:

In a second resolution mode, according to first pixel values read out by merging a plurality of full-color pixels corresponding to the full-color filters in the filter set, according to second pixel values read out by merging first color photosensitive pixels corresponding to a plurality of first color sub-filters in the first filter set, according to third pixel values read out by merging third color photosensitive pixels corresponding to a plurality of third color sub-filters in the second filter set containing the first color sub-filters, and according to fourth pixel values read out by merging second color photosensitive pixels corresponding to a plurality of second color sub-filters in the second filter set, obtaining a first merged image;

The step of obtaining a second combined image according to the fourth pixel values read out by combining the plurality of second color photosensitive pixels corresponding to the color filter includes:

According to the fourth pixel value which is read out in a merging way and is corresponding to the second color photosensitive pixels of the plurality of second color sub-filters in the first filter set, according to the second pixel value which is read out in a merging way and is corresponding to the first color photosensitive pixels of the plurality of first color sub-filters in the second filter set, and according to the third pixel value which is read out in a merging way and is corresponding to the third color photosensitive pixels of the plurality of third color sub-filters in the second filter set containing the second color sub-filters, a second merged image is obtained;

The step of obtaining a third combined image based on the sixth pixel value read out by combining the plurality of first color photosensitive pixels in the tenth diagonal direction and the seventh pixel value read out by combining the plurality of third color photosensitive pixels in the tenth diagonal direction in the first combined image includes:

A third combined image based on a sixth pixel value of the combined readout of the plurality of first-color photosensitive pixels in the tenth diagonal direction, a seventh pixel value of the combined readout of the plurality of third-color photosensitive pixels in the tenth diagonal direction, and an eighth pixel value of the combined readout of the plurality of second-color photosensitive pixels in the tenth diagonal direction in the first combined image;

the obtaining a fourth combined image based on the read eighth pixel values combined by the plurality of second color photosensitive pixels in the tenth diagonal direction in the second combined image includes:

And obtaining a fourth combined image based on the sixth pixel value read out by combining the plurality of first-color photosensitive pixels in the tenth diagonal direction, the seventh pixel value read out by combining the plurality of third-color photosensitive pixels in the tenth diagonal direction, and the eighth pixel value read out by combining the plurality of second-color photosensitive pixels in the tenth diagonal direction in the second combined image.

21. The method of claim 16, wherein the method further comprises:

in a second resolution mode, merging the read first pixel values according to a plurality of panchromatic pixels corresponding to the optical filter set to obtain a panchromatic image;

Obtaining a fifth combined image according to the second pixel values read out by combining the plurality of first color photosensitive pixels corresponding to the filter set and the third pixel values read out by combining the plurality of third color photosensitive pixels corresponding to the filter set;

combining the read fourth pixel values according to a plurality of second color photosensitive pixels corresponding to the filter set to obtain a sixth combined image;

and obtaining a third target image based on the full-color image, the fifth combined image and the sixth combined image.

22. The method of claim 21, wherein the color filters of the second filter set further comprise a first color sub-filter or a second color sub-filter; the merging the read-out second pixel values according to the plurality of first color photosensitive pixels corresponding to the filter set and the merging the read-out third pixel values according to the plurality of third color photosensitive pixels corresponding to the filter set to obtain a fifth merged image, which includes:

Obtaining a fifth combined image according to second pixel values read out in a combined mode according to first color photosensitive pixels corresponding to a plurality of first color sub-filters in the first filter set, third pixel values read out in a combined mode according to third color photosensitive pixels corresponding to a plurality of third color sub-filters in the second filter set containing the first color sub-filters, and fourth pixel values read out in a combined mode according to second color photosensitive pixels corresponding to a plurality of second color sub-filters in the second filter set;

the step of obtaining a sixth combined image according to the fourth pixel values read out by combining the plurality of second color photosensitive pixels corresponding to the filter set includes:

and according to the fourth pixel value which is read out in a merging way according to the second color photosensitive pixels corresponding to the plurality of second color sub-filters in the first filter set, according to the second pixel value which is read out in a merging way according to the first color photosensitive pixels corresponding to the plurality of first color sub-filters in the second filter set, and according to the third pixel value which is read out in a merging way according to the third color photosensitive pixels corresponding to the plurality of third color sub-filters in the second filter set containing the second color sub-filters, obtaining a sixth merging image.

23. An image generating device is applied to an image sensor, and is characterized in that the image sensor comprises a filter array and a pixel array, the filter array comprises a minimum repeating unit, the minimum repeating unit comprises a plurality of filter sets, each filter set comprises a color filter and a full-color filter, the light inlet amount of the full-color filter transmitted by the full-color filter is larger than the light inlet amount of the color filter transmitted by the full-color filter, the color filter and the full-color filter both comprise 4 sub-filters, the filter sets comprise a first filter set and a second filter set, and at least one of the color filters comprises a first color sub-filter and a second color sub-filter; the color filters in the second filter set at least comprise a third color sub-filter; the pixel array comprises a plurality of pixels, a first color photosensitive pixel of the pixel array is arranged corresponding to a first color sub-filter of the filter array, a second color photosensitive pixel of the pixel array is arranged corresponding to a second color sub-filter of the filter array, a third color photosensitive pixel of the pixel array is arranged corresponding to a third color sub-filter of the filter array, and the pixel array is configured to receive light passing through the filter array to generate an electric signal;

The device comprises:

The first merging module is used for merging the read first pixel values according to a plurality of full-color pixels corresponding to the full-color filters in the filter set, merging the read second pixel values according to a plurality of first color photosensitive pixels corresponding to the color filters, and merging the read third pixel values according to a plurality of third color photosensitive pixels corresponding to the color filters in a first resolution mode to obtain a first merged image;

The second merging module is used for merging the read fourth pixel values according to a plurality of second color photosensitive pixels corresponding to the color filters to obtain a second merged image;

An image generation module, configured to make the position of the second pixel value in the first combined image and the position of the fourth pixel value in the second combined image identical; and traversing the pixel values in the first combined image, reading a fourth pixel value which is the same as the second pixel value from the second combined image under the condition that the traversed pixel value is the second pixel value, and adjusting the read fourth pixel value to be adjacent to the second pixel value until the pixel values in the first combined image are traversed, so as to obtain a first target image.

24. An electronic device comprising a memory, a processor and an image sensor, the memory having stored therein a computer program which, when executed by the processor, causes the processor to perform the steps of the method of any of claims 16 to 22.

25. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the method according to any of claims 16 to 22.

26. A computer program product comprising a computer program which, when executed by a processor, implements the steps of the method according to any one of claims 16 to 22.