CN116074649A - A camera and image processing method thereof - Google Patents

Abstract

The embodiment of the application discloses a camera and an image processing method thereof, relates to the field of image sensors, and can reduce the time interval between two exposures and ensure that a moving object in an image shot by the two exposures does not obviously move. The specific scheme is as follows: the camera includes a CMOS sensor and a controller for each pixel circuit in the CMOS sensor for: generating a first pulse signal for turning on the first transistor after the first exposure is completed, and transferring the first electrons generated by the photodiode to the charge storage region; generating a second pulse signal for turning on the second transistor before the second exposure starts, transferring the first electrons stored in the charge storage region to the floating diffusion node; and generating a third pulse signal for turning on the first transistor after the second exposure is finished, and transferring the second electrons generated by the photodiode to the charge storage region.

Description

A camera and image processing method thereof

This application claims the priority of the Chinese patent application with the application number 202111241467.1 and the application title "A dark current suppression method for maintaining signals in FD" submitted to the State Intellectual Property Office on October 25, 2021, the entire content of which is passed References are incorporated in this application.

technical field

The embodiments of the present application relate to the field of image sensors, and in particular, to a camera and an image processing method thereof.

Background technique

Complementary metal oxide semiconductor (CMOS) sensors have been widely used in the field of image sensors due to their advantages of small size and low power consumption.

In intelligent traffic scenarios, the illuminance of areas such as license plates directly illuminated by car lights can reach thousands of lux, while the illuminance of other areas may only be a few lux, and the dynamic range is very large. If a CMOS sensor is used for a single exposure, it may cause overexposure or underexposure, so the imaging dynamic range of a single exposure is very limited. In order to improve the dynamic range of the image, a global shutter CMOS sensor can be used to perform two exposures, and wide dynamic synthesis is performed on the images captured by the two exposures. However, the use of traditional global shutter CMOS sensors is limited by the readout speed, and the time interval between each shot is relatively long. When shooting a high-speed moving object in an intelligent traffic scene, if there is an obvious motion displacement between the two closest images, it will cause motion artifacts in the result of wide dynamic synthesis of the two-exposure images. Therefore, how to reduce the time interval between two exposures has become an urgent problem to be solved.

Contents of the invention

Embodiments of the present application provide a camera and an image processing method thereof, which can reduce the time interval between two exposures and ensure that there is no obvious movement of moving objects in images captured by the two exposures.

In order to achieve the above purpose, the embodiment of the present application adopts the following technical solutions:

According to the first aspect of the embodiments of the present application, a camera is provided, the camera includes a complementary metal oxide semiconductor CMOS sensor and a controller, the CMOS sensor includes a plurality of pixel circuits, and each pixel circuit in the plurality of pixel circuits includes at least one a photodiode, a charge storage region, a first transistor coupled between the photodiode and the charge storage region, a second transistor coupled between the charge storage region and the floating diffusion node; for each pixel circuit, a controller for: Generate a first pulse signal for turning on the first transistor after the first exposure ends, and transfer the first electrons generated by the photodiode to the charge storage region; generate a first pulse signal for turning on the second transistor before the second exposure starts The second pulse signal transfers the first electrons stored in the charge storage area to the floating diffusion node; after the second exposure is completed, the third pulse signal used to turn on the first transistor is generated to transfer the second electrons generated by the photodiode to the charge storage area.

Based on this scheme, the first electrons generated by the first exposure are transferred from the photodiode and stored in the floating diffusion node, and the second electrons generated by the second exposure are transferred and stored in the charge storage region, that is, the first electrons generated by the first exposure The first electron and the second electron generated by the second exposure can be stored in different devices. Compared with the second exposure that must wait until the first exposure is fully read out, since the electrons generated by the two exposures are stored in the floating diffusion node and the charge storage area respectively in this scheme, the second exposure does not need to wait After the first exposure is read out, the time interval between the second exposure and the first exposure can be reduced, and double exposure without time gap can be realized.

In a possible implementation manner, the above pixel circuit further includes a third transistor and a fourth transistor, the gate of the third transistor is coupled to the floating diffusion node, the source of the third transistor is coupled to the drain of the fourth transistor, and the third transistor is coupled to the drain of the fourth transistor. The drains of the three transistors are coupled to a preset voltage, the gate of the fourth transistor is coupled to the controller, and the source of the fourth transistor is a signal output terminal; The fourth pulse signal of the four transistors reads out the noise signal on the floating diffusion node.

Based on this solution, after the CMOS sensor generates the third exposure, since the controller will not generate the first pulse signal to turn on the first transistor, the third electrons generated by the photodiode will not be transferred to the charge storage area, nor will it be transferred to the The floating diffusion node, so the dark current on the floating diffusion node generates FD voltage, which is converted into a third output voltage by the third transistor, and the third output voltage is the dark current noise signal. The image obtained after the conversion of the dark current noise signal collected by each pixel circuit among the multiple pixel circuits in the CMOS sensor may be called a dark frame.

In another possible implementation manner, the above-mentioned controller is further configured to generate a fifth pulse signal for turning on the first transistor after the fourth exposure is completed, so as to transfer the fourth electrons generated by the photodiode to the charge storage region ; Wherein, the duration of the interval between the fourth exposure and the third exposure is the same as the interval between the second exposure and the first exposure.

Based on this scheme, the third exposure and the fourth exposure can be used to calibrate the dark current noise sequentially through the CMOS sensor. After the CMOS sensor generates the third exposure, the controller will not generate the first pulse signal to turn on the first transistor, so the photodiode The generated third electrons will not be transmitted to the charge storage area, nor will they be transmitted to the floating diffusion node, so the dark current on the floating diffusion node generates FD voltage, which is converted into a third output voltage by the third transistor, and the third output voltage is is the dark current noise signal.

In yet another possible implementation manner, the camera above further includes a processor; the processor is configured to: acquire a first image and a dark frame, the first image is an image corresponding to the first exposure, and the dark frame is a plurality of An image obtained after converting the noise signal collected by each pixel circuit in the pixel circuit; according to the dark frame, the first image is processed to obtain the first image after denoising.

Based on this solution, since the first image contains image signals and noise signals, the processor performs noise correction on the first image affected by dark current according to the calibrated dark current noise (dark frame), thereby reducing the dark current in the first image. Influenced by the current, an image with a high signal-to-noise ratio is obtained.

In yet another possible implementation manner, the duration of the first exposure is shorter than the duration of the second exposure; the processor is further configured to , to get a wide dynamic image.

Based on this solution, by performing wide dynamic synthesis on the denoised first image and the second image, not only the kTC noise in the dark area can be eliminated, but also the dynamic range of the image can be improved.

In yet another possible implementation manner, the above processor is specifically configured to: align the brightness of the first image after denoising with the brightness of the second image to obtain a third image; An area whose intensity is lower than a preset threshold is fused with a corresponding area in the second image to obtain the wide dynamic image.

Based on this scheme, the area in the third image whose signal intensity is lower than the preset threshold is fused with the corresponding area in the second image, so that the area in the wide dynamic image whose signal intensity is higher than the preset threshold is the third image. The corresponding area of the wide dynamic image, the area whose signal strength is lower than the preset threshold is the corresponding area of the second image. It can be understood that after WDR synthesis, the darker area is mainly filled by the corresponding pixels of the second image. Since the second image can eliminate kTC noise, the signal-to-noise ratio of the dark area of the image will be significantly improved. The pixels in the brighter area are the corresponding pixels of the third image, and since the image signal in the brighter area is stronger, the influence of kTC noise is relatively small. Therefore, by performing wide dynamic synthesis on the first image after denoising and the second image, the kTC noise in the dark area can be further eliminated, and the effect of improving the dynamic range can be achieved at the same time.

In yet another possible implementation manner, the foregoing fusion processing includes weighted fusion or direct replacement.

Based on this scheme, the region whose signal strength is lower than the preset threshold in the third image can be directly replaced with the corresponding region in the second image, or the region whose signal strength is lower than the preset threshold in the third image and the first The weighted fusion of the corresponding areas in the two images can not only improve the signal-to-noise ratio of the dark area of the image, but also improve the dynamic range of the image.

In yet another possible implementation manner, the above CMOS sensor is a global shutter CMOS sensor.

Based on this scheme, the electrons generated by the two exposures are stored in the floating diffusion node and the charge storage area respectively, so that the second exposure does not need to wait for the first exposure to be read out, and the second exposure and the second exposure can be reduced. The time interval between one exposure can realize double global exposure without time gap.

The second aspect of the embodiments of the present application provides an image processing method in a camera, the camera includes a complementary metal oxide semiconductor CMOS sensor and a controller, the CMOS sensor includes a plurality of pixel circuits, each of the plurality of pixel circuits Each pixel circuit comprises at least one photodiode, a charge storage region, a first transistor coupled between the photodiode and the charge storage region, a second transistor coupled between the charge storage region and the floating diffusion node; for each pixel circuit, The above method includes: the above-mentioned controller generates a first pulse signal for turning on the first transistor after the end of the first exposure, and transfers the first electrons generated by the photodiode to the charge storage area; the controller generates a pulse signal before the start of the second exposure a second pulse signal for turning on the second transistor, and transfer the first electrons stored in the charge storage region to the floating diffusion node; the controller generates a third pulse signal for turning on the first transistor after the second exposure ends, The second electrons generated by the photodiode are transferred to the charge storage region.

In a possible implementation manner, the above pixel circuit further includes a third transistor and a fourth transistor, the gate of the third transistor is coupled to the floating diffusion node, the source of the third transistor is coupled to the drain of the fourth transistor, and the third transistor is coupled to the drain of the fourth transistor. The drains of the three transistors are coupled to a preset voltage, the gate of the fourth transistor is coupled to the controller, and the source of the fourth transistor is a signal output terminal; the above method also includes: the controller generates a signal for turning on after the third exposure ends The fourth pulse signal of the fourth transistor reads out the noise signal on the floating diffusion node.

In another possible implementation, the above method further includes: the controller generates a fifth pulse signal for turning on the first transistor after the fourth exposure, and transfers the fourth electrons generated by the photodiode to the charge storage region ; Wherein, the duration of the interval between the fourth exposure and the third exposure is the same as the interval between the second exposure and the first exposure.

In yet another possible implementation manner, the above-mentioned camera further includes a processor; the above-mentioned method further includes: the processor acquires a first image and a dark frame, the first image is an image corresponding to the first exposure, and the dark frame is the above-mentioned An image obtained after converting the noise signal collected by each pixel circuit in the plurality of pixel circuits; the processor processes the first image according to the dark frame to obtain the first image after denoising.

In yet another possible implementation manner, the duration of the first exposure is shorter than the duration of the second exposure; the method further includes: the processor, according to the denoised first image and the second image corresponding to the second exposure, Get a wide dynamic image.

In another possible implementation manner, the above-mentioned processor obtains the wide dynamic image according to the first image after denoising and the second image corresponding to the second exposure, including: the processor converts the brightness of the first image after denoising to Aligning with the brightness of the second image to obtain a third image; the processor performs fusion processing on the area of the third image whose signal strength is lower than the preset threshold and the corresponding area in the second image to obtain a wide dynamic image.

In yet another possible implementation manner, the foregoing fusion processing includes weighted fusion or direct replacement.

In yet another possible implementation manner, the above CMOS sensor is a global shutter CMOS sensor.

For descriptions of effects of various implementations of the second aspect above, reference may be made to descriptions of effects of corresponding implementations of the first aspect above, and details are not repeated here.

Description of drawings

FIG. 1 is a schematic diagram of the structure and signal timing of a pixel circuit in a CMOS sensor provided by an embodiment of the present application;

FIG. 2 is a schematic diagram of signal timing of the last row of pixels of a global shutter CMOS sensor provided by an embodiment of the present application;

FIG. 3 is a schematic diagram of the structure of a pixel circuit in a global shutter CMOS sensor provided in an embodiment of the present application and the signal timing of the last row of pixels;

Fig. 4 is a schematic diagram of overexposure and underexposure during a single exposure provided by the embodiment of the present application;

FIG. 5 is a schematic diagram of a wide dynamic imaging scheme of a global shutter CMOS sensor provided in an embodiment of the present application;

FIG. 6 is a schematic structural diagram of a camera provided in an embodiment of the present application;

FIG. 7 is a schematic diagram of a signal timing sequence of a pixel circuit in a video camera provided by an embodiment of the present application;

FIG. 8 is a schematic structural diagram of a camera provided in an embodiment of the present application;

FIG. 9 is an application schematic diagram of a shooting scene provided by an embodiment of the present application;

FIG. 10 is a schematic diagram of a wide dynamic synthesis process provided by an embodiment of the present application;

FIG. 11 is a schematic diagram of a shooting and image processing effect provided by the embodiment of the present application;

FIG. 12 is a schematic flowchart of an image processing method in a camera provided in an embodiment of the present application;

FIG. 13 is a schematic flowchart of another image processing method in a video camera according to an embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application. In this application, "at least one" means one or more, and "multiple" means two or more. "And/or" describes the association relationship of associated objects, indicating that there may be three types of relationships, for example, A and/or B, which can mean: A exists alone, A and B exist at the same time, and B exists alone, where A, B can be singular or plural. The character "/" generally indicates that the contextual objects are an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one item (piece) of a, b or c can represent: a, b, c, a and b, a and c, b and c, or, a and b and c, wherein a, b and c can be single or multiple. In addition, in order to clearly describe the technical solutions of the embodiments of the present application, in the embodiments of the present application, words such as "first" and "second" are used to distinguish the same or similar items with basically the same function and effect, Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and execution order. For example, "first" in the first transistor and "second" in the second transistor in the embodiment of the present application are only used to distinguish different transistors. The first, second, etc. descriptions that appear in the embodiments of this application are only for illustration and to distinguish the description objects, and there is no order, nor does it represent a special limitation on the number of devices in the embodiments of this application, and cannot constitute a limitation on the number of devices in this application. Any limitations of the examples.

Transfer and transport in this application have the same meaning, and both can mean that electrons can move from one device to another, or from one location to another.

It should be noted that, in this application, words such as "exemplary" or "for example" are used as examples, illustrations or illustrations. Any embodiment or design described herein as "exemplary" or "for example" is not to be construed as preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "such as" is intended to present related concepts in a concrete manner.

First, the nouns in the embodiments of the present application are introduced,

Reset noise (kTC noise) is the noise generated when the floating diffusion node (Floating Diffusion, FD) is reset.

Correlated double sampling (CDS) is a sampling method for eliminating noise in CMOS sensors. By acquiring two signals during the pixel read cycle, the first signal is the "reset signal" acquired when the pixel is reset state, and the second signal is the "image signal" acquired when the electrons are transferred to the readout node. Subtracting the values of the two signals on the subsequent circuit of the CMOS sensor can eliminate kTC noise and noise generated by components such as Analog to Digital Converter (ADC).

Double data sampling (double data sampling, DDS) is a sampling method proposed by this application. Due to the design limitation of CMOS sensor, if the "image signal" when the electron is transmitted to the readout node is collected first during the signal readout process, and then the re-acquisition The "reset signal" of the set state, and then subtracting the values of these two signals can only eliminate the noise generated by components such as ADC, but cannot eliminate kTC noise.

In the field of image sensors, CMOS sensors have been widely used due to their advantages of small size and low power consumption. FIG. 1 is a schematic diagram of the structure and signal timing of a pixel circuit in a CMOS sensor. The CMOS sensor may include a plurality of pixel circuits, and each pixel circuit may include the circuit shown in (a) of FIG. 1 .

As shown in (a) in Figure 1 and (b) in Figure 1, in the working process of the CMOS sensor, the photodiode (Photo Diode, PD) first generates photoelectric conversion, and converts the received photons into electrons. When the exposure is finished and the signal is read out, the transfer gate transistor (Transfer gate, TX) is turned on, and the electrons accumulated in the photodiode PD are transferred to the floating diffusion node FD. The electrons cause the voltage change of the floating diffusion node FD, which is output to the signal line by the source follower, and then converted and read by the ADC.

Taking the CMOS sensor as a global shutter CMOS sensor as an example, Figure 2 is a schematic diagram of the signal timing of the last row of pixels of a global shutter CMOS sensor, as shown in Figure 2, since the global shutter CMOS sensor can allow the photodiodes in all pixel circuits to simultaneously complete exposure, and then read out the pixels row by row. Therefore, for the last row of pixels, during the process of reading the previous rows of pixels, the signals collected by the last row of pixel circuits are always stored in the FD. Affected by factors such as structural and material defects, electronic storage in FD will be affected by dark current, resulting in color cast after image white balance. Moreover, the dark current noise will seriously affect the signal-to-noise ratio of the image, resulting in a significant decrease in image quality.

In order to reduce the influence of dark current in the global shutter CMOS sensor, the global shutter CMOS sensor can add a charge storage area (memory, MEM) between the photodiode PD and the floating diffusion node FD, and the charge storage area MEM is used to store the electrons transmitted by the photodiode PD . Through the design of the charge storage region MEM, the residence time of electrons in the floating diffusion node FD can be reduced, and the influence of dark current can be avoided.

FIG. 3 is a schematic diagram of the structure of a pixel circuit in a global shutter CMOS sensor and the signal timing of the last row of pixels. The CMOS sensor may include multiple pixel circuits, and each pixel circuit may include the circuit shown in (a) in FIG. 3 .

As shown in (a) of FIG. 3 , each of the plurality of pixel circuits includes a photodiode PD, a floating diffusion node FD, a charge storage area MEM, transfer gate transistors TX1 and TX2, reset switches RST and GRST, a source pole follower and select transistor. Among them, PD is used to realize photoelectric conversion. TX1 is a switch for controlling the transfer of electrons from the photodiode PD to the charge storage area MEM. The charge storage area MEM is used to store electrons generated by the photodiode PD. TX2 is a switch for controlling the transfer of electrons from the charge storage area MEM to the floating diffusion node FD. The floating diffusion node FD is used to store electrons generated by photoelectric conversion. The reset switches RST and GRST are used to reset the voltage signal in the pixel circuit. The source follower is a switch for controlling the transfer of electrons from the floating diffusion node FD to the signal line. The selection transistor is a switch for signal output, through which the signal collected by the pixel circuit can be controlled to output first, and which pixel circuit will output the signal collected later.

Combining (a) and (b) in Figure 3, the working process of the global shutter CMOS sensor is introduced. First, in the exposure stage, TX1, TX2, RST, GRST, source follower and selection transistor are all in the off state, and the PD receives light for photoelectric conversion, and converts the received photons into electrons. In the exposure stage, the reset switch RST and the selection transistor are turned on to reset the transistor, which can conduct the remaining electrons from the last signal transmission in the FD to avoid interference with this signal transmission. Then, after the exposure ends, TX1 is turned on, and the electrons accumulated in the PD are transferred to the MEM. Due to the global shutter CMOS sensor, the photodiodes in all pixel circuits can be exposed at the same time, and the pixels are read out row by row. For the last row of pixels, in the process of reading the previous rows of pixels, the signal collected by the last row of pixel circuits is always stored in the MEM. When the row is read out, TX2 is turned on, and the electrons on the MEM will be transmitted. to the FD. The electrons accumulated on the FD generate a voltage, which can turn on the source follower and control the select transistor to turn on, so that the electrons on the FD can be transmitted to the signal line. Finally, all transistors can be turned on to perform a reset and proceed to the next image acquisition.

Combining with (b) in Figure 3, it can be seen that for each row of pixels in the global shutter CMOS sensor, only when the row of pixels enters the signal readout stage, electrons will be transmitted to the FD and read out. Therefore, by setting the charge storage area in the pixel circuit, the residence time of electrons in the FD can be reduced, and the influence of dark current can be reduced.

In the intelligent traffic scene, the object of the camera is a high-speed moving vehicle. As shown in Figure 4, taking the intersection scene at night as an example, the illuminance of areas such as license plates directly illuminated by car lights can reach several thousand lux, while the illuminance of other areas may only be a few lux, so the dynamic range is very large. If a single exposure is used, the imaging dynamic range of a single exposure is limited, as shown in (a) in Figure 4. If the exposure is controlled for the full frame, overexposure is prone to occur, resulting in the loss of license plate information. As shown in (b) in Figure 4, if the exposure is controlled for the license plate, the overall picture will be underexposed, and the details will be too dark or even lost.

In order to improve the dynamic range of an image, an embodiment of the present application provides a wide dynamic imaging solution of a global shutter CMOS sensor. As shown in FIG. 5, after the first exposure is completed, the electrons generated by the photodiode PD are transferred to the charge storage region MEM. After the second exposure is completed, the electrons generated by the photodiode PD are also stored in the charge storage region, so the second exposure must wait until the first exposure is completely read out. Since the photodiodes in the global shutter CMOS sensor complete exposure at the same time, the pixels are read out row by row, so it takes a long time to read out all the first exposures, which will result in a gap between the first exposure and the second exposure. longer time interval.

However, in scenarios such as intelligent transportation, when vehicles or other objects to be photographed are moving rapidly, if the time interval between two exposures is long, there will be obvious motion displacement in the two frames of images, resulting in two-exposure images Motion artifacts can appear as a result of WDR compositing.

In order to alleviate the problem that the time interval between two exposures is long, resulting in obvious motion displacement of the two frames of images, causing motion artifacts in the result of wide dynamic synthesis of the two-exposure images, the embodiment of the present application provides a The camera, which can reduce the time interval between two exposures, ensures that there is no obvious movement of moving objects in the images taken by the two exposures, so that when the images taken by the two exposures are combined in wide dynamic range, clear wide-angle images can be obtained. dynamic image. Moreover, the present application can further improve the signal-to-noise ratio of the image by removing dark current noise and kTC noise.

An embodiment of the present application provides a camera. As shown in FIG. 6, the camera includes a CMOS sensor and a controller. The CMOS sensor includes a plurality of pixel circuits, and each pixel circuit in the plurality of pixel circuits includes at least one photodiode PD. The storage area MEM, the first transistor TX1 coupled between the photodiode PD and the charge storage area MEM, and the second transistor TX2 coupled between the charge storage area MEM and the floating diffusion node FD.

The gates of the first transistor TX1 and the second transistor TX2 in FIG. 6 may be coupled to a controller (not shown in FIG. 6 ). In the embodiment of the present application, the first transistor TX1 and the second transistor TX2 are generally in the off state. When the first transistor TX1 and the second transistor TX2 need to be turned on, the controller can provide The gate of the second transistor TX2 sends a pulse signal, under the action of the pulse signal, the first transistor TX1 and the second transistor TX2 are turned on.

For each pixel circuit, a controller for:

After the first exposure is finished, a first pulse signal for turning on the first transistor TX1 is generated to transfer the first electrons generated by the photodiode PD to the charge storage region MEM.

Before the second exposure starts, a second pulse signal for turning on the second transistor TX2 is generated to transfer the first electrons stored in the charge storage region MEM to the floating diffusion node FD.

After the second exposure is finished, a third pulse signal for turning on the first transistor TX1 is generated to transfer the second electrons generated by the photodiode PD to the charge storage region MEM.

Referring to FIG. 6 , as shown in FIG. 7 , when the CMOS sensor generates the first exposure, the photodiode PD receives the photons and performs photoelectric conversion to accumulate the first electrons. After the first exposure, the controller generates a first pulse signal to turn on the first transistor TX1, and the first transistor TX1 is turned on under the action of the first pulse signal, and the first electrons generated by the photodiode PD are transferred to the charge storage area MEM. Before the CMOS sensor produces the second exposure, the controller generates a second pulse signal to turn on the second transistor TX2, and the second transistor TX2 is turned on under the action of the second pulse signal, and the first electrons stored in the charge storage area MEM are transferred to Floating diffusion node FD. When the CMOS sensor generates the second exposure, the photodiode PD receives photons and performs photoelectric conversion to accumulate second electrons. After the second exposure, the controller generates a third pulse signal to turn on the first transistor TX1, and under the action of the third pulse signal, the first transistor TX1 is turned on, and the second electrons generated by the photodiode PD are transferred to the charge storage area MEM.

According to the solution provided by the embodiment of the present application, since the electrons generated in the first exposure have been transferred to the floating diffusion node FD before the start of the second exposure, the electrons generated in the second exposure can be directly stored in the charge storage region MEM. Compared with the scheme shown in Figure 5 in which the second exposure must wait until the first exposure is fully read out, in this scheme the first electrons generated due to the first exposure are transferred from the photodiode PD and stored in the floating diffusion node In FD, the second electrons generated by the second exposure are transferred and stored in the charge storage area MEM, that is, the first electrons generated by the first exposure and the second electrons generated by the second exposure can be stored in different devices, so the second The exposure does not need to wait for the readout of the first exposure, and the time interval between the second exposure and the first exposure can be reduced to realize global exposure without time slots.

Optionally, the above CMOS sensor can be a global shutter CMOS sensor, that is, the photodiodes PD in the CMOS sensor can complete exposure at the same time, and then read out row by row. For the above-mentioned first exposure and second exposure, first the first image corresponding to the first exposure can be read out line by line, and then the second image corresponding to the second exposure can be read out line by line. Since the readout methods of the first image and the second image are different, the first image may be called a DDS frame, and the second image may be called a CDS frame.

Optionally, as shown in FIG. 6, each pixel circuit may further include a third transistor TX3 and a fourth transistor TX4, the gate of the third transistor TX3 is coupled to the floating diffusion node FD, and the source of the third transistor TX3 is coupled to The drain of the fourth transistor TX4, the drain of the third transistor TX3 are coupled to a predetermined voltage, the gate of the fourth transistor TX4 is coupled to the controller, and the source of the fourth transistor is a signal output terminal.

Referring to FIG. 6 , as shown in FIG. 7 , when the first image corresponding to the first exposure is read out, the first electrons accumulated on the FD generate a first FD voltage, which is converted into a first output voltage by the third transistor. The controller generates a pulse signal to turn on the fourth transistor TX4, and the fourth transistor TX4 is turned on under the action of the pulse signal, so as to read out the first output voltage. When reading out the second image corresponding to the second exposure, the controller can generate a pulse signal to turn on the second transistor TX2. Under the action of the pulse signal, the second transistor TX2 is turned on, and the second electrons stored in the charge storage area MEM Transfer to floating diffusion node FD. The second electrons accumulated on the FD generate a second FD voltage, which is converted into a second output voltage by the third transistor. The controller again generates a pulse signal for controlling the fourth transistor TX4 to read out the second output voltage. It can be understood that since the first electrons generated by the first exposure are always stored in the FD, when reading out the first image corresponding to the first exposure, the first electrons on the FD can be directly read out. Since the second electrons generated by the second exposure are always stored in the MEM, when reading out the second image corresponding to the second exposure, the second electrons on the MEM can be transferred to the FD first, and then the second electrons on the FD can be transferred to the FD. electronic readout. It is precisely because the electrons generated by the two exposures are stored in the FD and MEM respectively, so the second exposure does not need to wait for the readout of the first exposure, and double global exposure without time slot can be realized.

It should be noted that when the signals of the corresponding pixel points in the first image are read out line by line, since the signals collected by the corresponding pixel circuits in the first image are always stored on the FD, affected by factors such as structural and material defects, the electrons stored in the FD will be affected by dark current, resulting in color cast phenomenon after image white balance. Moreover, the dark current will seriously affect the signal-to-noise ratio of the image, resulting in a significant decline in image quality. Because the dark current varies among different CMOS image sensors and has a strong temperature dependence. Therefore, it is not possible to compensate with pre-calibrated data or data calibrated when the camera was first installed. In order to alleviate the problem of poor signal-to-noise ratio of the first image caused by dark current, the embodiment of the present application also provides a solution, by calibrating the dark current noise and correcting the first image according to the calibrated dark current, it can Reduce the influence of dark current and improve the signal-to-noise ratio of the first image. The calibration method of dark current noise is introduced below.

The controller is further configured to generate a fourth pulse signal for turning on the fourth transistor TX4 after the third exposure is finished, and read out the noise signal on the floating diffusion node FD.

As shown in FIG. 6 , in the calibration mode, when the CMOS sensor generates the third exposure, the photodiode PD receives the photons and performs photoelectric conversion to accumulate the third electrons. After the third exposure, the controller will not generate the first pulse signal to turn on the first transistor TX1, so the third electrons generated by the photodiode PD will not be transferred to the charge storage region MEM, nor will they be transferred to the floating diffusion node FD. The dark current on the floating diffusion node FD generates a third FD voltage, which is converted into a third output voltage by the third transistor. The controller generates a pulse signal controlling the fourth transistor TX4 to read out the third output voltage. The third output voltage is the dark current noise signal. The image obtained after converting the dark current noise signal collected by each pixel circuit in the multiple pixel circuits in the CMOS sensor may be called a dark frame, or may be called a DDS dark frame. The DDS dark frame is used to represent the calibrated dark current noise.

Optionally, as shown in FIG. 6 , each pixel circuit may further include reset switches RST and GRST, and the reset switches RST and GRST may be used to reset the voltage signal in the pixel circuit. For example, after the end of the third exposure, the controller can generate a pulse signal for turning on the GRST. Under the action of the pulse signal, the GRST is turned on, and the third electrons generated on the PD can be conducted away, so as not to affect the subsequent exposure. The resulting signal causes interference. For another example, in the exposure stage, the controller can generate a pulse signal for turning on the RST. Under the action of the pulse signal, the RST is turned on, and the electrons remaining in the FD can be conducted away, avoiding the signal generated by the subsequent exposure. cause disturbance.

Optionally, in the calibration mode, the controller is further configured to generate a fifth pulse signal for turning on the first transistor after the fourth exposure, so as to transfer the fourth electrons generated by the photodiode to the charge storage region.

As shown in FIG. 6 , in the calibration mode, when the CMOS sensor generates the fourth exposure, the photodiode PD receives photons and performs photoelectric conversion to accumulate fourth electrons. After the fourth exposure, the controller generates a fifth pulse signal to turn on the first transistor TX1, and under the action of the fifth pulse signal, the first transistor TX1 is turned on, and the second electrons generated by the photodiode PD are transferred to the charge storage area MEM. When reading out the fourth image corresponding to the fourth exposure, the controller can generate a pulse signal to turn on the second transistor TX2. Under the action of the pulse signal, the second transistor TX2 is turned on, and the fourth electrons stored in the charge storage area MEM Transfer to floating diffusion node FD. The fourth electrons form a fourth FD voltage at the FD node, which is converted into a fourth output voltage by the third transistor. The controller again generates a pulse signal for controlling the fourth transistor TX4 to read out the fourth output voltage. The image obtained after conversion of the fourth output voltage collected by each of the plurality of pixel circuits in the CMOS sensor is the fourth image. This fourth image may be referred to as a CDS frame.

Optionally, the duration of the fourth exposure is the same as that of the second exposure, and the duration of the third exposure is the same as that of the first exposure. The duration of the interval between the fourth exposure and the third exposure is also the same as the interval between the second exposure and the first exposure.

It should be noted that when calibrating the dark current noise, the third exposure calibration dark current noise can be generated only by the CMOS sensor, or the third exposure calibration and the fourth exposure calibration dark current noise can be sequentially generated by the CMOS sensor. The following embodiments are based on The scheme of the present application is introduced by taking the CMOS sensor sequentially generating the third exposure and the fourth exposure to calibrate the dark current noise as an example.

It can be understood that the working mode in which the CMOS sensor generates the first exposure and the second exposure may be called a noise compensation mode, and the working mode in which the CMOS sensor generates the third exposure and fourth exposure may be called a noise calibration mode. Because the first image corresponding to the first exposure shot in the noise compensation mode is affected by the dark current, the color and signal-to-noise ratio of the first image are poor. Therefore, the first image can be corrected by using the calibrated dark current noise in the noise calibration mode, so as to reduce the influence of the dark current and improve the signal-to-noise ratio of the image. The correction method of the dark current noise is introduced below.

As shown in FIG. 8 , the camera may further include a processor configured to: acquire a first image and a dark frame, where the dark frame is an image converted from a noise signal collected by each pixel circuit in the plurality of pixel circuits. And according to the dark frame, the first image is processed to obtain the first image after denoising.

Exemplarily, the processor may use a noise correction algorithm to perform noise correction on the first image affected by the dark current, for example, may make a difference between the first image and the dark frame to obtain the first image after denoising. Since the first image contains an image signal and a noise signal, noise correction can be performed on the noise signal obtained by using the calibration mode, so as to obtain an image with a high signal-to-noise ratio.

It can be understood that the present application can approximate the noise distribution of the current image through the dark current noise calibration, and then use the noise correction algorithm to remove the noise effect caused by the dark current and improve the signal-to-noise ratio of the image.

In the above noise compensation mode, the first image corresponding to the first exposure is a DDS normal frame, the second image corresponding to the second exposure is a CDS normal frame, the dark frame obtained by the third exposure in the noise calibration mode is a DDS dark frame, and the fourth The fourth image corresponding to the exposure is a CDS normal frame as an example. When the camera captures a vehicle passing a specific capture line, the capture frequency is several frames per second, as shown in Figure 9, when the vehicle does not reach the capture line, the camera does not need to capture. At this time, the CMOS sensor can be configured in noise calibration mode to obtain a DDS dark frame and a CDS normal frame at a fixed frequency, where the DDS dark frame is used to update the noise distribution map, and the CDS normal frame is used to output the video stream. The update of the noise profile is related to the startup of the CMOS sensor. When the CMOS sensor is just started, the DDS dark frame obtained for the first time is used as the initial value of the noise distribution. Starting from the second obtained DDS dark frame, the existing noise distribution map is updated with the newly obtained DDS dark frame, and the update method can be weighted average or other methods. By continuously updating the noise distribution map, the influence of random noise can be reduced and the signal-to-noise ratio of the image after denoising can be improved.

As shown in Figure 9, when the vehicle reaches the capture line, the main control chip of the camera issues a capture command to change the working mode to the noise compensation mode. At this time, the CMOS sensor outputs a DDS normal frame and a CDS normal frame. The normal DDS frame uses the noise distribution map obtained in the calibration mode to perform noise suppression processing to obtain a noise-reduced DDS image, eliminate the influence of dark current noise, and improve the image signal-to-noise ratio.

For the non-slot dual global exposure scheme proposed in this application, if the duration of the first exposure is shorter than the duration of the second exposure, that is, the exposure time of the DDS frame is set to a short exposure time, and the exposure time of a CDS frame is set to a long exposure time. Since the floating diffusion node FD has been used to store the first electrons obtained by the first exposure during the first exposure readout, the "image signal" must be read out first, and then the floating diffusion node FD is reset, and then Readout of "reset signal". Subtracting the "reset signal" from the "image signal" at this point will result in failure to remove kTC noise. Since the distribution of kTC noise in the image is relatively uniform for DDS frames, the kTC noise is most obvious when the image signal is low (dark area), and the signal-to-noise ratio is the lowest. In order to reduce the influence of kTC noise, the present application proposes that the first image after the dark current noise has been eliminated can be combined with the second image in wide dynamic range, so as to reduce the influence of kTC noise.

The processor is further configured to obtain a wide dynamic image according to the denoised first image and the second image corresponding to the second exposure.

The processor is specifically configured to align the brightness of the denoised first image with the brightness of the second image to obtain a third image. And performing fusion processing on the area in the third image whose signal strength is lower than the preset threshold and the corresponding area in the second image to obtain a wide dynamic image.

Optionally, the fusion processing includes but not limited to weighted fusion or direct replacement.

As shown in FIG. 10 , take fusion processing as direct replacement as an example. The processor may obtain the third image by multiplying the denoised first image by an exposure ratio, and the exposure ratio may be a ratio of the exposure time of the second image divided by the exposure time of the first image. Then, the area in the third image whose signal intensity is lower than the preset threshold is directly replaced with the corresponding area in the second image to obtain a wide dynamic image (that is, the wide dynamic range (high-dynamic range, HDR) shown in Figure 10 Composite image). That is, the region in the wide dynamic image whose signal strength is higher than the preset threshold is the corresponding region in the third image, and the region in the wide dynamic image whose signal strength is lower than the preset threshold is the corresponding region in the second image. It can be understood that after wide dynamic synthesis, the darker area is mainly filled by the corresponding pixels of the second image (CDS frame). Since the CDS frame can eliminate kTC noise, the signal-to-noise ratio of the dark area of the image will be significantly improved. The pixels in the brighter area are the corresponding pixels of the third image, and since the image signal in the brighter area is stronger, the influence of kTC noise is relatively small. Obviously, by wide dynamic synthesis of the denoised first image and the second image, the kTC noise in the dark area can be further eliminated, and the effect of improving the dynamic range can be achieved at the same time.

FIG. 11 is a schematic diagram of shooting and image processing effects provided by an embodiment of the present application. As shown in FIG. 11 , the image shown in (A) in FIG. 11 is an image including dark current noise and kTC noise, and the noise is very obvious. The image shown in (B) among Fig. 11 is the image after wide dynamic synthesis is carried out to the image shown in (A) among Fig. 11, compared with (A) among Fig. 11, (B) among Fig. 11 The image shown has an improved signal-to-noise ratio, but the image quality is still poor. The image shown in (C) in Figure 11 is the image after removing the dark current noise from the image shown in (A) in Figure 11, and the signal-to-noise ratio has been significantly improved (for example, the signal-to-noise ratio has been improved by more than 8dB) . Because the DDS frame readout process cannot eliminate the kTC noise, and the noise is mainly obvious in the dark area. Through the wide dynamic synthesis method in this application, the dark area of the DDS frame can be replaced with the corresponding area in the CDS frame, so that the influence of kTC noise on the image can be reduced. The image shown in (D) in Figure 11 is the image after the kTC noise is removed from the image shown in (C) in Figure 11, and it can be seen that the image after wide dynamic synthesis has significantly improved the signal-to-noise ratio in the dark area, Compared with the image shown in (A) in FIG. 11 , the image quality has been significantly improved.

The embodiment of the present application also provides an image processing method in a camera, the camera includes a CMOS sensor and a controller, the CMOS sensor includes a plurality of pixel circuits, each of the plurality of pixel circuits includes at least one photodiode, and the charge storage region, a first transistor coupled between the photodiode and the charge storage region, and a second transistor coupled between the charge storage region and the floating diffusion node. As shown in FIG. 12, for each pixel circuit, the method includes the following steps S1201-S1203.

S1201. After the first exposure, the controller generates a first pulse signal for turning on the first transistor, and transfers the first electrons generated by the photodiode to the charge storage region.

S1202. Before the second exposure starts, the controller generates a second pulse signal for turning on the second transistor, and transfers the first electrons stored in the charge storage region to the floating diffusion node.

S1203. After the second exposure, the controller generates a third pulse signal for turning on the first transistor, and transfers the second electrons generated by the photodiode to the charge storage region.

Optionally, each pixel circuit further includes a third transistor and a fourth transistor, the gate of the third transistor is coupled to the floating diffusion node, the source of the third transistor is coupled to the drain of the fourth transistor, and the drain of the third transistor The pole is coupled to a preset voltage, the gate of the fourth transistor is coupled to the controller, and the source of the fourth transistor is a signal output terminal. As shown in FIG. 13 , the above method may further include steps S1204-S1208 in addition to steps S1201-S1203.

S1204. After the third exposure, the controller generates a fourth pulse signal for turning on the fourth transistor, and reads out the noise signal on the floating diffusion node.

S1205. After the fourth exposure, the controller generates a fifth pulse signal for turning on the first transistor, and transfers the fourth electrons generated by the photodiode to the charge storage region.

Wherein, the interval between the fourth exposure and the third exposure is the same as the interval between the second exposure and the first exposure.

The above steps S1201-S1203 may be referred to as a noise compensation mode. Steps S1204-S1205 may be referred to as a noise calibration mode. The embodiment of the present application does not limit the sequential execution order of the above steps S1201-S1203 and steps S1204-S1205, and FIG. 13 is only an example to illustrate that steps S1204-S1205 are executed before steps S1201-S1203. In practical applications, the camera can work in the noise calibration mode for many times to continuously update the dark current noise, and then work in the noise compensation mode once, combined with the calibrated dark current noise, the captured image can be denoised to reduce the dark current noise. effects of current noise.

Optionally, the camera may further include a processor, as shown in FIG. 13 , in order to reduce the influence of dark current noise and improve the signal-to-noise ratio of the image, the method may further include steps S1206-S1207.

S1206. The processor acquires a first image and a dark frame, where the first image is an image corresponding to the first exposure, and the dark frame is an image converted from a noise signal collected by each pixel circuit in the plurality of pixel circuits.

S1207. The processor processes the first image according to the dark frame to obtain the first image after denoising.

Optionally, as shown in FIG. 13 , when the duration of the first exposure is shorter than the duration of the second exposure, in order to further reduce the influence of kTC noise, the method may further include step S1208.

S1208. The processor obtains a wide dynamic image according to the denoised first image and the second image corresponding to the second exposure.

Exemplarily, the processor obtains the wide dynamic image according to the first image after denoising and the second image corresponding to the second exposure, including: the processor first compares the brightness of the first image after denoising and the brightness of the second image Align to get the third image. Then, the processor fuses the area in the third image whose signal strength is lower than the preset threshold with the corresponding area in the second image to obtain a wide dynamic image.

Optionally, fusion processing includes but not limited to weighted fusion or direct replacement.

It can be understood that, for specific implementation manners of the image processing method in the camera shown in FIG. 12 and FIG. 13 , reference may be made to relevant descriptions in the foregoing embodiments, and details are not repeated here.

The CMOS sensor in the embodiment of the present application may be a global shutter CMOS sensor or a rolling shutter CMOS sensor, which is not limited in the embodiment of the present application.

The foregoing mainly introduces the solutions provided by the embodiments of the present invention from the perspective of method steps. It can be understood that, in order to realize the above functions, the computer includes hardware structures and/or software modules corresponding to each function. Those skilled in the art should easily realize that the present application can be realized in a combined form of hardware and computer software in combination with the units and algorithm steps of each example described in the embodiments disclosed herein. Skilled artisans may use different methods to implement the described functions for each specific application, but such implementation should not be regarded as exceeding the scope of the present invention.

The steps of the methods or algorithms described in connection with the disclosure of this application can be implemented in the form of hardware, or can be implemented in the form of a processor executing software instructions. Software instructions can be composed of corresponding software modules, and software modules can be stored in random access memory (random access memory, RAM), flash memory, erasable programmable read-only memory (erasable programmable ROM, EPROM), electrically erasable Programmable read-only memory (electrically EPROM, EEPROM), registers, hard disk, removable hard disk, compact disc (CD-ROM) or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor such the processor can read information from, and write information to, the storage medium. Of course, the storage medium may also be a component of the processor. The processor and storage medium can be located in the ASIC. In addition, the ASIC may be located in the core network interface device. Certainly, the processor and the storage medium may also exist in the core network interface device as discrete components.

Those skilled in the art should be aware that, in the above one or more examples, the functions described in the present invention may be implemented by hardware, software, firmware or any combination thereof. When implemented in software, the functions may be stored on or transmitted over as one or more instructions or code on a computer-readable medium. Computer-readable media includes both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. A storage media may be any available media that can be accessed by a general purpose or special purpose computer.

The specific embodiments described above have further described the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above descriptions are only specific embodiments of the present invention and are not intended to limit the scope of the present invention. Protection scope, any modification, equivalent replacement, improvement, etc. made on the basis of the technical solution of the present invention shall be included in the protection scope of the present invention.

Claims (16)

1. A video camera, characterized in that the video camera includes a complementary metal oxide semiconductor CMOS sensor and a controller, the CMOS sensor includes a plurality of pixel circuits, and each pixel circuit in the plurality of pixel circuits includes at least one a photodiode, a charge storage region, a first transistor coupled between the photodiode and the charge storage region, a second transistor coupled between the charge storage region and a floating diffusion node; For each of the pixel circuits, the controller is configured to: generating a first pulse signal for turning on the first transistor after the first exposure is finished, and transferring the first electrons generated by the photodiode to the charge storage region; generating a second pulse signal for turning on the second transistor before the second exposure starts, and transferring the first electrons stored in the charge storage region to the floating diffusion node; After the second exposure is finished, a third pulse signal for turning on the first transistor is generated to transfer the second electrons generated by the photodiode to the charge storage region. 2. The camera according to claim 1, wherein the pixel circuit further comprises a third transistor and a fourth transistor, the gate of the third transistor is coupled to the floating diffusion node, and the third transistor The source of the fourth transistor is coupled to the drain of the fourth transistor, the drain of the third transistor is coupled to a preset voltage, the gate of the fourth transistor is coupled to the controller, and the source of the fourth transistor Very signal output terminal; The controller is further configured to generate a fourth pulse signal for turning on the fourth transistor after the third exposure is finished, and read out the noise signal on the floating diffusion node. 3. The video camera according to claim 2, wherein the controller is further configured to generate a fifth pulse signal for turning on the first transistor after the fourth exposure is finished, and turn on the photodiode The generated fourth electrons are transferred to the charge storage area; wherein, the interval between the fourth exposure and the third exposure is the same as the interval between the second exposure and the first exposure. 4. The camera according to claim 2 or 3, wherein the camera further comprises a processor; the processor is configured to: Acquire a first image and a dark frame, the first image is an image corresponding to the first exposure, and the dark frame is an image obtained after conversion of the noise signal collected by each pixel circuit in the plurality of pixel circuits ; The first image is processed according to the dark frame to obtain a denoised first image. 5. The camera according to claim 4, wherein the duration of the first exposure is shorter than the duration of the second exposure; The processor is further configured to obtain a wide dynamic image according to the denoised first image and the second image corresponding to the second exposure. 6. The video camera according to claim 5, wherein the processor is specifically used for: aligning the brightness of the first image after denoising with the brightness of the second image to obtain a third image; performing fusion processing on the area in the third image whose signal strength is lower than the preset threshold and the corresponding area in the second image to obtain the wide dynamic image. 7. The camera according to claim 6, wherein the fusion processing comprises weighted fusion or direct replacement. 8. The camera according to any one of claims 1-7, wherein the CMOS sensor is a global shutter CMOS sensor. 9. An image processing method in a video camera, wherein the video camera includes a complementary metal oxide semiconductor CMOS sensor and a controller, the CMOS sensor includes a plurality of pixel circuits, each of the plurality of pixel circuits a pixel circuit comprising at least one photodiode, a charge storage region, a first transistor coupled between the photodiode and the charge storage region, a second transistor coupled between the charge storage region and a floating diffusion node; For each of the pixel circuits, the method includes: The controller generates a first pulse signal for turning on the first transistor after the first exposure, and transfers the first electrons generated by the photodiode to the charge storage region; The controller generates a second pulse signal for turning on the second transistor before the second exposure starts, and transfers the first electrons stored in the charge storage region to the floating diffusion node; The controller generates a third pulse signal for turning on the first transistor after the second exposure, and transfers the second electrons generated by the photodiode to the charge storage region. 10. The method according to claim 9, wherein the pixel circuit further comprises a third transistor and a fourth transistor, the gate of the third transistor is coupled to the floating diffusion node, and the third transistor The source of the fourth transistor is coupled to the drain of the fourth transistor, the drain of the third transistor is coupled to a preset voltage, the gate of the fourth transistor is coupled to the controller, and the source of the fourth transistor Extremely signal output terminal; Described method also comprises: The controller generates a fourth pulse signal for turning on the fourth transistor after the third exposure is finished, and reads out the noise signal on the floating diffusion node. 11. The method of claim 10, further comprising: The controller generates a fifth pulse signal for turning on the first transistor after the fourth exposure is completed, and transfers the fourth electrons generated by the photodiode to the charge storage area; wherein, the fourth exposure and The duration of the interval between the third exposures is the same as the duration of the interval between the second exposure and the first exposure. 12. The method according to claim 10 or 11, wherein the camera further comprises a processor; the method further comprises: The processor acquires a first image and a dark frame, the first image is an image corresponding to the first exposure, and the dark frame is the conversion of the noise signal collected by each pixel circuit in the plurality of pixel circuits The resulting image; The processor processes the first image according to the dark frame to obtain a denoised first image. 13. The method according to claim 12, wherein the duration of the first exposure is shorter than the duration of the second exposure; the method further comprises: The processor obtains a wide dynamic image according to the denoised first image and the second image corresponding to the second exposure. 14. The method according to claim 13, wherein the processor obtains a wide dynamic image according to the denoised first image and the second image corresponding to the second exposure, comprising: The processor aligns the brightness of the denoised first image with the brightness of the second image to obtain a third image; The processor fuses an area in the third image whose signal strength is lower than a preset threshold with a corresponding area in the second image to obtain the wide dynamic image. 15. The method according to claim 14, wherein the fusion process comprises weighted fusion or direct replacement. 16. The method according to any one of claims 9-15, wherein the CMOS sensor is a global shutter CMOS sensor.

Images