TWI722542B - Method and apparatus for performing auto-exposure control in depth sensing system including projector - Google Patents

Abstract

A method for performing auto-exposure (AE) control in a depth sensing system includes: capturing a first reference frame and a second reference frame of a scene based on different illumination conditions; calculating a plurality of first block means with respect to first blocks of the first reference frame and calculating a plurality of second block means with respect to second blocks of the second reference frame; calculating a plurality of block mean differences according to the first block means and the second block means; determining a plurality of first region of interest (ROI) blocks from the first blocks and a plurality of a second ROI blocks from the second blocks; and adjusting power of a light source of the projector according to blocks means of the first ROI blocks and blocks means of the second ROI blocks.

Description

Method for performing automatic exposure control in a depth sensing system including a projector and Device

The present invention relates to a depth sensing system, and more particularly to an automatic exposure control device and automatic exposure control method in the depth sensing system.

In active depth sensing, the light that uses light with a known pattern to illuminate a scene or object in a program is usually called "structured light." The structure of the patterned light projected onto the scene can be used to encode the depth information of the scene. Once the light pattern is found in a frame captured or captured by a sensor, a three-dimensional scene can be reconstructed. The known relationship between the projected light pattern and the decoded light pattern can be used to derive the depth information of the captured scene.

For this depth sensing technology, the intensity of the projected light is necessary, and may affect the power consumption of the entire depth sensing system or the accuracy of the sensed depth information. Therefore, there is a need for an appropriate automatic exposure control applied in the depth sensing system.

An object of the present invention is to provide an automatic exposure control method used in a depth sensing system Method and device. The present invention obtains reasonable automatic exposure and power control of the projector in the depth sensing system by estimating the lighting conditions of a scene and/or setting the sensing device in the depth sensing system. In order to quickly estimate the lighting conditions, the present invention uses pixel binning technology to read out signals from the sensing device to obtain a reference frame with a lower resolution, which can effectively reduce power consumption and computational burden. In addition, the present invention utilizes reference frames captured under different lighting conditions to analyze the difference between the intensity of the ambient light of the scene and the predetermined intensity of the light source of the projector, thereby determining the appropriate power of the light source for driving the projector. In order to obtain accurate lighting condition estimation results, the present invention uses a region of interest (ROI) determination procedure to eliminate information that is meaningless or that would hinder the estimation of the lighting conditions of the scene. By appropriately and accurately performing auto-exposure (AE) control, the depth sensing system can obtain more accurate depth information of the scene and avoid power consumption.

At least one embodiment of the present invention provides a method for performing automatic exposure control in a depth sensing system including a projector, including: capturing a first reference frame of a scene and a second reference according to different lighting conditions Frame; Calculate the average value of the first blocks corresponding to the plurality of first blocks of the first reference frame and calculate the average value of the second blocks corresponding to the plurality of second blocks of the second reference frame ; According to the first block average value and the second block average value to calculate the difference of the complex block average value; according to the first block average value, the second block average value and the The difference between the average values of the blocks determines a plurality of first region of interest (ROI) blocks from the first blocks and a plurality of second ROI blocks from the second blocks; and according to the first ROI blocks The average value of the corresponding complex blocks and the average value of the complex blocks corresponding to the second ROI blocks adjust the power of a light source of the projector.

At least one embodiment of the present invention provides a device for performing automatic exposure control in a depth sensing system including a projector, including: a sensing device, a block average calculation unit, and a block average Difference calculation unit, a region of interest (ROI) determination unit and an AE control unit. The sensing device is used for capturing a first reference frame and a second reference frame of a scene according to different lighting conditions. The block average calculation unit is used to calculate the complex number corresponding to the first block of the first reference frame The average value of the first block and the average value of the plurality of second blocks corresponding to the plurality of second blocks of the second reference frame are calculated. The block average difference calculation unit is used for calculating the difference between the plurality of block averages according to the first block average value and the second block average value. The region of interest (ROI) determining unit is used to determine the plurality of first ROI blocks and the second block from the first block according to the difference between the first block average value, the second block average value, and the block average value A plurality of second ROI blocks. The AE control unit is used for adjusting the power of a light source of the projector according to the average value of the plurality of blocks corresponding to the first ROI block and the average value of the plurality of blocks corresponding to the second ROI block.

10: Depth sensing system

12: Projector

14: Sensing device

15: readout circuit

16: processing circuit

18: Depth decoding circuit

20: Sensor

100: AE control device

120: Block average calculation unit

140: Block average difference calculation unit

160: ROI decision unit

180: AE control unit

B11-B112: The first block

B21-B212: The second block

D1-D12: The difference between the average value of the block

M21-M212, M11-M112: block average

RF1: The first reference frame

RF2: Second reference frame

FIG. 1 shows a block diagram of a depth sensing system with an automatic exposure control device according to an embodiment of the present invention.

Fig. 2 is a schematic diagram showing how to calculate the block average value, the difference between the block average value, and the relationship according to an embodiment of the present invention.

Figure 3 is a schematic diagram showing how to determine the lighting conditions of a scene according to an embodiment of the present invention.

Figure 4 is a flowchart showing the operation of the automatic exposure control device according to an embodiment of the present invention.

In the following, many specific details are described to provide a thorough understanding of the embodiments of the present invention. However, those skilled in the art will still understand how to implement the present invention without one or more specific details or relying on other methods, elements or materials. In other cases, well-known structures, materials, or operations are not shown or described in detail to avoid obscuring the main concept of the present invention.

Throughout the specification, reference to “one embodiment”, “one embodiment”, “one example” or “one The reference to "example" means that a specific feature, structure, or characteristic described in conjunction with the embodiment or example is included in at least one of the multiple embodiments of the present invention. Therefore, the phrases "in one embodiment," "in an embodiment," "in an example," or "in an example" appearing in various places throughout this specification do not necessarily all refer to the same embodiment. Or example. In addition, specific features, structures or characteristics can be combined in any suitable combination and/or sub-combination in one or more embodiments or examples.

Overview

FIG. 1 shows a block diagram of a depth sensing system 10 having an auto-exposure (AE) control device 100 (hereinafter referred to as AE control device 100) according to an embodiment of the present invention. The depth sensing system 10 may include a projector 12, a sensing device 14 and a processing circuit 16. Generally speaking, the projector 12 can be used to project light with a known pattern onto a scene or an object. The sensing device 14 can be used to capture an image frame or frame of the scene or object. The processing circuit 16 can be used to perform operations related to processing such as pixel-binning and/or down-sampling. In addition, the depth sensing system 10 can also include a depth decoding circuit 18, which can be used to decode depth information that has been encoded in a known pattern. In particular, the depth decoding circuit 18 decodes the depth information according to the frame of the scene or object on which the known pattern is projected. In one embodiment, the projector 12 has a light source, which can be used to emit infrared rays, and the sensing device 14 can sense infrared rays.

The AE control device 100 can at least control the power used to drive the projector 12 (for example, the intensity of the light projected by the light source) and the exposure time of the sensor 20 of the sensor device 14.

The AE control device 100 may include a block average calculation unit 120, a block average difference calculation unit 140, a region of interest (ROI) determining unit 160, and an AE control unit 180. The block average calculation unit 120 is used to calculate the average value of a plurality of blocks of a plurality of reference frames captured by the sensing device 14 for a scene. The block average difference calculation unit 140 is used for calculating the difference between the block averages based on the block averages calculated by two different reference frames. ROI decision The unit 160 can be used to determine a block of the interest area from the block of the reference frame. The AE control unit 180 is used to determine the lighting conditions of the scene, and correspondingly perform AE control on the depth sensing system 10.

Image capture

In order to determine the appropriate power of the light source driving the projector 12 and/or the appropriate exposure time of the sensor 20 for a scene, the AE control unit 180 first needs to collect the lighting conditions of a scene under the surrounding light (ie, ambient light). News. Therefore, the AE control unit 180 requests the sensing device 14 to capture a first reference frame RF1 of the scene when there is no active light source projected on the scene. At this time, the light source of the projector 12 is turned off, and the sensing device 14 captures the first reference frame RF1 of the scene with only ambient light. Then, the AE control unit 180 requests the sensing device 14 to capture a second reference frame RF2 of the scene when an active light source is projected onto the scene. At this time, the light source of the projector 12 is turned on and driven at a certain intensity level. The light source projects light onto the scene, and the sensing device 14 correspondingly captures the second reference frame RF2 of the scene when there is ambient light and an active light source.

Typically, the sensor 20 of the sensor device 14 can sense light, and a readout circuit 15 of the sensor device 14 can read out signals from each pixel of the sensor 20 as sensing data. However, in order to save computational burden and power consumption, the sensing data can be read out in a pixel-combined manner to combine the signals of multiple pixels of the sensor 20 into a single signal. In particular, the processing circuit 16 can perform a pixel combining operation on the signal read by the readout circuit 15 to combine the signals of multiple pixels into a single signal. Correspondingly, the process of reducing the sampling frequency can also be executed by the processing circuit 16 on the read signal to reduce the amount of data. That is, the data of multiple pixels on the sensor 20 will be combined, thereby obtaining a frame with a lower resolution than the original resolution provided by the sensor 20. For example, the sensor 20 can output a frame with an original resolution of 960×960 pixels. Through the operation of pixel binning (2x pixel binning), a frame with a resolution of 480x480 pixels can be obtained. Through the process of downsampling frequency (4x row downsampling frequency), a frame with a resolution of 480x120 pixels can be obtained. It is worth noting that, in different embodiments of the present invention, the scale factor of the above-mentioned pixel binning and the scale factor of downsampling frequency may be different. On the other hand, if There is no need to worry too much about power consumption, and one of pixel binning and downsampling frequency can also be omitted.

The AE control device 100 of the depth sensing system 10 can analyze the lighting conditions based on the data after performing pixel binning and downsampling operations through the processing circuit 16. However, since the processing of pixel binning and downsampling frequency will result in the loss of depth information, thereby affecting the accuracy of depth sensing, the depth decoding circuit 18 of the depth sensing system 10 will not be processed according to the pixel binning and downsampling frequency processing. The data is deeply decoded.

Block average

Once the first reference frame RF1 and the second reference frame RF2 are captured, their pixel data will be sent to the block average calculation unit 120. As shown in Figure 2, the block average calculation unit 120 divides the first reference frame RF1 and the second reference frame RF2 into a plurality of first blocks B11-B112 and a plurality of second blocks B21-B212 . It should be noted that the present invention does not limit the number of blocks divided by the reference frame. According to various embodiments of the present invention, the first reference frame RF1 and the second reference frame RF2 can be divided into more or less blocks.

Since the first blocks B11-B11N and the second blocks B21-B21N are spatially identical, each of the first blocks B11-B11N is associated with one of the second blocks B21-B21N. For example, as shown in Figure 2, the first block B11 is associated with the second block B21, the first block B12 is associated with the second block B22, and the first block B13 is associated with the second block B23. Correlation, and so on. Typically, each first block and second block have the same size (for example, including m times n pixels).

The block average calculation unit 120 calculates the block average for each first block B11-B112 according to the sensing value of the pixels included in a first block. In addition, the block average calculation unit 120 calculates the block average for each second block B21-B212 according to the sensing value of the pixels included in a second block. When the first block B11-B112 belongs to the first reference frame RF1 captured with only ambient light, and the second block B21-B212 belongs to the corresponding situation with ambient light and active light source The captured second reference frame RF2, the average value of the blocks corresponding to the first blocks B11-B112 The block average values M21-M212 corresponding to M11-M112 and the second block B21-B212 will be different. After the block average value M11-M112 corresponding to the first block B11-B112 and the block average value M21-M212 corresponding to the second block B21-B212 are calculated, the block average difference calculation unit 140 will Calculate the block average difference between the block average of the first block and the block average of the associated second block. For example, the block average difference calculation unit 140 will calculate the average difference D1 between the block average M11 of the first block B11 and the block average M21 of the second block B21 The average difference D2 between the block average M12 of the block B12 and the block average M22 of the second block B22...The area between the block average M112 of the first block B112 and the second block B212 The average value difference D12 between the block average values M212 and so on.

Determine ROI

The block average value M11-M112 corresponding to the first block B11-B112, the block average value M21-M212 corresponding to the second block B21-B212, and the difference D1-D12 of the block average value are calculated After that, the procedure for determining the region of interest (ROI) can be executed. The purpose of performing the ROI determination procedure is to exclude those blocks whose average value is meaningless or blocks that will hinder the estimation of the lighting conditions of the scene. For example, if a block is associated with a low/high-reflectivity object in the scene or the distance in the scene, since this block cannot properly reflect the lighting conditions of the scene, the block will be exclude. Similarly, if a block is associated with an over-exposed or under-exposed object or part of the scene, this block will also be excluded because the block cannot properly reflect the lighting conditions of the scene. For example, when the average value of the block corresponding to a first block is higher than an average threshold value, when determining the first ROI block and the second ROI block, the average value corresponding to this block can be excluded The first block and the associated second block.

In addition, when one of the first blocks B11-B112 is over-exposed, the corresponding one of the second blocks B11-B112 is inevitably over-exposed. However, when one of the first blocks B11-B112 is not overexposed, the corresponding one of the second blocks B11-B112 may be overexposed due to the excessive power of the projector 12.

In order to exclude those areas with a low/high reflectivity in the scene or blocks associated with the distance in the scene, the difference D1-D12 of the average value of each block will be compared with a difference threshold. This is because the block average of those objects with low/high reflectivity in the scene or the blocks associated with the distance in the scene will not change significantly with changes in lighting conditions. Therefore, the difference between the average values of the corresponding blocks will be very small. If the average difference of a block is less than the difference threshold, the corresponding block will be excluded when determining the ROI block. For example, if the block average difference D3 between the block average value M13 of the first block B13 and the block average value M23 of the second block B23 is less than the difference threshold, the first block B13 and The second block B23 will be excluded. Conversely, if the block average difference D3 between the block average M13 of the first block B13 and the block average M23 of the second block B23 is higher than the difference threshold, the first block B13 and The second block B23 is determined as the first ROI block ROIB13 and the second ROI block ROIB23. When one of the first blocks B11-B112 is underexposed, if the projector 12 provides moderate illumination for the scene/object, then a block corresponding to the second block B21-B212 can be moderately exposed .

Correspondingly, the ROI determination unit 160 can determine the first block B11-B112 from the first block B11-B112 based on the above-mentioned principle of excluding those with low/high reflectivity in the scene or the block associated with the distance in the scene. An ROI block ROIB11-ROIB1N and a second ROI block ROIB21-ROIB2N are determined from the second block B21-B212.

Decide lighting conditions

After the first ROI block ROIB11-ROIB1N is determined from the first block B11-B112 and the second ROI block ROIB21-ROIB2N is determined from the second block B21-B212, the AE control unit 180 will correspondingly determine the first ROI block ROIB11-ROIB1N The lighting conditions of the scene where a reference frame RF1 and a second reference frame RF2 are captured. According to the determined scene lighting conditions, the AE control unit 180 controls the projector 12 and/or the sensing device 14 appropriately.

For a better understanding, please refer to Figure 3, where Figure 3 shows the process of determining the lighting conditions of a scene. According to various embodiments of the present invention, the AE control unit 180 can illuminate a scene The conditions are determined to be one of extremely dark conditions, extreme overexposure conditions, partial overexposure conditions, good exposure conditions, and underexposure conditions.

At the beginning of this process, the AE control unit 180 checks whether there are any ROI blocks. In particular, the AE control unit 180 will check whether the number of the second ROI blocks ROIB21-ROIB2N is zero. As described above, if the illumination of the first reference frame RF1 is similar to the illumination of the second reference frame RF2, the number of ROI blocks determined by the ROI determination unit 160 will be zero.

If the number of second ROI blocks is 0, it means that the lighting condition of this scene is extremely dark or extremely overexposed. That is, all the second blocks B21-B212 are excluded due to extreme overexposure or extreme underexposure. Therefore, no second block B21-B212 is judged as the second ROI block.

Therefore, the reason why the number of second ROI blocks is 0 can be determined. An average value of the block average values corresponding to the second blocks B21-B212 can be compared with an overexposure threshold (for example, a grayscale value of 250) and an underexposure threshold (for example, a grayscale value of 5). ) In comparison, if the average value of the average values of the blocks corresponding to the second blocks B21-B212 is lower than the underexposure threshold, it can indicate that the lighting conditions of the scene are extremely dark conditions. In this extremely dark condition, the AE control unit 180 will request the depth sensing system 10 to significantly increase the power of the light source of the projector 12, or increase the exposure time of the sensor 20, thereby obtaining brighter lighting conditions for the scene. On the other hand, if the average value of the block average values corresponding to the second blocks B21-B212 is higher than the overexposure threshold, it can indicate that the lighting condition of the scene is an extreme overexposure condition. Under extreme overexposure conditions, the AE control unit 180 will request the sensing device 14 to significantly reduce the exposure time of the sensor 20 to avoid reading the self-sensing when the power of the light source of the projector 12 is at the lowest gear. The signal of the device 20 will be truncated.

If the number of the second ROI blocks ROIB21-ROIB2N is not 0 (meaning that not all the second blocks B21-B212 are extremely dark or extremely overexposed), the AE control unit 180 will take these first ROI blocks The average value of one of the block averages of ROIB11-ROIB1N is calculated to calculate the first ROI frame average FM1, and the first ROI frame average value FM1 is calculated by taking one of the averages of the second ROI blocks ROIB21-ROIB2N. Two ROI frame average value FM2.

When the number of blocks with a block average value higher than the overexposure threshold in the second ROI block (ie, overexposed blocks) is higher than a block number threshold TH, the AE control unit 180 will determine this scene The lighting conditions are partial overexposure conditions. For example, if there are 5 blocks in the second ROI block ROIB21-ROIB2N whose block average is higher than the overexposure threshold (for example, 250), the number of second overexposed blocks is 5, when When this number is higher than the block number threshold TH (for example, 3), it can be judged that the lighting condition of this scene is a partial overexposure condition. Therefore, the AE control unit 180 calculates an adjustment factor based on a ratio (or an offset can be added) of the first ROI frame average value FM1 and the second ROI frame average value FM2. According to the calculated adjustment factor, the AE control unit 180 requests the depth sensing system 10 to slightly reduce the power of the light source of the projector 12, thereby achieving a more balanced lighting condition for the scene. On the other hand, the AE control unit 180 can also request the depth sensing system 10 to reduce the exposure time of the sensor 20 according to the calculated adjustment factor, thereby achieving a more balanced lighting condition for the scene.

On the other hand, when the number of blocks with a block average value higher than the overexposure threshold (ie, overexposed blocks) in the second ROI block is lower than the block number threshold TH, the AE control unit 180 will Judge the lighting conditions of this scene as well-exposed or under-exposed conditions. For example, if there are 2 blocks in the second ROI block ROIB21-ROIB2N whose block average is higher than the overexposure threshold (for example, 250), the number of second overexposed blocks is 2, when When this number is lower than the block number threshold TH (for example, 3), it can be judged that the lighting condition of the scene is a good or underexposed condition.

In order to distinguish between well-exposed and under-exposed conditions, it is necessary to confirm whether the average value FM1 of the first ROI frame is higher than a minimum illumination intensity. If the value of the first ROI frame average value FM1 is higher than the value of the minimum illumination intensity, the AE control unit 180 will determine that the illumination condition of the scene is good exposure. In view of this, the AE control unit 180 will calculate an adjustment factor based on a ratio (or an offset can be added) of the first ROI frame average FM1 and the second ROI frame average FM2. According to the calculated adjustment factor, the AE control unit 180 requests the depth sensing system 10 to adjust the power of the light source of the projector 12. If the first ROI frame is average If the value of the value FM1 is lower than the value of the minimum lighting intensity, the AE control unit 180 will determine that the lighting condition of the scene is an underexposure condition. In view of this, the AE control unit 180 calculates an adjustment factor based on the ratio of the value of the minimum illumination intensity to the average value FM2 of the second ROI frame. According to the calculated adjustment factor, the AE control unit 180 requests the depth sensing system 10 to adjust the power of the light source of the projector 12, thereby obtaining brighter lighting conditions for the scene.

In one embodiment, when the lighting condition is determined to be partially overexposed, the AE control unit 180 will request the sensor 20 to reduce the exposure time according to the extent of the block overexposure. More specifically, determine the number of overexposed ROI blocks. According to the number of over-exposed ROI blocks, the AE control unit 180 will request the sensor 20 to reduce the exposure time according to a corresponding scale factor. For example, the number of overexposed ROI blocks can be classified into the following levels: 1x, 2x, 4x, 8x, and 16x. Therefore, the corresponding scaling factors used to reduce the exposure time can be 0.625x, 0.5x, 0.375, respectively. x, 0.25x, and 0.125x.

Fig. 4 is a flowchart showing the main operation steps of the AE control device 100. This process includes the following steps:

Step 410: Capture a first reference frame and a second reference frame of a scene according to different lighting conditions.

Step 420: Calculate the average value of the plurality of first blocks corresponding to the plurality of first blocks of the first reference frame and calculate the average value of the plurality of second blocks corresponding to the plurality of second blocks of the second reference frame.

Step 430: Calculate the difference between the average values of the plural blocks according to the average values of the first blocks and the average values of the second blocks.

Step 440: Determine a plurality of first ROI blocks from the first blocks according to the average value of the first blocks, the average values of the second blocks, and the difference between the average values of the blocks The second block determines the plural second ROI block.

Step 450: Adjust the power of the light source of the projector according to the average value of the blocks corresponding to the first ROI blocks and the average value of the blocks corresponding to the second ROI blocks.

As the principle and operation of the AE control device 100 have been described in detail above, the detailed description of each step shown in FIG. 4 is omitted here and will not be repeated here.

In summary, the present invention provides an automatic exposure control device and automatic exposure control method in a depth sensing system. The present invention obtains reasonable automatic exposure control for the projector and/or sensor in the depth sensing system by estimating the lighting condition of a scene. In order to quickly estimate the lighting conditions, the present invention uses pixel binning technology to read out signals from the sensing device to obtain a reference frame with a lower resolution, which can effectively reduce power consumption and computational burden. In addition, the present invention utilizes reference frames captured under different lighting conditions to analyze the difference between the intensity of the ambient light of the scene and the predetermined intensity of the light source of the projector, thereby determining the appropriate power of the light source for driving the projector. In order to obtain accurate lighting condition estimation results, the present invention uses an ROI determination procedure to eliminate information that is meaningless or that will hinder the estimation of the lighting conditions of the scene. By appropriately and accurately performing AE control, the depth sensing system can obtain more accurate depth information of the scene and avoid power consumption.

The embodiments of the present invention can be implemented using hardware, software, firmware, and related combinations thereof. With an appropriate command execution system, software or firmware stored in a memory can be used to implement the embodiments of the present invention. As far as the hardware is concerned, it can be accomplished by applying any of the following technologies or related combinations: a specific arithmetic logic with a logic gate that can perform logic functions based on data signals, a specific application integrated with a suitable combinational logic gate Circuit (application specific integrated circuit, abbreviated as ASIC), programmable gate array (abbreviated as PGA), or a field programmable gate array (abbreviated as FPGA), etc.

The flow and block diagrams in the flowchart show possible structures, functions, and operations of the system, method, and computer program product in various embodiments of the present invention. In this regard, each block in the flowchart or block diagram may represent a module, segment, or part of the program code, which contains one or more executable instructions for implementing specified logical functions. In addition, it should be noted that each block in the block diagram and/or flowchart illustration, as well as the combination of blocks, can be implemented through features that can perform specific functions or operations. It is realized by a special purpose hardware-based system, or it is realized by a combination of special function hardware and computer instructions. Computer program instructions can also be stored in a computer-readable medium, which can instruct a computer or other programmable data to function in a specific manner, so that the instructions stored in the computer-readable medium produce an article including the instruction means. The instruction device implements the functions/actions specified in the flowchart and/or block diagram blocks.

The foregoing descriptions are only preferred embodiments of the present invention, and all equivalent changes and modifications made in accordance with the scope of the patent application of the present invention should fall within the scope of the present invention.

10: Depth sensing system

12: Projector

14: Sensing device

15: readout circuit

16: processing circuit

18: Depth decoding circuit

20: Sensor

100: AE control device

120: Block average calculation unit

140: Block average difference calculation unit

160: ROI decision unit

180: AE control unit

Claims (11)

A device for performing automatic exposure (auto-exposure, AE) control in a depth sensing system including a projector, including: a sensing device for capturing a scene according to different lighting conditions. Reference frame and a second reference frame; a block average calculation unit for calculating the average value of the plurality of first blocks corresponding to the plurality of first blocks of the first reference frame, and calculating the second The average value of the plurality of second blocks corresponding to the plurality of second blocks of the reference frame; a block average difference calculation unit for calculating based on the average value of the first blocks and the average value of the second blocks The difference between the average values of the plural blocks; a region of interest (ROI) determining unit for determining the average value of the first block, the second block, and the block average Difference value, determining a plurality of first ROI blocks from the first blocks and determining a plurality of second ROI blocks from the second blocks; and an AE control unit for determining the values of the first ROI blocks The power of a light source of the projector is adjusted by corresponding to the average value of the plurality of blocks and the average value of the plurality of blocks corresponding to the second ROI blocks. The device described in claim 1, wherein the sensing device captures the first reference frame in ambient light, and captures the projection light of the projector with the depth sensing system The second reference frame. The device described in claim 1, wherein when the sensing device captures the first reference frame and the second reference frame, it is combined with a pixel from a sensor of the depth sensing system Ways to read out complex signals, and perform down-sampling operations on the read-out signals. For the device described in claim 1, wherein when the average value of a first block is higher than an average threshold, the ROI determination unit determines the first ROI blocks and the second ROI areas A first block corresponding to the average value of the first block and a second block associated with the first block are excluded. For the device described in claim 1, wherein when the difference between the average value of a block is lower than a difference threshold, the ROI determination unit determines the first ROI block and the second ROI A first block and an associated second block corresponding to the difference of the average value of the block are excluded. For the device described in item 1 of the scope of patent application, when the number of one of the second ROI blocks is 0 and an average value of the average values of the second blocks corresponding to the second blocks is lower than When an underexposure threshold is reached, the AE control unit determines that a lighting condition of the scene is an extremely dark condition, and accordingly increases the power of the light source of the projector of the depth sensing system. For the device described in item 1 of the scope of patent application, when the number of one of the second ROI blocks is 0 and an average value of the average values of the second blocks corresponding to the second blocks is higher than When an overexposure threshold is reached, the AE control unit determines a lighting condition of the scene as an extreme overexposure condition, and accordingly reduces the exposure time of a sensor of the depth sensing system. For the device described in item 1 of the scope of patent application, when the number of one of the second ROI blocks is not 0, the AE control unit determines that one of the lighting conditions of the scene is a partial overexposure condition and a good exposure condition Or an underexposure condition, calculating a first ROI frame average value and a second ROI frame average value, calculating an adjustment factor based on at least the second ROI frame average value, and adjusting the projector according to the adjustment factor The power of the light source and the exposure time of one of the sensors of the depth sensing system. The device described in item 8 of the scope of patent application, wherein the AE control unit calculates the first ROI frame average value by taking one of the average values of the blocks corresponding to the first ROI blocks , And the second ROI frame average value is calculated by taking an average value of one of the average values of the blocks corresponding to the second ROI blocks. The device described in item 8 of the scope of patent application, wherein among the second ROI blocks, a number of second ROI blocks having a block average value higher than an overexposure threshold is higher than a number of blocks Threshold, the AE control unit determines that the lighting condition is the partial overexposure condition; when there is the number of second ROI blocks whose average value is higher than the overexposure threshold among the second ROI blocks Less than the block quantity valve Value and the average value of the first ROI frame is higher than a minimum illumination intensity, the AE control unit determines that the illumination condition is the good exposure condition; and the AE control unit corresponds to the average value of the first ROI frame and the The adjustment factor is calculated by the ratio of one of the average values of the second ROI frame. For the device described in item 8 of the scope of patent application, when the number of second ROI blocks with a block average value higher than an overexposure threshold among the second ROI blocks is lower than a block number threshold And when the average value of the first ROI frame is lower than a minimum illumination intensity, the AE control unit determines that the illumination condition is the underexposure condition, and corresponds to one of the minimum illumination intensity and the average value of the second ROI frame The ratio calculates the adjustment factor.

Images