WO2024135722A1 - Imaging device and image data generation method - Google Patents

Abstract

An imaging device comprising a plurality of imaging elements (10) for each of which a gain value can be set and an image generation unit (30) that generates first image data (200A) and second image data (200B) on the basis of imaging information (100) output from the imaging elements (10) in one image capture, wherein: the imaging elements (10) include a first imaging element group (10A) set to a first gain value and a second imaging element group (10B) set to a second gain value larger than the first gain value; the image generation unit (30) generates the first image data (200A) on the basis of the first imaging information (100A) output from the first imaging element group (10A), and generates the second image data (200B) on the basis of the second imaging information (100B) output from the second imaging element group (10B); and the second image data (200B) has higher average brightness than the first image data (200A).

Description

Imaging device and image data generating method

This disclosure relates to an imaging device and an image data generation method.

Patent Document 1 discloses a vehicle lamp that employs a variable light distribution device known as an ADB (Adaptive Driving Beam), which uses a camera to detect the presence or absence of a shading target, such as an oncoming vehicle, and can dim or turn off the light in the area corresponding to the shading target.

International Publication No. 2021/070783

In a vehicle lamp such as that described in Patent Document 1, when an image is captured using a camera to detect an object, a camera (image sensor) equipped with HDR (High Dynamic Range) technology is used, which combines images captured with different exposure times to expand the dynamic range in order to eliminate whiteout and blackout problems in the captured image.
However, realizing HDR technology requires high-performance cameras, which increases the cost of vehicle lighting fixtures.

The present disclosure therefore aims to provide an imaging device and image data generation method that is simple in configuration and can easily detect objects.

An imaging device according to one aspect of the present disclosure includes:
A plurality of image pickup elements each capable of setting a gain value;
an image generating unit that generates first image data and second image data based on imaging information output from the imaging element in one imaging operation,
The imaging element includes:
a first imaging element group set to a first gain value;
a second imaging element group set to a second gain value greater than the first gain value;
The image generating unit includes:
generating the first image data based on first imaging information output from the first imaging element group;
The second image data is generated based on second imaging information output from the second imaging element group, and the second image data has a higher average luminance than the first image data.

In addition, the image data generating method according to the embodiment of the present disclosure includes:
1. An image data generating method for generating image data based on imaging information output from a plurality of imaging elements, each of which can have a set gain value, in one imaging operation, comprising:
generating first image data based on first imaging information output from a first imaging element group set to a first gain value among the imaging elements;
The second image data is generated based on second imaging information output from a second imaging element group among the imaging elements, the second imaging element group being set to a second gain value greater than the first gain value, and the second image data has a higher average luminance than the first image data.

The imaging device and image data generation method disclosed herein can provide an imaging device and image data generation method that can generate image data that is easy to detect objects with a simple configuration.

FIG. 1 is a block diagram of an imaging device according to an embodiment. FIG. 2 is a side view illustrating an example of an imaging element according to an embodiment. FIG. 3 is a front view illustrating an example of an imaging unit according to the embodiment. FIG. 4 is a flowchart showing a control form of the image data generating method according to the embodiment. FIG. 5 is a schematic diagram showing an image data generating method according to an embodiment. FIG. 6A is a diagram comparing an actual view ahead of a vehicle with image data generated by an image data generating device according to an embodiment. FIG. 6B is a diagram comparing an actual view ahead of the vehicle with image data generated by the image data generating device according to the embodiment. FIG. 6C is a diagram comparing an actual view ahead of the vehicle with image data generated by the image data generating device according to the embodiment. FIG. 6D is a diagram comparing an actual view ahead of the vehicle with image data generated by the image data generating device according to the embodiment. FIG. 7A is a diagram showing an actual view ahead of a vehicle and image data generated by an image data generating device according to an embodiment. FIG. 7B is a diagram showing an actual view ahead of the vehicle and image data generated by the image data generating device according to the embodiment. FIG. 7C is a diagram showing an actual view ahead of the vehicle and image data generated by the image data generating device according to the embodiment. FIG. 7D is a diagram showing an actual view ahead of the vehicle and image data generated by the image data generating device according to the embodiment.

[Details of the embodiment of the present disclosure]
Specific examples of the imaging device 1 according to the embodiment of the present disclosure will be described below with reference to the drawings. Note that the present disclosure is not limited to these examples, but is defined by the claims, and is intended to include all modifications within the meaning and scope equivalent to the claims.

The imaging device 1 according to this embodiment will be described in detail with reference to Figs. 1 to 3. Fig. 1 is a block diagram of the imaging device 1 according to this embodiment. In Fig. 1, the vehicle X has a vehicle control unit X1 and a vehicle lamp 3, and the imaging device 1 is provided in the vehicle lamp 3. As shown in Fig. 1, the imaging device 1 has an imaging unit 2, a signal processing unit 20, an image generation unit 30, and an image recognition unit 40. The imaging unit 2 has a plurality of imaging elements 10 and converts light from the outside into a signal. The signal processing unit 20 converts the signal received from the imaging unit 2 into imaging information 100 (see Fig. 5) required to generate an image. The image generation unit 30 converts the imaging information 100 received from the signal processing unit 20 into image data. The image recognition unit 40 detects an object from the image data generated by the image generation unit 30. Note that, although the image recognition unit 40 is illustrated as a part of the imaging device 1 in Fig. 1, the image recognition unit 40 does not have to be a part of the imaging device 1. For example, it may be a part of the vehicle control unit X1 mounted on the vehicle X.

2 is a side view showing an example of an image sensor 10 according to an embodiment. As shown in FIG. 2, the image sensor 10 has an on-chip lens 11, a color filter 12, a photodiode 13, and an amplifier 14. The on-chip lens 11 is a lens directly mounted on the photodiode 13 on which the color filter 12 is mounted. An optical image of a subject is incident on the photodiode 13 through the on-chip lens 11. The color filter 12 blocks light other than a specific wavelength range and transmits only light in a specific wavelength range. As a result, only light in the specific wavelength range is incident on the photodiode 13. The photodiode 13 counts photons incident through the on-chip lens 11 and the color filter 12, and outputs the counted number of photons as a signal value.
Furthermore, each of the image sensors 10 has an amplifier 14 that can amplify or attenuate the intensity of a signal when converting photons incident on the photodiode 13 into a signal. In this embodiment, one amplifier 14 is provided for each photodiode 13, and each amplifier 14 can set the degree of amplification and attenuation (gain) of the signal of the photodiode 13 individually.

3 is a front view showing an example of the imaging unit 2 according to the embodiment. As shown in FIG. 3, the imaging unit 2 is configured with a plurality of imaging elements 10 arranged in a lattice pattern, and one color filter 12 is arranged for each imaging element 10. In this embodiment, the imaging element 10 has a red imaging element 10r that receives red light, a blue imaging element 10b that receives blue light, and a green imaging element 10g that receives green light. In this embodiment, the green imaging element 10g that receives green light has a first green imaging element 10g1 and a second green imaging element 10g2. The red imaging element 10r is provided with a color filter R that transmits red light, the blue imaging element 10b is provided with a color filter B that transmits blue light, and the first green imaging element 10g1 and the second green imaging element 10g2 are provided with a color filter G that transmits green light.
The imaging section 2 in this embodiment has a Bayer array in which one unit formed of four imaging elements 10, namely a red imaging element 10r, a blue imaging element 10b, a first green imaging element 10g1, and a second green imaging element 10g2, is repeatedly arranged.

The imaging element 10 is divided into a first imaging element group 10A and a second imaging element group 10B. In this embodiment, the first imaging element group 10A is composed of the second green imaging element 10g2. The second imaging element group 10B is composed of imaging elements that are not included in the first imaging element group 10A, and in this embodiment, is composed of a red imaging element 10r, a blue imaging element 10b, and a first green imaging element 10g1.

Next, the operation of the imaging device 1 according to this embodiment will be described with reference to Figures 4 and 5. Figure 4 is a flowchart showing the processing of the signal processing unit 20 and the image generating unit 30 according to this embodiment. When the imaging unit 2 receives light from an object, the imaging unit outputs imaging information 100 according to the light. The signal processing unit 20 receives the imaging information 100 from the imaging unit 2 (STEP 1). Thereafter, the signal processing unit 20 extracts the first imaging information 100A acquired by the first imaging element group 10A from the imaging information 100 received from the imaging unit 2 (STEP 2) and transmits it to the image generating unit 30. The image generating unit 30 generates first image data 200A based on the first imaging information 100A received from the signal processing unit 20 (STEP 3). Thereafter, the signal processing unit 20 extracts the second imaging information 100B acquired by the second imaging element group 10B from the imaging information 100 received from the imaging element 10 (STEP 4) and transmits it to the image generating unit 30. The image generation unit 30 generates second image data 200B based on the second imaging information 100B (STEP 5).

5 is a schematic diagram showing a process in which the first image data 200A and the second image data 200B are generated from the imaging information 100. For the sake of explanation, the imaging information 100 transmitted from the imaging unit 2 (imaging element 10) is shown as data arranged in four rows and four columns in FIG. 5, but in reality, the imaging information 100 corresponds to the number of imaging elements 10. In FIG. 5, a certain piece of data is given a code representing the color of a color filter provided on the imaging element 10 that output the data, and a two-digit number corresponding to the row number and column number. In addition, the imaging information 100 corresponds to the arrangement of the imaging elements 10 shown in FIG. 3, and the grouping of the first imaging element group 10A and the second imaging element group 10B is the same as in FIG. 3. Therefore, for example, the imaging information 100 based on a signal emitted by the imaging element 10 arranged in the first row and first column of the imaging elements 10 shown in FIG. 3 corresponds to _G11 in the imaging information 100 shown in FIG. 5. Furthermore, for example, among the imaging elements 10 shown in FIG. 3, the imaging information 100 based on signals emitted by the imaging elements 10 arranged in the first row, first column, the first row, second column, and the second row, first column, which are imaging elements 10 belonging to the second imaging element group 10B, correspond to G ₁₁ , R ₁₂ , and B ₂₁ , respectively, in FIG. 5.

When the signal processing unit 20 extracts the first imaging information 100A from the imaging information 100, the signal processing unit 20 extracts G ₂₂ , G ₂₄ , G ₄₂ , and G ₄₄ , which correspond to the data received from the imaging elements 10 belonging to the first imaging element group 10A. The second imaging information 100B generated in this manner is output as data of two rows and two columns in the example of Fig. 5.

The image generating unit 30 generates first image data 200A based on the first imaging information 100A generated by the signal processing unit 20. The image generating unit 30 generates a single pixel from each piece of data in the first imaging information 100A. In this embodiment, the first imaging information 100A is composed only of signals received from the second green imaging element 10g2, which is provided with the same green color filter, so the first image data 200A is output as a grayscale image.

When the signal processing unit 20 extracts the second imaging information 100B from the imaging information 100, the second imaging information 100B extracts G ₁₁ , R ₁₂ , and B ₂₁ , which correspond to the data received from the second imaging element group 10B. The image generating unit 30 uses the second imaging information 100B extracted from the imaging information 100 to generate GRB ₁₁ , which is data constituting the second imaging information 100B. In the same manner as in generating GRB ₁₁ , the image generating unit 30 extracts G ₁₃ , R ₁₄ , and B ₂₃ to generate GRB ₁₂ , extracts G ₃₁ , R ₃₂ , and B ₄₁ to generate GRB ₂₁ , and extracts G ₃₃ , R ₃₄ , and B ₄₃ to generate GRB ₂₂ . In this manner, the signal processing unit 20 generates second imaging information 100B consisting of GRB ₁₁ , GRB ₁₂ , GRB ₂₁ , and GRB ₂₂ and arranged in two rows and two columns.

Using the above-mentioned method, the image generating unit 30 generates the second image data 200B based on the second imaging information 100B generated by the signal processing unit 20. The data _GRB11 can include color information because it is generated based on the data _G11 of the first green imaging element 10g1 provided with a green color filter, the data _R12 of the red imaging element 10r provided with a red color filter, and the data _B21 of the blue imaging element 10b provided with a blue color filter. The second image data 200B generated by such pixels is output as a color image or a grayscale image. In this embodiment, the second image data 200B is output as a color image.

In this embodiment, the gain set in the imaging elements 10 belonging to the second imaging element group 10B is set higher than the gain set in the imaging elements 10 belonging to the first imaging element group 10A. As a result, the second image data 200B generated using the signal transmitted from the second imaging element group 10B has a higher average luminance than the first image data 200A generated using the signal transmitted from the first imaging element group 10A. In other words, even when the images are of the same object, the second image data 200B generated from the second element group 10B tends to be a brighter image than the first image data 200A generated from the first element group 10A because the gain of the second element group 10B is higher than the gain of the first element group 10A.

For example, when a group of pixels having a brightness equal to or greater than a predetermined value has a specific size and shape and appears in a specific area, the image recognition unit 40 identifies the pixel as a head lamp (hereinafter referred to as HL) or a tail lamp (hereinafter referred to as TL). Alternatively, in the second image data 200B, which is a color image, the image recognition unit 40 identifies the pixel as an HL when a group of pixels having a brightness equal to or greater than a predetermined value is white, and identifies the pixel as a TL when the pixel having a brightness equal to or greater than a predetermined value is red. Note that the method of identifying HL or TL from an image by the image recognition unit 40 is not limited to the above embodiment.

In this way, the imaging device 1 of this embodiment generates first image data 200A and second image data 200B with different average luminances in a single shooting. How the HL of an oncoming vehicle and the TL of a leading vehicle are detected using such an imaging device 1 will be explained using Figures 6A to 6D and 7A to 7D.

In the following explanation, the light spot that appears in image data 200 and corresponds to HL is referred to as light spot Z1, and the light spot that appears in image data 200 and corresponds to TL is referred to as light spot Z2.

FIGS. 6A to 6C show a scene where there is an oncoming vehicle ahead of vehicle X. FIG. 6A is a diagram showing the state ahead of vehicle X. FIG. 6B is a diagram showing first image data 200A1 generated in the situation shown in FIG. 6A. FIG. 6C is a diagram showing second image data 200B1 generated in the situation shown in FIG. 6A. As described above, the first image data 200A1 is a grayscale image, and the second image data 200B1 is a color image. Furthermore, the average luminance of the second image data 200B1 is higher than the average luminance of the first image data 200A1. Note that average luminance means the average luminance of the previous pixels contained in the image.

As shown in Fig. 6C, in the second image data 200B1, pixels around the position corresponding to the HL of the oncoming vehicle are blown out. This is because the second image data 200B1 is generated based on data obtained from the imaging element 10 with a high gain setting. In the second image data 200B1, the light emitted from the HL is too bright, so the area around the HL is also bright and blown out. For this reason, the shape of the HL does not appear clearly in the second image data 200B1, and the image recognition unit 40 cannot identify the HL from Fig. 6C.
However, the imaging device 1 of this embodiment generates the first image data 200A1 in Fig. 6B at the same time as generating the second image data 200B1. This first image data 200A1 is generated based on data obtained from the imaging element 10 with a low gain setting. Therefore, when generating the first image data 200A1, the light emitted from HL is not too bright, and the shape of HL appears clearly in the first image data 200A1. Therefore, the image recognition unit 40 can identify HL from Fig. 6B.
6B and 6C, the average luminance of the second image data 200B1 is higher than the average luminance of the first image data 200A1 even though the images were taken at the same moment. The image recognition unit 40 can identify the light spot Z1 from each of the first image data 200A1 and the second image data 200B1.

FIGS. 7A to 7C show a scene where a vehicle ahead of vehicle X is present. FIG. 7A is a diagram showing the state ahead of vehicle X. FIG. 7B is a diagram showing first image data 200A generated in the situation shown in FIG. 7A. FIG. 7C is a diagram showing second image data 200B generated in the situation shown in FIG. 7A.

7A to 7C, unlike the example shown in Figures 6A to 6C, the image recognition unit 40 cannot identify the TL of the vehicle in front from the first image data 200A2 shown in Figure 7B, but can identify the TL from the second image data 200B2 shown in Figure 7C.
As shown in Fig. 7B, in the first image data 200A2, the TL of the vehicle ahead is too dark, and the light spot Z2 corresponding to the TL does not appear in the data obtained from the image sensor 10 with the gain set to low. Therefore, the image recognition unit 40 cannot identify the TL from Fig. 7B.
However, the imaging device 1 of this embodiment generates the second image data 200B2 of Fig. 7C at the same time as generating the first image data 200A2. This second image data 200B2 is generated based on data obtained from the imaging element 10 with a high gain setting. Therefore, when generating the second image data 200B2, the light emitted from the TL is not too dark, and the shape of the TL appears clearly as a light spot Z2 in the second image data 200B2. The image recognition unit 40 can identify the TL from Fig. 7C.

The image recognition unit 40 in the present disclosure recognizes HL and TL in the above-mentioned manner. The image recognition unit 40 transmits the recognition result to a vehicle control unit X1 provided in the vehicle X and a lamp control unit 3b provided in the vehicle lamp 3, and the vehicle control unit X1 and the lamp control unit 3b change the lighting state of the vehicle lamp 3 based on the recognition result of the image recognition unit 40. Here, "changing the lighting state of the vehicle lamp 3" means changing the light distribution pattern to be formed.
6D and 7D show a manner in which the image recognition unit 40 changes the lighting state of the light source 3a based on the result of recognizing HL and TL. In Fig. 6D and Fig. 7D, the light distribution pattern formed by the vehicle lamp 3 is shown by a hatched area.
In FIG. 6D, the image recognition unit 40 is able to identify the light spot Z1 (HL) from the first image data 200A1, so the lamp control unit 3b controls the light source 3a to block or dim the portion where the light spot Z1 is located, thereby forming the light distribution pattern Hi1 as shown in the figure.
In Figure 7D, the image recognition unit 40 is able to identify the light spot Z2 (TL) from the second image data 200B2, so the lamp control unit 3b controls the light source 3a to block or dim the area where the light spot Z2 is located, forming the light distribution pattern Hi2 as shown in the figure.
The lighting state of the vehicle lamp 3 incorporating the imaging device 1 according to the present disclosure can be changed as described above.

As described above, according to the imaging device of the present disclosure, among the data output from the imaging elements in one imaging operation, first image data is generated based on the first imaging information output from the first imaging element group, and second image data with high average luminance is generated based on the second imaging information output from the second imaging element group. Note that "high average luminance" here can also be rephrased as a high average pixel illuminance or a high average pixel gamma value.

In the past, the common way to eliminate blown-out highlights and crushed shadows in captured images was to use HDR (High Dynamic Range), which expands the dynamic range by synthesizing images captured with different exposure times. However, when HDR is installed in a camera, the system becomes complicated because images are captured with different exposure times, which can increase costs.

The inventor therefore considered generating image data that makes it easy to recognize objects without using HDR technology, by generating two types of image data with different brightness from the imaging information generated by a single imaging session.
According to the imaging device of the present disclosure, the imaging element 10 of the imaging device 1 is divided into a first imaging element group 10A and a second imaging element group 10B, and first image data 200A is generated from a signal generated by the first imaging element group 10A, and second image data 200B is generated from a signal generated by the second imaging element group 10B.
In addition, since the second imaging element group 10B has a higher gain set than the first imaging element group 10A, the second image data 200B generated from the second imaging element group 10B is capable of capturing dark objects brighter than the first image data 200A.

Therefore, according to the configuration of the present disclosure, first image data and second image data with different average luminance can be generated simultaneously with one shooting. By using image data with different average luminance, the luminance distribution can be expanded using two images. This effectively expands the dynamic range, making it possible to recognize bright objects while also being able to recognize dark objects. In this way, according to the configuration of the present disclosure, it is possible to provide an imaging device that is easily capable of detecting objects with a simple configuration.

Though the embodiments of the present disclosure have been described above, the aspects of the present disclosure are not limited to the above-described embodiments. For example, the first image data 200A and the second image data 200B may both be grayscale images. Even if the first image data 200A and the second image data 200B are both grayscale images, the average luminance of the first image data 200A and the second image data 200B is different, so that bright objects can be recognized as bright objects, and it is also possible to recognize dark objects that were previously difficult to recognize using image data generated by a single capture.

In addition, in the above-described embodiment, the image sensor 10 has a color filter 12, and the color filter 12 has a Bayer array composed of three colors, red, green, and blue, but the present disclosure is not limited to this. For example, a white filter may be included, or a filter that enables the image sensor to detect light other than visible light may be included. In addition, a configuration in which no color filter is provided at all may also be used.

In addition, in the above-described embodiment, an example has been shown in which the signal processing unit 20 and the image generating unit 30 create the first image data 200A from the imaging information 100 and then create the second image data 200B, but the present disclosure is not limited to this. In the present disclosure, it is sufficient that the first image data 200A and the second image data 200B are generated from imaging information obtained by a single imaging operation, and the first image data and the second image data may be generated sequentially or simultaneously.

The above describes an embodiment of the present disclosure, but it goes without saying that the technical scope of the present disclosure should not be interpreted as being limited by the description of this embodiment. This embodiment is merely an example, and it will be understood by those skilled in the art that various modifications of the embodiment are possible within the scope of the invention described in the claims. The technical scope of the present disclosure should be determined based on the scope of the invention described in the claims and its equivalents.

This application is based on Japanese Patent Application No. 2022-204744, filed on December 21, 2022, the contents of which are incorporated herein by reference.

Claims (8)

A plurality of image pickup elements each capable of setting a gain value;
an image generating unit that generates first image data and second image data based on imaging information output from the imaging element in one imaging operation,
The imaging element includes:
a first imaging element group set to a first gain value;
a second imaging element group set to a second gain value greater than the first gain value;
The image generating unit includes:
generating the first image data based on first imaging information output from the first imaging element group;
An imaging device that generates the second image data based on second imaging information output from the second imaging element group, the second image data having a higher average luminance than the first image data. 1. An image data generating method for generating image data based on imaging information output from a plurality of imaging elements, each of which can have a set gain value, in one imaging operation, comprising:
generating first image data based on first imaging information output from a first imaging element group set to a first gain value among the imaging elements;
An image data generating method, comprising: generating second image data based on second imaging information output from a second imaging element group among the imaging elements, the second imaging element group being set to a second gain value greater than the first gain value; and generating second image data having an average luminance higher than that of the first image data. An image data generating device that generates image data based on imaging information output from a plurality of imaging elements, each of which can have a set gain value, in one imaging operation, comprising:
generating first image data based on first imaging information output from a first imaging element group set to a first gain value among the imaging elements;
An image data generating device that generates second image data based on second imaging information output from a second imaging element group among the imaging elements, the second imaging element group being set to a second gain value greater than the first gain value, and the second image data has a higher average luminance than the first image data. the imaging element includes a color filter and a photodiode that receives light transmitted through the color filter;
the color filters provided in the first imaging element group are of the same color;
The imaging device according to claim 1 , wherein the color filters provided in the second imaging element group are of at least three types: red, blue, and green. The imaging device of claim 1, wherein the first image data is grayscale and the second image data is color. The imaging device of claim 1, wherein the first image data and the second image data are grayscale. A vehicle lamp that changes its illumination state based on light spots detected from the first image data generated by the imaging device described in claim 1 and light spots detected from the second image data. The vehicle lamp according to claim 7, which detects headlamp light spots from the first image data and detects taillamp light spots from the second image data.

Images