WO2003054579A1 - Method and device for producing 3d range images - Google Patents

Abstract

The invention relates to a method and a device for producing 3D range images which are calculated on the basis of a first and a second partial image and optionally an environmental image. An image point resolving optoelectronic sensor is provided, said sensor comprising a plurality of photosensitive cells which are respectively associated with an image point of the 3D image. At least two charge-coupled memories are, in turn, associated with each photosensitive cell, charges of the two partial images being stored in said memories. Said partial images are essentially simultaneously produced for the same directions, in such a way that the 3D range image can be calculated on the basis of partial images having the same spatial configuration, without parallax error.

Description

 Method and device for generating 3D Entferungsbi 1 dem

The invention relates to a method and a device for generating 3D distance images.

Methods and devices, for example in the form of sensor systems, are already known, with which distance images are obtained on the basis of a point-by-point determination of distances, which are calculated either on the runtime principle or by means of phase modulation of a laser beam, for example with scanners. However, the disadvantage here is the need to use mechanically moving parts to deflect the laser beam and the sequential determination of the individual pixels of an image. The latter leads in moving scenarios that the distance image represents a superposition of all occurring spatial configurations of the time interval that is required for the pixel-by-point recording of the distance image. Methods using pixel-resolving detectors such as CCDs have also been described.

A color-coded SD image recognition method is known from DE 44 47 117 C1, in which the three-dimensional position of the object points is calculated from the color distributions of the object points from two images, which are obtained by recording the object with light pulses generated by light intensity modulators, one of the light intensity modulators is switched to transmission in one picture and the other light intensity modulator in the other picture is driven with a suitable time delay. In this way, modulated light beams of different wavelengths are emitted onto the object and a light switch or light intensity modulator connected upstream of an image detector is actuated in such a way that the instantaneous modulation value of a reflected light beam detected thereby is a function of the different spatial position of the illuminated object point. As an alternative to pulsed operation, continuous periodic operation with suitably selected phase shifts of the light intensity modulators is also possible.

A method and a device are known from WO 99/34235, in which the object is briefly exposed, for example with laser diodes. An image sensor with high light sensitivity is used to record the image, which can be read out in a pixel-resolution manner and can be read out at will, and has an integration time that can be set for each pixel. By evaluating the backscattered light pulses in two integration windows with different integration times and by averaging over several light pulses, three-dimensional distance images can be generated.

Common disadvantages of these methods or devices are that the pixel values that are required for the calculation of the respective distance image points are derived from the echo of different light pulses and are either generated in chronological order and / or are obtained in different light-sensitive cells.

Since the measured values for calculating the distance pixels (within the distance images) are recorded sequentially and therefore at different times, the spatial configuration of the scene to be measured may have changed in the meantime.

Due to the temporal offset between the exposure processes, a calculation error occurs in the case of moving scenery or a moving sensor for all distance pixels that are affected by the movement due to incorrect standardization.

In addition, the higher need for light pulses leads to an unnecessarily high emitted light output, so that problems with eye safety may arise.

If the acquisition of the measured values for calculating the distance pixels in different If light-sensitive cells of a pixel-resolving sensor occurs, an incorrect standardization can result even when the scenery is still.

Furthermore, it is known from DE 43 05 011 C2 to integrate backscattering pulses in at least two successive time subintervals and to calculate distances from the integrated signals of the subintervals.

The invention is therefore based on the object of providing a method and a device for generating SD distance images which can be recorded in particularly short time intervals and with a significantly lower light output.

Furthermore, the invention is intended to provide a method and a device for generating 3D distance images, with which all the measurement values required for generating distance image points can be obtained in particularly short time intervals by the scattering of a single laser pulse.

Finally, the invention is also intended to create a method and a device for generating SD distance images, in which all measured values for a distance image point can be obtained in the same light-sensitive cell.

This task is solved with a method for generating 3D distance images from the following steps: Emitting a light pulse in the direction of an object of a scenery to be imaged; Imaging an object on a light-sensitive and spatially resolving sensor, the cells of which generate charges corresponding to the exposure and registration of the light pulses scattered back from the various object points within a time interval M, which is dimensioned and shifted in time relative to the start of the light pulse transmission such that the backscatter pulses for everyone Pixels are completely or partially detected depending on their running time, - storing the charges generated during a first subinterval (M of the time interval (M) in a first line memory assigned to the relevant pixel as well as during a second subinterval subsequent to the first subinterval (M) (M ₂ ) of the time interval M generated charges in a charge memory assigned to the same pixel, and calculating a 3D distance image from the charges stored for each pixel in the first and second charge memories.

The object is further achieved with a device for carrying out the method with the following combination of features, a light source for emitting a light pulse in the direction of an object to be imaged

Scenery;

Means for imaging the object on a light-sensitive and spatially resolving sensor whose cells correspond to the exposure

Generate loads and means of registration of the light pulses scattered back from the various object points within a time interval (M), which is dimensioned and shifted in time relative to the start of the light pulse transmission in such a way that the backscattered pulses for each pixel are recorded in whole or in part depending on their transit time, - at least first and second Charge stores, which are each assigned to the cells of the sensor, for storing charges generated by the cell in question during a first subinterval (M) and during a second subinterval (M ₂ ) of the time interval (M) following the first subinterval (M _x ) , - and means for calculating a 3D distance image from the charges stored for each pixel in the associated charge stores.

The invention is therefore based on the knowledge that by using an image-resolving optoelectronic sensor, the individual image points of which are provided with a plurality of charge stores, an extremely fast device for obtaining distance images can be created with simultaneous complete use of the energy of the emitted light pulses, with all measured values can be obtained with a light pulse to determine the distance image. One advantage of this solution is that the light energy emitted can be reduced to a minimum in the interest of eye safety through its optimal use. In addition, no mechanically moving parts are required.

Another advantage of this solution is that, if necessary, the shooting processes can be repeated at short intervals, so that multiple exposures and charge integrations are possible during the taking of an image. Even with multiple exposures, no calculation errors can occur due to temporal sequences, since the accumulated partial signals are error-free relative to one another.

The subclaims have advantageous further developments of the invention.

According to claim 3, a third charge storage device is assigned to each light-sensitive cell in order to correct the influence of stray light on the measurement signals obtained, so that an environmental image can be generated that contains these stray light components.

Further details, features and advantages of the invention will become apparent from the following description of a preferred embodiment with reference to the drawing. It shows :

1 is a schematic representation of a basic structure for generating 3D images according to the invention, 2 shows a schematic representation of a first memory architecture of a sensor,

3 shows a schematic illustration of a first generation of different partial images;

4 is a timing diagram;

Fig. 5 different diagrams for calculating a standardized quantity Q as a measure of the distance,

6 shows a schematic illustration of a second generation of different partial images; and

Fig. 7 is a schematic representation of a second memory architecture of a sensor.

Figure 1 shows schematically the structure with which distance images (3D images) are obtained according to the invention. Light pulses 2 are emitted from a light source 1 onto an object 3 (or a scenery), which are reflected there. Depending on the scattering and reflection properties of the object 3, the light pulse 2, with a correspondingly reduced identity, returns as a scattered light pulse 4 to a recording camera 5 and falls there through a lens 6 onto an optoelectronic, pixel-resolving element

(Pixel-resolving) sensor 7. The intensity values detected by the sensor are evaluated with a signal evaluation device 8 in the manner described below, and a distance image (3D image) is calculated therefrom. The light source 1 is preferably a structural component of the camera 5.

FIG. 2 shows the basic architecture and organization of the pixel-resolving optoelectronic sensor 7. The sensor comprises a light-sensitive cell Z _1D for each pixel, to which a first, a second and a third charge store S ^ _k (k = 1, 2, 3) are assigned , The light-sensitive cells are arranged or organized in rows (index i) and columns (index j) according to the geometry of the 3D image to be generated.

The sequence of the method according to the invention and the function of the device for SD image generation according to the invention will now be described with reference to FIGS. 3 and 4.

The distance image (3D image) is calculated from two partial images, an environmental image possibly also being determined and taken into account when calculating the distance image.

By emitting a light pulse 2 with the length L at time t ₀ according to FIG. 4b through the light source 1 preferably mounted on the camera 5 in the direction of the object 3, both partial images are generated. According to FIG. 3a, the light pulse is reflected at different object points 10, 11, 12 of the object, which have different distances z within a range between a minimum distance z _min and a maximum distance z ,, _^ from the light source 1. To cover these different distances, the backscattered ones are Light pulses 4 are detected within a first time interval M> L (FIG. 4d), ie the camera 5 is activated synchronously with the emission of the light pulses 2 at a time t _x for the first time interval M. The length and the temporal position of the first interval M is advantageously chosen such that the light pulses scattered at the relevant object points 10, 11, 12 depend on their transit time, ie on the distance of the object points from the light source, within the first time interval M can be completely captured.

Figure 3b shows the time interval M and two subintervals M _λ and M ₂ and schematically the temporal position of the backscattered light pulses. The light backscattered from the object point 10 is completely detected in the first subinterval M _x . The backscattered light of the object point 11 partly falls into the first subinterval M and partly into the second subinterval M ₂ , while the light backscattered from the object point 12 is no longer detected in the first subinterval M _x , but falls completely into the second subinterval M ₂ , The charges generated during the detection are collected in charge detectors S _{1D of} the detector, the charges generated in the first subinterval M _x in S _ι (FIG. 4e) and the charges generated in the second subinterval M ₂ in S ^ ₂ (FIG. 4f) get saved. This means that the backscattered light pulse is trimmed to different _extents by the subintervals of M in accordance with its transit time, and thus the transit time of the light echo influences the charge _captured in the charge _stores S _13l , S _1-2 . The detection process can be for all light-sensitive cells Z _{1D of} the detector can be carried out simultaneously, with only a single emitted light pulse being required to obtain all the data required for calculating the distance image if the intensity is sufficient.

Since not only the transit time of the backscattered light, but also the generally different scattering properties of object surfaces influence the charges registered in the subintervals, normalization of the detected echo intensities is necessary. For normalization, the light echo intensity detected as a charge can be used within a partial interval M or the total interval M of the detector pixel in question. If, for example, the charges q _{x: 1} in the charge _store S _{1D1 are normalized} with the charges q _{ι: 2} in the charge _store S _1D2 , then a size can be obtained for the pixel ij, which represents a measure of the transit time of the light pulse and so that the distance of the corresponding object point to the detector represents.

In addition, by combining all the charges detected during the sub-intervals M _λ and M _2, a size can be obtained for each pixel that is independent of the transit time of the light pulse for all objects in the measurement interval and can thus serve as a gray-scale pixel.

If the object to be displayed is exposed to an intense stray light such as sunlight, it is furthermore advantageous or depending on the intensity relationships between the light pulse and stray light to generate an environmental image that contains these stray light components (FIG. 4a) and is used to correct the partial images. For this purpose, the environmental light is detected for each pixel during a second time interval K according to FIG. 4c, that is to say the charges generated in each light-sensitive cell Z ₁₃ during the second time interval K are stored in the third charge range S.- ₃ assigned to the cell in question ,

The time interval K for the acquisition of the environmental light image is advantageously chosen such that stray light components of the light pulse from objects outside the scene under consideration have no or only a negligible influence and are as close as possible to the interval M in order to represent the environmental light conditions applicable at that time as precisely as possible to ensure. The time interval K is preferably set immediately before the emission of the light pulse (FIG. 4c).

If the intensity of the light pulse is sufficient, the entire amount of information for a distance image is obtained with a light pulse in a time interval of, for example, 50 ns. This means that the method according to the invention or a corresponding device is also suitable for recording very fast processes.

With a suitable architecture of the optoelectronic sensor, in order to improve the signal-to-noise ratio, it is possible to repeat this process several times and to accumulate the charges generated by several light pulses in the charge stores of the light-sensitive cells. For example, when using laser diodes with a duty cycle of 1% as the light source, a pulse length of 10 ns and an image frequency of 100 Hz, it is possible to accumulate the charges of 10,000 light pulses. The exact value depends on the readout time of the optoelectronic sensor.

Each pixel of the distance image is then calculated from the respective first to third charge stores of each light-sensitive cell.

FIG. 5 shows two possible ways of calculating a quantity Q from the contents q _{1Dk of} the charge _store S _{ι: k} , which is monotonously dependent on the distance of the respective object point. 5a and 5b are the charges registered in the subintervals M. and M ₂ and the charges q _{11 l} q captured in the charge _stores S _{13l /} S _1-2 . _-2 displayed depending on the distance of the object point. FIG. 5c shows the charge q _1D3 which is collected in the charge store S. ₃ during the interval K. In the interval K there is no echo of the incident light pulse, only ambient light is registered.

The following equations I and II can serve as examples for the compensation of the ambient light, the normalization to the scattering properties of the object and the irradiated light intensity to obtain the quantity Q:

[Eq. I]: Q _τ = {q _13l - q (t ₂ t _x ) / (t _s t ₄ ⁾ } / {q_ ₃ - - q- ,, ₃ (t ₃ - t ₂ ) / (t ₅ - t ₄ ) } and

[Eq. II]: Q _I {_ ₃ 2 - q _lD3 (t ₃ ^■ t ₂ ) / (t ₅ t ₄ ⁾ } / {q _13l + q-. _D 2 - _1D 3 (t ₃ t / (t ₅ - t ₄ )} The curves of the functions Q- (z) and Q _ΣI (z) are shown schematically in _FIGS . 5d and 5e. Other types of standardization by swapping the roles of q _1Dl and q _1D - or by forming 1 / Q ^ or 1 / Q _{ι: r also} lead to monotonous functions that can be used to determine the distance.

In order to deduce the distance z from the normalized quantity Q, the inverse function of Q (z) has to be used, which can be done, for example, via a lookup table operation. The gray image, which is independent of the distance, can be calculated from the sum of the charges q _13l and q.- ₂ for each pixel.

In a further embodiment of the invention, the measurement interval is broken down into more than two subintervals and each light-sensitive cell of the sensor is accordingly provided with a corresponding number of memories.

With an advantageous choice of the length of the subintervals and a given size of the charge stores, the dynamics of the method can be increased, for example. FIG. 3c shows schematically the decline in backscattered identity with increasing distance of the object points 10, 11, 12 and an exemplary division of the measurement interval into three sub-intervals M ₁ ', M''and M ₂ . The distance images can then be determined, for example, completely analogously to the method shown in equation (1) or equation (2) if q _13l = q _ι '+ q _ι ''is set and q _13l ' and q _{X] 1} '' represent the charges accumulated in the intervals M 'and M ₁ ''. Another method for determining distance images becomes particularly interesting when the interval M is broken down into a larger number of subintervals. An object 11 (FIG. 6a), which is illuminated by a light pulse 2, generates the backscattered light pulse 4, which is detected by each light-sensitive cell Z. ^ during the interval M and in the n sub-intervals M _x - M _n (FIG. 6b) depending on the arrival time at the detector either charges or no charges are generated. The charges obtained during each subinterval are transferred to the associated charge _stores S _{1Dk of} the photosensitive cell Z ₁₃ . Figure 7 shows schematically the architecture of the corresponding sensor.

To determine a distance pixel, a check is carried out to determine whether the memory contents exceed a threshold value and whether there is a signal. If this is the case, the value 1 is assigned to all such memories and the value 0 to all other memories. The distance for each pixel can then be determined from the known duration of the partial intervals and the phase reference for the emission of the light pulse by determining the first memory of each pixel, the content of which exceeds the threshold value (FIG. 7). The threshold value can, for example, be set to a constant value for all pixels or can be dynamically adapted to the scenery for each pixel.

For a constant duration of the partial intervals M _x - M _n , the distance for each distance _pixel E _i3 is: z (E _X3 ) = z ₀ + zk '

where z (E _{ι:) is} the distance in the pixel E ₁₃ , k ^{1 is} the index of the first memory of the corresponding light-sensitive cell, the signal of which is above the threshold value, and z represents the change in distance belonging to the duration of the subintervals.

In summary, this embodiment of the invention has the following advantages:

The light generated with the light pulses is fully used. Furthermore, the three image types mentioned above are each obtained at the same location, that is to say at the same image points of the optoelectronic sensor, so that the image points of the distance image are calculated from charge densities that represent or depict identical object points and that no parallax errors occur. Since the three types of images mentioned can be obtained essentially simultaneously, that is to say from a single incident light pulse, the respective charges also relate to identical spatial configurations.

Claims

P a t e n t a n s r u c h e Method for generating 3D distance images, characterized by the following steps: Emitting a light pulse in the direction of an object of a scenery to be imaged; Imaging an object on a light-sensitive and spatially resolving sensor, the cells of which generate charges corresponding to the exposure and registration of the light pulses scattered back from the various object points within a time interval M, which is dimensioned and shifted in time relative to the start of the light pulse transmission such that the backscatter pulses for everyone Pixels can be captured in whole or in part depending on their duration; Storage of the charges generated during a first subinterval (M) of the time interval (M) in a first line memory assigned to the relevant pixel as well as the charges generated during a second subinterval (M ₂ ) of the time interval M following the first subinterval (M _x ) a charge store assigned to the same pixel; and Calculate a 3D distance image from the charges stored for each pixel in the first and second charge memories. 2. The method according to claim 1, characterized in that the charges generated during the first and second subinterval are stored substantially simultaneously. 3. The method according to claim 1 or 2, characterized by recording an environmental image by detecting light emanating from the object within a second time interval before the light pulse is emitted for each pixel and storing a charge density generated thereby in a third charge memory associated with the pixel in question Compensation of stray light when calculating the 3D distance image. 4. Device for generating SD distance images with: a light source for emitting a light pulse in the direction of an object to be imaged in a scenery; - Means for imaging the object on a light-sensitive and spatially resolving sensor, the cells of which generate charges corresponding to the exposure, and means for registering the light pulses scattered back from the various object points within a time interval (M), which is dimensioned in this way and temporally compared to the start of the light pulse transmission is shifted that the backscatter pulses for each pixel are fully or partially detected depending on their duration; - At least first and second charge stores, which are each assigned to the cells of the sensor, for storing by the cell in question during a first sub-interval (M) and during a second, to the first sub-interval (M _x ) subsequent subinterval (M ₂ ) of the time interval (M) generated charges, - and - Means for calculating a 3D distance image from the charges stored in the associated charge stores for each pixel. 5. The device according to claim 4, characterized in that each light-sensitive and spatially resolving sensor is associated with a third charge storage (S. _D3 ) for detecting charges generated by an environmental image.