WO2020254445A1 - Stereo camera system - Google Patents

Abstract

The present invention relates to a stereo camera system for measuring three-dimensional surface topologies. In this way, depth information of objects, which move along on a conveying device in front of a stereo camera system, can be detected. The stereo camera system can be a component of a production monitoring device.

Description

Stereo camera system

The present invention relates to a stereo camera system for measuring dreidimensiona len surface topologies. With this, depth information of objects moving on a conveyor in front of the stereo camera system can be recorded. The stereo camera system can be part of a production monitoring device.

Production monitoring devices with a conveyor device and a camera for optically scanning the products conveyed on the conveyor device are known and are used, for example, in package sorting systems or production systems.

Usually, a measuring device, such as. B. a light curtain, the height of the product or object to be scanned is determined. The camera has an adjustable lens that is set to a previously detected height of an object to be scanned in order to correctly scan its surface. The camera can have a zoom lens whose focal length is variable, so that it can be adjusted accordingly for scanning objects of different heights. It is also possible to use a lens with a fixed focal length, the focus of which is tracked so that different heights are sharply imaged on the sensor.

With zoom lenses, several lenses are moved relative to one another. The zoom lens has a movable mechanism within the lens that moves several lenses towards one another. With a zoom lens, the scale can be kept constant over the adjustment path.

If a lens with a fixed focal length is used, the entire lens must be shifted along the optical axis so that the focus area is at the desired height. The image scale increases when the optical length is reduced. Both a zoom lens and a fixed focal length lens have mechanical parts that need to be moved. These movements are subject to wear. In industrial applications, usage cycles of several million are required. These usually far exceed the life of these moving parts.

If height information is to be evaluated in the product monitoring device in addition to the optical information, various methods have become established.

One possibility is the so-called light section method. A thin, straight strip of light is thrown onto the object to be measured and an area camera then records this strip of light. The surface of the object can be calculated back using the distortion that the object generates on the light strip.

Another option is stereoscopy. A camera system records the object from two different viewing directions and can determine the topological surface information of the object due to the offset of corresponding image points. However, the disadvantage of this method is the restriction to one focal plane.

Such stereo camera systems have two camera modules with which the same object area can be scanned from two different viewing directions. The camera modules are provided with a lens in order to image the object area on a sensor of the respective camera module. An optical axis of the lens is an axis of symmetry of the lens, which runs approximately transversely to the surfaces of one or more lenses contained in the lens. The main plane of the objective on the object side is considered to be the objective plane. A sensor plane is the plane in which the light-sensitive sensor surface lies. An object plane is the plane that is sharply imaged from the lens onto the sensor surface.

In conventional stereo camera systems, the lens plane, the object plane and the sensor plane are usually arranged parallel to one another. The parallel alignment of the object plane to the lens plane and to the sensor plane is referred to below as the parallelism arrangement.

A base angle α, which is the angle from the viewing directions, is decisive for the resolution of a three-dimensional surface topology with a stereo camera system the two camera modules of the stereo camera system is stretched over the object area. The base angle a is essentially determined by the size of a base B, which is the distance between the two sensor surfaces of the two camera modules S1, S2. This means that the larger the base B, the larger the base angle α and the correspondingly better the height resolution of the stereo camera system (FIG. 8).

Unless otherwise stated, the following is based on the assumption that the distance between the sensor plane SP (sensor plane), the objective plane LP (lens plane) and the object plane OP (object plane) remains constant, so that the base B alone determines the base angle a.

Lenses have an opening angle y which indicates the maximum angle of incidence of the incident light to the optical axis, within which an object can be imaged essentially without distortion (FIG. 8). Depending on the design of the lens, the maximum opening angle of the lens can vary. In the following, the beam path is referred to as the beam through the lens, which includes all beams within the opening angle and which extend between the sensor surface and the object.

In the case of a stereo camera system with the parallelism arrangement explained above, however, the base cannot be selected to be arbitrarily large, since if the base is too large, the sections in the object area limited by the angle of incidence only overlap slightly or not at all, and thus the sharply mapped section of the object area small or nonexistent.

Only an overlapping area, i.e. the area of the object that is detected by both sensors at the same time, can be used for an evaluation of the three-dimensional surface topology. The edge areas of this overlap area are imaged at a large opening angle of the objective y. The distortion is therefore greatest in the edge areas of the overlap area. Furthermore, the imaging performance, which can be characterized, among other things, by a color error and a modulation transfer function (MTF), is worse in the edge area of the overlap area than in the center of the overlap area. The color error describes the loss of color reproduction. The MTF describes a loss of detail in contrast.

In "Scheimpflug Stereocamera for particle image velocimetry in liquid flows", Ajay K. Prasad and Kirk Jensen, APPLIED OPTICS 34, 30 (1995), a stereo video camera with a mpflug condition and an area sensor. The system is used to determine the velocity distribution of small particles in liquids.

With such a Scheimpflug camera, the parallelism arrangement is canceled by tilting the lens plane LP relative to the sensor plane, so that a large base is possible without the object area being restricted by the limited angle of incidence. In a Scheimpflug camera, the so-called Scheimpflug condition is met, according to which the lens plane, the sensor plane and the object plane intersect in a common cutting axis. A Scheimpflug angle β is spanned by the object plane OP and the sensor plane SP (FIG. 9). If the Scheimpflug condition is met, the object area is sharply mapped onto the sensor by the lens. In the literature, a cutting axis is sometimes described at an infinite distance from the stereo camera system. In this special case, the Scheimpflug condition would correspond to the parallelism arrangement. For the following consideration, however, the Scheimpflug condition is not met if these planes or axes intersect at infinity. They must therefore intersect at a finite point so that the Scheimpflug condition is fulfilled. One or more mirrors can be provided in the beam path, whereby the beam path is folded and the actual lens plane, the sensor plane and the object plane do not intersect in a common cutting axis. The Scheimpflug condition is also met with such an arrangement if the objective plane, the sensor plane and the object plane intersect in a common sectional axis with an unfolded beam path without the respective mirrors but with an identical arrangement of the respective objective, sensor and object.

With stereo camera systems with Scheimpflug condition, on the other hand, a wider base angle can be selected than with stereo camera systems without Scheimpflug condition with the same objectives and image quality. With a large base, tilting the lenses can reduce the angle of incidence and thus optimize it. Lenses with a small opening angle y are therefore also suitable. Due to the smaller opening angle, the condition is that the image quality, especially the MTF, is more constant over the overlap area than with stereo camera systems without Scheimpflug condition. A linear distortion along the overlap area is accepted as this can be corrected by rectification. The larger the base selected, the more the sensor plane and lens plane have to be tilted relative to the object plane in order to reduce the angle of incidence. Another problem with stereo camera systems without Scheimpflug condition is back reflection. Due to the parallel arrangement with a narrow base angle, the lens not only images the object on the sensor, but also the sensor on the object. This image of the lens can be imaged back onto the sensor by reflection on a reflective surface of the object. The sensor thus records its own mirror image. By tilting it, for example by Scheimpflug, a broad base angle can be selected, whereby such back reflection is avoided. Even if the reflection continues, the image of the reflection can also represent a neutral background, for example. The effects of reflection are thus reduced.

Further measuring systems with a stereo camera in which the Scheimpflug condition is fulfilled emerge from WO 2014/017977 A1, US Pat. No. 6,671,397 B1 and WO 02/040 970 A1.

WO 2014/017977 A1 discloses a method and a device for determining the coplanarity in integrated circuit housings. To determine planarity, two sensors, each with two beam paths, are arranged in such a way that they follow the Scheimpflug principle.

WO 02/040 970 A1 discloses a measuring system with a camera with an objective and a separate sensor. The measuring system has a first camera with a lens and a separate sensor, which are mounted so that their planes intersect in an object plane according to the Scheimpflug principle. A second camera is arranged perpendicular to the object plane. The camera systems are area cameras.

From WO 02/040 970 A1 an optical method and a device for the inspection of large planar objects emerge. In one embodiment, two area cameras record a wafer, which are arranged under Scheimpflug condition.

The advantages of stereo camera systems with Scheimpflug condition are explained above. However, with stereo camera systems with Scheimpflug condition, effects occur which negatively influence the quality of the stereoscopy when using area sensors. This is due to the so-called keystone effect, which describes the distortion of the projected sensor surface. A two-dimensional image of the object area is recorded with an area sensor. In conventional stereo camera systems, one edge of the area sensor is arranged parallel to the base. The main axes of the recorded two-dimensional image thus also extend either parallel to the base and transversely to the base. The direction parallel to the base is hereinafter referred to as the base direction and the direction transverse to the base is hereinafter referred to as the transverse direction.

When the object area is imaged by a stereo camera system, the image is compressed both in the transverse direction and in the base direction. This compression is caused by the fact that the individual points of a flat object are spaced differently from the sensor surface. The further two points are away from the sensor surface, the smaller the distance between the points is shown. The compression becomes stronger the larger the base is. A transverse compression occurs in the transverse direction and a base compression occurs in the base direction.

For example, a square object area with the smallest possible base and a central arrangement of the camera systems is mapped almost distortion-free. With a large base, however, the image of the square forms the shape of a trapezoid or keystone. The distance between the corners that are further away from one another is shown smaller than the distance between the corners that are closer to the objective.

The effect occurs with stereo camera systems both with and without Scheimpflug condition. In the case of those without Scheimpflug condition, this effect is also limited by the limitation of the basis. Since stereo camera systems with Scheimpflug condition can have a particularly large base, the effect is particularly pronounced here.

The keystone effect causes a loss of resolution. The resolving power is the minimum distance that two point objects must have so that they can be perceived as separate objects. As a result of the compression, two points that are further away from the sensor surface are mapped closer together. From a certain distance from the lens, these points can no longer be perceived as separate. The resolving power parallel to the base is hereinafter referred to as the basic resolving power and the resolving power across the base is hereinafter referred to as the transverse resolving power. Furthermore, the keystone effect increases the computational effort. In order to use the images for an evaluation of the three-dimensional surface topology, both the base and the transverse compression must be calculated from the image data by rectification. Rectification is a process for eliminating geometric distortions in image data that can arise, for example, from central perspective recordings. As a result of the rectification, the individual pixels are shifted in such a way that when the pixels of the two sensors overlap, the same position on the object is displayed. Furthermore, the distortion is corrected so that the calculated disparity is linearly dependent on the height profile of the object.

In stereoscopy, each point of an image of a camera module is assigned a point of an image of the second camera module. A rectangular sensor detects a trapezoid TI of the object surface when the sensor is inclined relative to the object due to the keystone effect (FIG. 10). The other area sensor detects a second trapezoid T2 of the object area. When assigning the points, these trapezoids overlap, the long front side coinciding with the short front side of the other trapezoid. This results in a hexagonal overlap area (hatched in FIG. 10).

The disadvantage here is that part of the information recorded is lost from the non-overlapping areas. Only the overlap area can be used to evaluate the three-dimensional surface topology.

Furthermore, the opening angle of the lens must be large enough that the overlap area is completely covered. This is the case when a diagonal D of the hexagonal overlap area is captured by the lens.

DE 10 2015 11 11 20 A1 discloses a method for scanning surfaces with a stereo camera with at least two camera modules. The camera modules are arranged in such a way that they each capture an image of a common area of a surface to be scanned. The camera modules can have a line camera with several different sensor lines. A separate line image is recorded with each sensor line. The individual line images are superimposed by means of a geometric mapping. Stereo camera systems with line sensors and a parallelism arrangement have the same problems as stereo camera systems with area sensors and a parallelism arrangement. As described above, the base cannot be chosen to be arbitrarily large, since if the base is too large, the sections in the object area limited by the angle of incidence only overlap slightly or not at all and the sharply mapped section of the object area is therefore small or no longer present .

In principle, stereo camera systems are conceivable that meet neither the Scheimpflug condition nor the parallelism arrangement. Thus, only the object plane OP and the sensor plane SP could be parallel to one another, while the objective plane LP is tilted to the other two planes in such a way that the object area is not restricted by the limited angle of incidence. An imaging plane, which corresponds to the plane on which the objective images the object planes in focus, is arranged at an angle to the object plane or the object axis. The object area is imaged sharply on the camera sensor only in the intersection area of the imaging plane and the sensor plane (FIG. 11). The greater the tilt of the objective axis relative to the object axis, the narrower is a focus area in which the object is sharply imaged. The greater the depth of field TS, the wider the focus area. With a narrow focus area, sufficient information is only available from a small section of the object for an evaluation of the three-dimensional surface topology.

However, a sharp image is very important for an image of the three-dimensional surface topology. It is also desirable to use the entire surface of a sensor.

In the following, stereo camera systems without Scheimpflug condition mean stereo camera systems which have the parallelism arrangement.

A system is known from "Line cameras for inspecting the sealing surfaces of an O-ring groove", Photonik 5/2012, in which three line cameras are operated in parallel in order to image one side of a groove each. The side walls of the groove are plowed by line cameras with shear -Condition shown. From US Pat. No. 7,724,362 B1, a system for macro wafer inspection with an oblique incidence emerges. The wafers are captured by a line camera that is directed at an angle to the wafer, the beam path fulfilling the Scheimpflug condition.

DE 10 2013 103 897 A1 discloses a camera module for scanning an object line by line. The camera module comprises a cellular sensor and a lens for imaging the object on the sensor. The camera module has several line sensors. The line sensors are arranged at different distances from the lens, whereby image lines with different distances from the lens are formed on the respective line sensors.

The invention is based on the object of creating a stereo camera system with which a good spatial resolution is effected over a large scanning area in a simple manner.

Another object of the invention is to create a stereo camera system which allows reliable scanning of spatial objects.

Another object of the invention is to create a stereo camera system with which a large image area can be captured in a simple manner.

An additional object of the invention is to provide a stereo camera system that is compact.

Another object of the invention is to create a stereo camera system that images various object planes well.

One or more objects are achieved by the subjects of the independent claims. Advantageous developments and preferred embodiments form the subject matter of the subclaims.

A stereo camera system according to the invention for measuring three-dimensional surface topologies of an object has at least two cell-shaped sensor areas and at least one lens for imaging an object onto the sensor areas. The objective is arranged in such a way that an object area is located on two independent beam paths is mapped onto a sensor area. The stereo camera system is characterized in that the Scheimpflug condition is fulfilled along at least one of the beam paths.

As already explained, a stereo camera system, taking into account the Scheimpflug condition, can have a larger base than a stereo camera system with a parallel arrangement without the images being recorded by the lens. The larger base increases the resolution of the three-dimensional surface topology.

Such a stereo camera system can be used to scan an object area line by line transversely to the line direction, the stereo camera system being moved relative to the object. The direction parallel to the line sensors is referred to as the line direction and the direction transverse to the line sensors is referred to as the scanning direction. The distance between two points that lie in the scanning direction is recorded with the same resolution regardless of the keystone effect and the position along the line direction. The image is not compressed along the scanning direction. This increases the resolution along the scanning direction.

Trapezoidal formation is also prevented. Both line sensors can cover the same area with the entire sensor line. No information is lost. The overlap area of line sensors is called the overlap line in the following.

In the case of a line sensor, the opening angle of the lenses only needs to encompass the overlapping line. With the same sensor length, the overlap line is smaller than the diagonal of the overlap area of two area sensors, which, as described above, forms a hexagonal shape. As a result, the opening angle required for a stereo camera system with Scheimpflug condition and area sensors is greater than with a stereo camera system with Scheimpflug condition and line sensors. In the case of a stereo camera system with Schei mpflug condition and line sensors, this results in less distortion than with a stereo camera system with Scheimpflug condition and area sensors.

By reducing the compression, the required rectification is also reduced. This reduces the computational effort and the image data is thus available more quickly. In summary, it can be stated that by using stereo camera systems with Scheimpflug conditions, the base can be selected to be large, without distortions occurring as a result of stereo camera systems without Scheimpflug conditions. A large base allows precise height resolution. The Scheimpflug condition allows small opening angles of the lenses with good image quality. However, the Scheimpflug condition causes unwanted distortions in both directions in area sensors due to the keystone effect. When using line sensors, the distortion only occurs in one direction, which means that the quality of the image is higher. In addition, the required opening angle of the lenses is smaller for line sensors than for area sensors, since the projected width of the line sensor is smaller than the projected diagonal of the area sensors. In addition, in contrast to area sensors, all image points of the sensor can be used.

Therefore, with a stereo camera system that satisfies the Scheimpflug condition and has line sensors, images can be generated with comparatively simple means that enable a high height resolution.

The Scheimpflug condition is preferably fulfilled along all beam paths. In principle, the beam paths can have different arrangements of the sensor areas and lenses. Because all beam paths meet the Scheimpflug condition, a larger base can be selected than with stereo camera systems in which only one beam path fulfills the Schei mpflug condition.

The beam paths are preferably arranged mirror-symmetrically. Adjustment of the camera system is simplified by the mirror-symmetrical arrangement. Asymmetries are easily noticed during the adjustment and can be corrected. The evaluation of the image data is also simplified because each sensor is equidistant from the object area. The recorded pixels are thus also offset by the same distance.

One or more of the beam paths are preferably steered with at least one mirror. This allows the beam paths to be folded by mirrors. The image of the measuring object is reflected once or several times by mirrors, which means that the length of the beam path is retained. However, the volume that is taken up by the arrangement of the optical lenses is in some cases considerably reduced. In particular, a large base for the stereo camera system can be selected and the two sensor halves still remain sealed together. This means that more compact designs are possible than with structures without a beam path folding.

When folding the beam path, a distinction can be made between an object-side folding in which the mirrors that fold the beam path are arranged between the lens and the object, and a sensor-side folding in which the mirrors that fold the beam path between the lens and the sensor are arranged, and a double-sided folding in which the mirrors that fold the beam path, are arranged both between lens and object and between lens and sensor's rule. With object-side folding, the working distance, the distance between the lens and the object, can be reduced.

The beam paths are preferably projected onto a separate part of a single sensor. These parts each form a sensor area of one of the at least two camera modules. The sensor can be divided into several sensor areas, each of which is assigned a beam path. For example, in a stereo camera system with two beam paths, the sensor is divided into two areas. If the stereo camera system has three beam paths, the sensor is divided into three parts. This also increases the system stability. In addition, it enables a compact design with simple adjustment.

Alternatively, the camera can be designed in such a way that the beam paths are each projected onto a separate sensor and that a mirror is provided for each beam path and the beam path is configured in a Scheimpflug arrangement. This design enables a compact form with a larger sensor area than with a single sensor. This increases the resolution.

Another alternative is characterized in that the beam paths are each projected onto a separate sensor and that two mirrors are provided for each beam path and the beam path is configured in a Scheimpflug arrangement. This construction enables a compact shape.

The cell-shaped sensor areas are preferably arranged next to one another on a line, each of which is represented by a line sensor. Adjustment can be simplified by this line arrangement, since the cell-shaped sensor areas can be attached to a common carrier. For example, if the carrier tilts, the sensor areas do not have to be realigned to one another. In the case of stereoscopic methods, it is useful if the offset takes place in only one direction. This means that the sensors are arranged parallel or perpendicular to a sensor axis. This is supported by the line arrangement.

According to a further modification, the cell-shaped sensor areas can be arranged parallel to one another. Here, too, the two beam paths are only offset in one direction.

It is possible for the cell-shaped sensor areas to be lines of pixels of an area sensor. An existing stereo line camera system can be easily retrofitted.

A common objective can be provided for all beam paths. In this way, particularly compact stereo line camera systems can be implemented. By reducing the number of lenses, costs can also be kept low.

The beam paths can have different distances between the object plane and the sensor plane. In this case, it can be expedient to compensate for the optical path due to the different distances by means of one or more glass elements which is / are arranged in the respective shorter beam paths.

The speed of the light inside the glass element is slowed down by the glass element. This means that the light needs a longer time to reach the sensor. This has the same effect as an optical path extension. Different distances can thus be compensated.

According to a further aspect of the invention, a stereo line camera system for measuring three-dimensional surface topologies comprises at least one color sensor, the color sensor having several pixels that are sensitive to different colors, and at least one hyperchromatic lens is provided for imaging the object on the sensor. The color sensor is arranged in relation to the lens in such a way that different focal planes are mapped onto the image points of the same color.

With hyperchromatic objectives, the focus depends on the wavelength. Each color line of the sensor is thus assigned to a different focal plane. This can be used to measure three-dimensional surface topologies. Each different height of the object area has a different focal plane. This means that each of the heights is sharply mapped on the sensor in a specific color range. The resolution of the height corresponds to that of the color resolution. This procedure is referred to below as hyperchromatic depth determination. It is particularly suitable for perceiving small differences in height.

The hyperchromatic depth determination is preferably used together with the stereoscopic method with the Scheimpflug condition. With the hyperchromatic depth determination, fine height differences are recorded, whereas with the stereoscopic method height differences are detected over a larger area.

The color lines are preferably tilted in relation to the lens. The tilting takes place around an axis that is parallel to the individual color lines. The Scheimpflug condition is retained along a line. This tilting changes the distance between a color line and the lens, which also shifts the focal plane. In this way, the maximum difference in height that can be measured by the hyperchromatic depth determination can be varied effectively.

The color channels of the color sensor are preferably evaluated separately. This allows the different focus levels to be calculated separately.

According to a further aspect of the invention, a stereo line camera for measuring three-dimensional surface topologies comprises at least two area sensors, the area sensors being designed in such a way that they have several parallel rows of pixels arranged next to one another, and at least two lenses for imaging the object on one each of the area sensors. The stereo line camera is characterized by the fact that an object area with two independent beam paths is imaged simultaneously on one of the sensors, so that a cell-shaped object area is imaged on the image point lines of an area sensor, with the beam paths in the line plane that the cell-shaped Object area intersects, the Scheimpflug condition is met, and that the area sensor is tilted around an axis in such a way that the area is not parallel to an object area to be scanned, so that the individual lines of the area sensor map different levels of the cell-shaped object area. As a result of the tilting, each line of image points has a different distance from the lens. The Scheimpflug condition is retained along a line. Each line thus has its own focus level. Each different height of the object area is thus sharply imaged on a different pixel line. The resolution of the height corresponds to the line resolution. This method is called inclination-part depth determination in the following (Latin: inclinatio = inclination).

The maximum difference in height, which can be measured by determining the inclination, can be varied by changing the tilt.

The resolution of the altitude can be improved by combining the inclination-based depth determination with the stereoscopic method with Scheimpflug condition. The height is now recorded using two different methods, which reduces a measurement error.

The sensors are preferably color sensors and the lens is a hyperchromatic lens. As a result, the individual focus planes that arise from the tilting of the surface sensor can be subdivided again, resulting in a finer resolution of the focus planes.

The image data are preferably rectified in-line. Here, the image data of the images of the stereo image pair are matched line by line, with both an offset and a different distortion being corrected by the two lenses. Through this rectification, which is also possible off-line, the position information is adapted to the respective distortion, so that the image data are displayed without distortion and can be further calculated. Since a further conversion step of the position information is not necessary, the speed of the calculations increases.

Line sensors with charge-coupled device (CCD) sensors have shift registers in which detected brightness values are stored. These shift registers are read out serially. In general, when reading out the shift register of the individual lines, uniform position information is assigned to the respective image points. The pixel lines of sensors are preferably synchronized in such a way that they record line-synchronously and have a simultaneous image start. This significantly reduces the effort involved in rectifying. In particular in combination with rectification, a recorded image point of a camera module is assigned the same point on the object at the same time as a second recorded image point of the second camera module. This has the advantage that, during the stereo calculation, each line of a sensor area is automatically assigned the appropriate line of the second sensor area. This means that the stereo calculation is only necessary in one dimension.

Another aspect of the present invention relates to a surface topology detection device. The surface topology detection device has at least one stereo camera system, as explained above, in order to detect three-dimensional surface topologies of a measurement object. The surface topology detection device comprises a transport device for transporting the measurement object or the camera system, a synchronization device in order to synchronize the speed of the transport with the cell-shaped scans of the three-dimensional surface topology, and an evaluation device in order to evaluate the measurements of the three-dimensional surface topology.

A transport device can for example be a conveyor belt.

The synchronization device outputs a signal to the stereo camera system and / or to the evaluation unit in order to synchronize the speed of the transport with the measurement of the three-dimensional surface topology. If the signal goes to the stereo camera system, the signal can trigger a measurement. If the signal reaches the evaluation unit, the speed of the transport device can be determined. With a known image generation frequency, it can be determined how far one image line is from the next.

The synchronization device can, for example, be an incremental encoder that is triggered by the transport device. If the transport device is a conveyor belt, the wheel position can be determined by the synchronization device, for example by a wheel that is coupled to the conveyor belt. A signal is then issued for certain wheel positions. Alternatively, the synchronization device could have a camera unit which detects markings on the conveyor belt and uses them to output a signal. The synchronization device can also be designed as a module of the evaluation unit. If the image generation frequency and the speed of the transport device are known, the distance between two recorded image lines can be determined by calculation.

The evaluation unit is a module on a computing unit and calculates the three-dimensional surface topology from the recorded image data.

The invention is explained in more detail below using the examples shown in the drawings. The drawings show schematically in a side view:

Figure 1 shows a first embodiment of a camera with two lenses and two

Sensors,

Figure 2a shows a second embodiment of a camera with two lenses,

four mirrors and one sensor,

Figure 2b shows a modification of the second embodiment of a camera with two

Lenses, four mirrors and two sensors,

FIG. 2c shows a further modification of the second exemplary embodiment of a camera with two color filters, two lenses, four mirrors and two sensors,

Figure 3 shows a third embodiment of a camera with two lenses,

two mirrors and two sensors,

FIG. 4 shows a hyperchromatic objective, with different focal planes falling on different colored lines,

Figure 5 shows a fourth embodiment of a camera with three lenses,

four mirrors and a color line sensor,

Figure 6 shows a modification of the fourth embodiment of a camera with a

Lens, four mirrors, a glass element and a color line sensor,

FIG. 7 shows a surface topology detection device which uses a stereo camera according to the invention, in a block diagram,

FIG. 8 shows a schematic arrangement of sensors and lenses of two camera modules of a stereo camera which does not correspond to the invention,

FIG. 9 shows a schematic arrangement of sensors and lenses of two camera modules of a stereo camera, and FIG FIG. 10 projected sensor areas on an object plane, and

FIG. 11 shows a surface topology detection device which uses a stereo camera according to the invention, in a block diagram.

In a first exemplary embodiment, a stereo camera system 25 according to the invention comprises two camera modules 1 a and 1 b (FIG. 1).

Each camera module 1 is formed from an objective 2 and a line sensor 3. The line sensors 3 each detect with a sensor area 13 an image imaged thereon by the objective 2. The two camera modules 1 are arranged such that they scan a common object area 6 from two different viewing directions.

In each camera module 1, a camera axis 5 runs through a center point of the sensor area 13 and a center point of the lens 2. The two camera axes 5 thus run in the viewing direction of the sensor areas 13 to the object area 6.

The two camera axes 5 lie in an optical plane. The two line sensors 3 or their sensor areas 13 also lie in this optical plane. This optical plane is congruent with the plane of the drawing in FIG.

The base B is the distance between the centers of the two sensor areas 13. The camera axes 5 intersect in an object axis 10 and limit the base angle a.

Since the camera system uses line sensors 3, only a sensor axis 8, an objective axis 9 and an object axis 10 are relevant for further consideration, which are the straight lines in the optical plane and at the same time in the sensor plane defined above, the objective plane or the Object level.

The line sensor 3 records an image of the object region 6, which is imaged onto the line sensor 3 by the objective 2. The line sensor 3 is a CCD sensor and comprises a shift register and a one-dimensional array of photo detectors, each photo detector serving to record a pixel. For each line sensor 3, the sensor axis 8 is congruent with the sensor line. The lenses 2 of the camera modules 1 are arranged in relation to the respective line sensor 3 in such a way that the object area 6 is imaged sharply in the plane of the line sensor 3. The objective 2 can be formed from a single lens or from several lenses. An optical axis of the objective 2 is an axis of symmetry of the objective 2, which runs perpendicular to the objective axis 9 in the optical plane.

The object area 6 lies on the object axis 10 and is imaged sharply on the sensor areas 13.

The sensor axis 8, the objective axis 9 and the object axis 10 intersect at a common point which, however, is not at infinity. The Scheimpflug condition is thus fulfilled.

A second exemplary embodiment (FIG. 2a) is explained below, the same elements as in the first exemplary embodiment being provided with the same reference numerals. The explanations given above apply to the same elements, unless otherwise stated below.

In the second exemplary embodiment, a stereo camera system 25 according to the invention again comprises two camera modules la and lb. Each camera module 1 comprises an objective 2, two mirrors 11, 12 and a line sensor area 13. The camera axis 5 is deflected by the two mirrors 11, 12. The line sensor area 13 of each camera module 1 a and 1 b lies in a line behind one another and together forms a line sensor 3.

The mirrors 11, 12 are arranged between the objective 2 and the line sensor area 13 in such a way that the object area 6 falls on the line sensor area 13.

The sensor axis 8 is mirrored on the mirror 11. The sensor axis 8 ′ (not shown) is the mirrored sensor axis 8. The sensor axis 8 ′ is in turn mirrored on the mirror 12. The sensor axis 8 ″ (not shown) is the mirrored sensor axis 8 ′ or the double mirrored sensor axis 8.

A base B of the stereo camera system 25 is given by the distance between the centers of the projections of the two line sensors 3, the projection of a line sensor The place is where the line sensor 3 would be if the image of the lens area 6 would not be mirrored.

The sensor axis 8 ″, the objective axis 9 and the object axis 10 intersect at a common point which, however, is not at infinity. The Scheimpflug condition is thus fulfilled.

The second exemplary embodiment differs from the first exemplary embodiment in that two mirrors 11, 12 per camera module 1 a, 1 b deflect the foci of the object region 10 in such a way that they lie on a common axis, the respective sensor axis 8. Thus, no two separate sensors are required, but both beam paths can be projected onto a line sensor 3, which is divided into two line sensor areas 13 un.

In this exemplary embodiment, the sensor axis 8 is parallel to the object axis 10. However, it is also conceivable that these two axes are not parallel. Both camera modules are then not constructed with mirror symmetry.

Alternatively, the stereo camera system 25 has a distance between the two lines of sensor areas 13 (FIG. 2b). Each line sensor 3 now includes a line sensor area 13. This exemplary embodiment has certain similarities to the hyperscope, which however is not subject to the Scheimpflug conditions.

A further possibility of this embodiment is that the two optical axes intersect, so that the first image of the object area 6 is projected onto the opposite line sensor 3b or line sensor area 13b. Without the Scheimpflug condition, this arrangement is known as a pseudoscope.

Another embodiment is that the stereo camera system 25 has a structure according to the second embodiment (Fig. 2c), but the sensor area 13 are not arranged along the sensor axis 8 in a line next to each other, but perpendicular to the sensor axis 8 parallel to each other. As will be explained in more detail below, these sensor areas 13 can each be formed by the different color lines of a line sensor 3. Both beam paths are arranged offset from one another perpendicular to the line color sensor 3 in such a way that the two different beam paths hit line color sensor 3 slightly offset. The two mirrors 11 are arranged next to one another. . A color filter 26 is then positioned along each beam path. Thus, the image of a camera module la hits the line color sensor 3 as a red image, for example, and the image of the other camera module lb hits the line color sensor 3 as a green image, for example. As a result, a line color sensor 3 can simultaneously record two or even several images of the object area 6 . The surface of the object should not be in the color of one of the color filters 26, since otherwise the image information on the surface will be filtered out. The beam path with the color filter of a first color thus hits the sensor line that detects the first color. The beam path with the color filter of the second color hits the sensor line that detects the second color.

Object surfaces that are white or have gray levels are preferred. White or grayscale have the same intensity values for each color in the color spectrum. If the light passes through the respective color filter 26, the integrated intensity of the light is reduced across all colors, regardless of the color of the color filter. If the color channels of the line sensor 3 are evaluated as gray levels, the resulting images of the two beam paths differ in their gray level intensity only due to the different viewing angles to the object and not due to the different color of the beam path.

In this exemplary embodiment, the color filters 26 are arranged in front of the objective 2a and 2b (FIG. 2c). However, they can also be arranged between mirror 12 and objective 2a, 2b or between mirror 11 and 12 or between line color sensor 3 and mirror 11.

Alternatively, instead of the color filters 26, different polarizing filters can be used along the beam path, provided that the line sensor 3 can record images of the object region 6 with correspondingly differently polarized images. Each beam path of the stereo camera system 25 thus has a different polarization when it hits the line sensor 3. If the phase and the intensity of the images are recorded, this method can also be used in holography. The surface of the object must not change the polarization, since otherwise the polarizing filter will filter out image information that is necessary to calculate the height information.

A third exemplary embodiment (FIG. 3) is described below, the same elements as in the first and second exemplary embodiments with the same reference numerals are provided. The explanations given above apply to the same elements, unless otherwise stated below.

In the third exemplary embodiment, a stereo camera system 25 again comprises two camera modules la and lb. Each camera module 1 comprises an objective 2, a mirror 11 and a line sensor 3. The camera axis 5 is deflected by the mirror 11.

The mirror 11 is arranged between the lens 2 and the line sensor 3 in such a way that the object area 6 falls on the line sensor 3.

The sensor axis 8 is mirrored on the mirror 11. The sensor axis 8 '(not shown) is the mirrored sensor axis 8. The two sensor axes 8 of the two camera modules 1 are not necessarily parallel to one another in this exemplary embodiment.

A base B of the stereo camera system 25 is given by the distance between the centers of the projections of the two line sensors 3, the projection of a line sensor being the location where the line sensor 3 would be if the image of the lens area 6 were not mirrored.

The sensor axis 8 ', the objective axis 9 and the object axis 10 intersect at a common point which, however, is not at infinity. The Scheimpflug condition is thus fulfilled.

In contrast to the second exemplary embodiment, the two line sensor areas 13 are no longer in a line one behind the other. Furthermore, by means of the mirrors 11, the sensors 3 can be arranged perpendicularly or almost perpendicularly to the object area 6, as a result of which a very compact design can be selected.

A fourth exemplary embodiment (FIG. 5) is explained below, the same elements as in the first exemplary embodiment being provided with the same reference symbols. The explanations given above apply to the same elements, unless otherwise stated below.

In the fourth exemplary embodiment, a stereo camera system 25 according to the invention comprises three camera modules la, lb and lc. Instead of a simple line sensor, the stereo camera system 25 a three-line sensor 15. The three-line sensor 15 comprises three parallel sensor areas 13 arranged next to one another. The sensor axes 8 are arranged perpendicular to the optical plane.

Each camera module 1 a and 1 c includes an objective 2, two mirrors 11, 12 and one of the three line sensor areas 13. The camera module lb includes an objective 2 and one of the three line sensor areas 13. The camera modules 1 scan a common object area 6 from three different viewing directions .

In the case of the outer camera modules 1 a and 1 c, the sensor axis 8 is mirrored on the mirror 11. The sensor axis 8 '(not shown) is the mirrored sensor axis 8. The sensor axis 8' is in turn mirrored on the mirror 12. Sensor axis 8 ″ (not shown) is the mirrored sensor axis 8 ', or the double mirrored sensor axis 8.

If three camera modules 1 are used in a stereo camera system 25, then three basic lengths B are possible for each combination of two camera modules 1. A base B is given by the distance between the center points of projections of the two line sensors 3, the projection of a line sensor being the location where the line sensor 3 would be if the image of the lens area 6 were not mirrored.

The sensor axis 8 or 8 ″, the objective axis 9 and the object axis 10 intersect at a common point which, however, is not at infinity. The Scheimpflug condition is thus fulfilled.

Alternatively, instead of the three lenses 2a to 2c, the stereo camera system 25 comprises only a single lens 2, which encloses all three camera modules la, lb and lc. To compensate for this, the middle camera module 1b has a glass element 19 (FIG. 6).

The fourth exemplary embodiment differs from the preceding exemplary embodiments in that the line sensor areas 13 are arranged parallel to one another and not in a line one behind the other.

In an alternative embodiment of the above-mentioned exemplary embodiments, the objective of each camera module 1 is a hyperchromatic objective 14, the focus of which is particularly dependent on the respective wavelength (see FIG. 4). Furthermore, the Three-line sensor 15 is a three-line color sensor 15, which has a red 16a, green 16b and blue line 16c. Due to the hyperchromatic objective 14, the respective red, green and blue lines (16a, 16b, 16c) of the three-line color sensor 15 each have a different focal plane 17a, 17b and 17c. As a result, each color line 16a, 16b and 16c measures a different height measuring area 18a, 18b and 18c. The resolution of the fleas corresponds to that of the color resolution.

The color lines 16 are preferably tilted in relation to the objective 14. The tilting takes place around an axis which is parallel to the individual color lines 16. The Scheimpflug condition is retained along a color line 16. This tilting changes the distance of a color line 16 to the lens 14, whereby the focal plane 17 is shifted ver.

Another possibility of the invention is that the camera modules 1 have an area sensor instead of a line sensor 3. The area sensor is a two-dimensional array of photodetectors, each photodetector corresponding to one pixel. In this exemplary embodiment, the area sensor is viewed as a series of parallel Zeilensenso Ren 3.

The area sensor is tilted about a tilt axis. The tilt axis is parallel to the individual line sensors 3. The individual lines and their associated sensor axes 8, or 8 'or 8 "are arranged in such a way that they intersect the objective axis 9 and the object axis 10 at one point and thus the Fulfill the Scheimpflug condition.

The amount of the angle between the normal of the area sensor and the optical axis is less than 90 °. Each line of the area sensor has a different flea measuring area 18. The adjacent flea measuring areas lead to an expansion of the flea area. The same camera structure can therefore be used for objects with different fleas. Each flea of the object is assigned at least one corresponding line of the area sensor with the associated flea measurement area.

In an alternative embodiment of the above-mentioned exemplary embodiments, instead of a straight mirror 11, mirrors curved in such a way are used so that the images of the measurement object 1 impinge on the line sensor 3 without distortion. The above-mentioned exemplary embodiments each include two or three camera modules 1. In principle, any number, but at least two camera modules can be provided for stereoscopic determination of surface topologies. The higher the number of camera modules, the smaller the statistical error. However, the costs also increase. The number of base lengths results from the triangle number. For example, with two camera modules 1, only one base is possible, with three camera modules 1 there are three, with four there are six options and with five there are ten different base lengths.

A surface topology detection device 20 is explained below (FIG. 7).

The surface topology detection device comprises a stereo camera system 25, as it was carried out in one of the above exemplary embodiments according to one of FIGS. 1 to 6, for measuring three-dimensional surface topologies of a measurement object 4, a transport device 21 for transporting the measurement object 4, a synchronization device 22, to synchronize the speed of the transport with the measurement of the three-dimensional surface topology, and an evaluation device 23 to evaluate the measurements of the three-dimensional surface topology.

In this exemplary embodiment, the transport device 21 is a conveyor belt.

In this exemplary embodiment, the synchronization device 22 is an incremental encoder 22. The incremental encoder is coupled to the conveyor belt 21 so that it rotates when the conveyor belt moves forward. A clock signal is output when rotating through a predetermined angle of rotation. The conveyor belt thus covers a predetermined distance between two successive clock signals. These clock signals are transmitted to an evaluation device 23. The evaluation device 23 is also connected to the chamber modules 1 in such a way as to receive the image signals recorded by the sensors 3 arranged in the chamber modules. The evaluation device 23 is designed in such a way that it controls the point in time at which the respective camera modules scan the measurement object 4. In this way, the evaluation device 23 can scan the measurement object 4 with the movement of the conveyor belt in accordance with the clock signal received from the incremental encoder 22. of the measurement object 4. This synchronization is preferably formed in such a way that the measurement object is scanned in the direction of movement 7 with the same local distance. Several consecutive one-dimensional scans can be can be combined to form a two-dimensional image in the evaluation device 23. Thus, three-dimensional information is obtained from a surface topology.

The captured images can be corrected. On the one hand, a displacement error can be eliminated that arises from the fact that the measurement object 7 is moved relative to the camera and the individual pixels of the line camera are successively switched to be sensitive. As a result, at high speeds the image has a certain oblique position to the direction of movement 7.

Furthermore, the images can be rectified. The rectification eliminates geometric distortions in the image data. These can arise from the Scheimpflug arrangement, among other things.

The correction of the rectification can also be carried out directly in-line during the image registration. The data is corrected immediately after an image line has been recorded. Here at, after the sensor signal has been digitized, the image signals are preprocessed in an FPGA (Field Programmable Gate Array) on the camera. If the parameters of the directory of the two camera modules are known, e.g. from a calibration of the system, the same correction model can be applied in the FPGA as conventionally on a PC. A further correction in the arithmetic unit is therefore not necessary, whereby the processing speed is increased.

The images corrected in this way are used for a depth reconstruction of the recorded surface of the measurement object 4. A typical method for depth reconstruction is stereo triangulation. Each point in the first image is assigned a second point in the second image. The distance between the two points depends on the actual spatial depth of the point, more precisely the distance of the sensor 3 to the object area 6 and the base B of the stereo camera system 25. Each point of the first image is assigned a depth information.

Alternatively, blocks, i.e. a group of pixels, e.g. a 3x3 matrix to be assigned to. This process is called block matching.

For each recorded image line, an image line containing depth information is calculated. If several image lines of a measurement object 4 are recorded, the lines can len are combined with the depth information in order to generate a three-dimensional surface topography of the measurement object. The distances between the individual lines can be calculated using information from the synchronization device 22.

If instead of several line sensors 3 line sensor areas 13 are used, the image data of all camera modules 1 are generated on the same line sensor 3. The resulting digital double image is divided in the evaluation device 23. The resulting two images are corrected as described above and used for the depth construction.

If a three-line color sensor 15 is used to record the measurement object, a separate image is generated for each color. As described above, the independent images are corrected separately for each color and used for the depth construction profile of the target surface can be calculated. Each point of an image is assigned a point in the other two color area images, but of the same camera module 1. The position of the point does not differ between the three images. For this, the image sharpness of the point is different depending on the depth of the real point on the measurement object 4 and the color. If a point is deep, it is shown in focus in the blue image, in the case of medium depth in the green and in the case of low depth in the red image. A sharpness detection can e.g. can be carried out by comparing the contrast of the images in a specific pixel region. If a point is sharper, the contrast is higher.

In combination with the stereoscopic calculation, a precise determination of the surface topology is possible.

As an alternative to the individual calculation of the individual color images, a gray-scale image can also be calculated from a composite RGB image, which is then analyzed.

If a surface sensor is used to record the measurement object, an independent image is generated for each line. As described above, the independent images are corrected separately for each line and used for the depth construction. The sharpness of a real point on the measurement object 4 is then determined in each independent image. In this way the line can be determined which recorded the point most sharply. Since the position of the line and the focal length are known, the distance to the point can be become true. If all points are connected to one another, a height profile of the measurement object 4 is created.

In addition to the three-line sensors 15 mentioned above, other combinations are also conceivable, e.g. a sensor with three lines of RGB sensor points, a sensor with three lines of sensor points that also detects gray levels, a sensor with three lines of sensor points that only detects gray levels, etc.

The sensors mentioned above can, as mentioned above, be CCD sensors, but other types of sensors, such as, for example, complementary metal-oxide-semiconductor (CMOS) sensors, are also conceivable.

Bezuaszeichen

1 camera module

2 lens

3 sensor

4 target

5 camera axis

6 Contract area

7 Shift direction

8 sensor axis

9 lens axis

10 object axis

11 mirror 1

12 mirror 2

13 Line sensor area

14 Hyperchromatic objective

15 three-line sensor

16 color line

17 focus plane

18 Height measuring range

19 glass element

20 Surface topology acquisition device

21 Transport device

22 Synchronization facility

23 Evaluation device

25 stereo camera system

26 color filters

a base angle

ß Scheimpflug angle

y opening angle of the lens

B base

Claims

1. Stereo camera system for measuring three-dimensional surface topologies of an object at least two cell-shaped sensor areas and at least one lens for mapping the object onto the sensor areas, wherein the sensor areas and the objective (s) are arranged in such a way that an object area with two independent beam paths is imaged simultaneously on one of the sensor areas, characterized, that the Scheimpflug condition is met along at least one of the beam paths. 2. Stereo camera system according to claim 1, characterized, that the Scheimpflug condition is fulfilled along all beam paths. 3. Stereo camera system according to claim 1 or 2, characterized, that one or more of the beam paths are directed with at least one mirror. 4. stereo camera system according to one of claims 1 to 3, characterized, that the beam paths are each projected onto a separate part of a single sensor, which is divided into the number of beam paths. 5. Stereo camera system according to one of claims 1 to 3, characterized, that the beam paths are each projected onto a separate sensor and that at least one mirror is provided per beam path and the beam path is configured in a Scheimpflug arrangement for each beam path. 6. Stereo camera system according to one of claims 1 to 3, characterized, that the beam paths are each projected onto a separate sensor and that two mirrors are provided per beam path and the beam path per beam path in Scheimpflug arrangement is configured. 7. stereo camera system according to one of claims 1 to 6, characterized, that the cell-shaped sensor areas are arranged side by side on a line, each of which is represented by a line sensor. 8. Stereo camera system according to one of claims 1 to 5, characterized, that the cell-shaped sensor areas are arranged parallel to one another. 9. stereo camera system according to claim 8, characterized, that the cell-shaped sensor areas are lines of pixels of an area sensor. 10. Stereo camera system according to one of claims 1 to 9, characterized, that a common lens is arranged in all beam paths. 11. Stereo camera system according to one of claims 1 to 10, characterized, that a glass element is arranged in at least one of the beam paths. 12. Stereo camera system for measuring three-dimensional surface topologies egg nes object, in particular according to one of claims 1 to 11, with - At least one color sensor, wherein the color sensor has several pixels that are sensitive to different colors, - at least one hyperchromatic objective for imaging the object on the sensor, characterized, that the color sensor is arranged in relation to the objective in such a way that different focal planes are mapped onto the image points of the same color. 13. Stereo camera system according to claim 12, characterized, that color channels of the color sensor are evaluated separately. 14. Stereo camera system for measuring three-dimensional surface topologies of an object - at least two area sensors, the area sensors being designed in such a way that they have several parallel rows of pixels arranged next to one another, - At least two lenses for imaging the object on one of the surface sensors, characterized, that an object area with two independent beam paths is imaged simultaneously on one of the sensors, that a cell-shaped object area is mapped onto the pixel lines of an area sensor, wherein the Scheimpflug condition, in particular according to one of claims 1 to 11, is met in the line plane of the beam paths which intersect the cell-shaped object region, and that the area sensor is tilted about an axis in such a way that the area is not parallel to an object area to be scanned, so that the individual lines of the area sensor map different planes of the cell-shaped object area. 15. Stereo camera system according to claim 14, characterized, that the sensors are color sensors and the objective is a hyperchromatic objective according to claim 11 or 12. 16. Stereo camera system according to one of claims 1 to 15, characterized, that the recorded data are registered in-line in order to correct the rectification after. 17. Stereo camera system according to one of claims 1 to 16, characterized, that the pixel lines of sensors are synchronized in such a way that so that they record line-synchronously and have a simultaneous picture start. 18. Surface topology detection device with at least - a stereo camera system according to one of claims 1 to 17 for measuring three-dimensional surface topologies of a measurement object, - a transport device for transporting the measurement object, - A synchronization device to synchronize the speed of the transport with the measurements of the three-dimensional surface topology, and - An evaluation device to evaluate the measurements of the three-dimensional surface topology.