EP2932434A1

EP2932434A1 - Method and device for anaylzing trafficability

Info

Publication number: EP2932434A1
Application number: EP13826921.2A
Authority: EP
Inventors: Ralph GREWE; Stefan Hegemann; Matthias Zobel
Original assignee: Conti Temic Microelectronic GmbH
Current assignee: Conti Temic Microelectronic GmbH
Priority date: 2012-12-11
Filing date: 2013-12-06
Publication date: 2015-10-21
Also published as: US20150324649A1; WO2014090245A1; DE102012112104A1; US9690993B2; DE112013005909A5

Abstract

The invention relates to a method for analyzing trafficability using a computer, said method comprising the following steps: • - receiving image data of surroundings (10) in front of a vehicle (S10); • - analyzing the image data to identify different zones (12, 14, 16, 18, 24) in an image of the surroundings (S12, S14, S16, S18, S20); and • - analyzing identified different zones (16, 18) in terms of trafficability for the vehicle (S22, S24); • - stereo images being used for image analysis, and only the pixels of the estimated ground plane being segmented.

Description

PROCESS AND DEVICE FOR PROCESSABILITY ANALYSIS

The invention relates to a method and a device for trafficability analysis, which are particularly suitable for use in driver assistance systems.

Camera-based driver assistance systems, which recognize the course of their own lane on the basis of the lane markings, are now established in the market, and their use is already required by law in certain application areas. Typically, these driver assistance systems recognize the course of the markings of the own and the neighboring lanes and estimate therefrom the position of the own vehicle relative to the lane markings. Unintentional leaving the lane can thus be detected early and the system can initiate a suitable reaction, the z. B. warns the driver before leaving or prevented by driving the steering a leaving the lane.

Further developed driver assistance systems, which warn not only when leaving a lane or prevent the departure, but to support a driver, for example in an evasive maneuver need for such a function more information about the possible path of the own vehicle than by the above-mentioned pure lane marking detecting systems is determined. If, for example, the goal of a driver assistance system is to prevent an accident by means of suitable automatic evasion, such a system also requires reliable information about whether a possible avoidance path is passable at all, in addition to information about one's own traffic lane, so that the vehicle is no longer damaged by the evasion or caused, as it would be by an accident in case of non-evasion. The determination of such information is referred to herein as trafficability analysis.

The object of the present invention is now to propose a method and a device for trafficability analysis. This object is solved by the subject matters of the independent claims. Further embodiments of the invention will become apparent from the dependent claims.

An idea on which the invention is based is to identify different areas in the analysis of image data or data of an environment in front of a vehicle and to analyze these with regard to being navigable by one's own vehicle. For the purposes of the present invention, image data is understood to mean not only data generated by the camera-based system, but also those of all environmental detection systems, for example radar, lidar-based systems, which can supply data to an environment. According to a particular aspect, the detection of different areas is performed on the basis of an estimated ground level of the environment, whereby computing time can be saved and thus, as a rule, faster analysis results are obtained. According to a further particular aspect of the invention, driving activities recognized in the trafficability analysis are taken into account in different areas, as a result of which a more reliable analysis result can be obtained. These two particular aspects of the present invention may, for example, be jointly implemented in a driver assistance system, which may enable the system to provide a relatively reliable and rapid trafficability analysis.

An embodiment of the invention now relates to a trafficability analysis method using a computer comprising the steps of: receiving image data of an environment in front of a vehicle, analyzing the image data to recognize different areas in an environment image, and analyzing detected different areas with respect to the trafficability the vehicle. Analyzing the image data to detect different regions may include the following steps:

Calculating the position of pixels in space from the received image data of multiple images using a stereo vision method,

Estimating a ground plane based on the calculated positions of pixels in the space,

Determining pixels relevant to a segmentation of the estimated groundplane,

Performing segmentation of the estimated ground plane on the relevant pixels, and

Outputting different areas recognized by the segmentation.

As a stereo-vision method, a method based on environmental images acquired with a plurality of camera optics or camera beams may be employed or a method based on recording with a camera optics at various positions utilizing motion stereo may be employed. The segmentation of the estimated ground plane on the relevant pixels can be performed on the basis of color, saturation, intensity and / or texture information.

The segmentation of the estimated ground plane on the relevant pixels can be performed on the basis of variance information of the calculated positions of pixels in space.

The analysis of detected different areas with regard to the trafficability by the vehicle may include the detection of obstacles, in particular raised objects, and / or the detection of driving activities. In this context, a driving activity is understood in particular to mean that a vehicle other than its own is currently driving in an area or has already traveled. In particular, the driving activity can Include information about the direction of travel of the other vehicle, which counter or cross traffic can be considered. In particular, due to a detected driving activity, an area can be excluded from trafficability. This applies, for example, to oncoming traffic even if driving in principle would be possible, but actual driving offers a great risk of a frontal accident.

Also possible is a temporary exclusion of such a range from a general release, if a danger was currently detected. The recognition of driving activities may include receiving and evaluating data of a camera-, radar- and / or lidar-based object recognition and / or receiving and evaluating an object list generated with a camera, radar and / or lidar-based object recognition. The recognition of driving activities can also have a long-term observation of driving activities to increase the recognition reliability, the transfer of a trafficability classification to similar image areas and / or a dynamic exclusion of trafficability when detected danger by driving activity observation.

A further embodiment of the invention relates to a trafficability analysis apparatus using a computer having first means for receiving image data of an environment in front of a vehicle, second means for analyzing the image data for detecting different areas in an environment image, and third means for analyzing detected different ones Areas with regard to vehicle passability.

The second means may be arranged to carry out a method according to the invention and as described above, and the third means may be designed to carry out a method according to the invention and as described above. Another embodiment of the invention relates to a driver assistance system having a device according to the invention and as described herein. Further advantages and possible applications of the present invention will become apparent from the following description in conjunction with the / in the drawing (s) illustrated embodiment (s).

In the description, in the claims, in the abstract and in the drawing (s), the terms and associated reference numerals used in the list of reference numerals recited below are used.

The drawing (s) show / show in

1 shows a flow chart of an embodiment of a method for trafficability analysis according to the invention; and

Fig. 2 shows an example of a detected with a digital camera

An environmental image in front of a vehicle segmented by a trafficability analysis method according to the present invention;

Fig. 3 shows another example of a detected with a digital camera

An environmental image in front of a vehicle segmented by a trafficability analysis method according to the present invention; a block diagram of an embodiment of an apparatus for Trafficability analysis according to the invention.

In the following description, the same, functionally identical and functionally connected elements with the same reference numerals be provided. Absolute values are given below by way of example only and are not to be construed as limiting the invention.

The flowchart of a program executed by a computer, shown in FIG. 1, is used to analyze image data generated, for example, by a stereovision camera which captures images of the surroundings in front of a vehicle and may belong to a camera-based driver assistance system with regard to vehicle driveability For example, to be able to quickly determine a suitable alternative route in the event of an evasive maneuver.

In the method implemented by the program, first, in a step S10, digital image data of the surroundings in front of the vehicle are received by the stereovision camera for the drivability analysis. for example via a special image data transmission line, a vehicle bus or a radio link.

The received image data is then analyzed in subsequent steps S12-S20 to detect different regions in the environment image. With a further analysis in steps S22-S24, the different areas identified in the preceding steps are analyzed with regard to their trafficability by the vehicle and trafficable areas are recognized. The recognized as passable areas can then be issued to be processed by a driver assistance system, which should assist a driver in an evasive maneuver and to signal him drivable alternative routes.

For driveability analysis, a drivable range can typically be determined by analysis of changing or consistent textures, e.g. B. be detected by detecting a transition from a tar track to a turf at the roadside. However, an assessment of the trafficability of the areas adjacent to the own roadway from image information alone is often not reliably possible. So could one adjacent lane be equipped with a different surface than its own lane, but in the image as an unattached strip of sand. This problem of separation and detection of different areas can also occur in the stereovision methods which are often used today, which calculate a spatial (3D) coordinate for each pixel of images captured with a 3D camera. With the aid of these methods, it is basically possible to separate raised objects from the ground level, as is indicated by way of example in FIG. 2 on an exemplary image 10 of a typical scene of the environment in front of a vehicle. In this case, raised objects 20 and 22 can be separated from the ground plane 12. However, if different areas are located within the ground plane 12, for example an asphalt road 14 bounded by meadows (right and left side area 16 or 18), then reliable separation of the areas is often impossible (separation into trafficable / not passable).

A separation of different areas even within a plane such as the ground plane 12 could in principle be carried out by means of, for example, color, intensity or texture-based segmentation of mono images, ie the separation of different areas within the ground plane 12 such as the asphalt road 14 and the adjacent meadows 16 and 18 and the right object / obstacle 20 (FIG. 2) and 30, 32 (FIG. 3), which do not protrude beyond the horizontal line of the ground plane 12. A major disadvantage of this method, however, is the required high computational effort, which speaks so far against a series application, especially in driver assistance systems.

The method according to the invention now combines in the following steps a texture-based segmentation with stereo vision in order to obtain the advantages of both methods with reduced computational complexity. For this purpose, the segmentation can be carried out only on the area which can not be subdivided by the stereo-vision or stereo method (only the ground plane 12 in FIG. 2 instead of the complete, framed area including the objects). In contrast to a pure stereo method, a segmented ground plane in which passable areas are separated from non-drivable areas is obtained, the computational outlay being reduced compared to a segmentation of the full screen.

For this purpose, in step S12, the position of pixels in space is calculated from a plurality of images captured by the stereo-vision camera with the aid of a stereo-vision approach. Based on the space points, an estimate of the ground plane 12 is made in the next step S14. With the help of the determined ground level, the relevant pixels for the segmentation of the ground level 12 can be determined in step S16. On these pixels, a segmentation of the ground plane 12 is performed in step S18. Since the number of pixels to be segmented is significantly lower than in the original image, the computational outlay for the segmentation step S18 can be significantly reduced. The result is a segmentation of the ground plane 12, which provides additional information about passable / non-drivable areas.

As a stereo method, methods based on multiple optics / lumens or motion stereo approaches may be used. The segmentation of the selected pixels can be performed by color, intensity or texture. In addition to these features available from the mono image for a segmentation (eg color, saturation, intensity, texture) can be obtained from the stereo method additional information for a segmentation, for example, the variance of the height of the spatial points (the variance is eg for a meadow in addition to the road surface higher than for a flat road surface) or a small height deviation can be used. The technical advantage of this procedure is a segmentation of points in the ground plane on the basis of features in the image (eg intensity, color, texture), compared to a segmentation of the frame by a suitable selection of pixels (stereo ground level) computing time is saved and also additional information for the segmentation (eg variance of the height of the space points) are made available. Decisive here is the selection of pixels to be segmented from the image with the aid of an estimate of the (relevant) ground plane, which is carried out with the aid of a stereo method.

The different regions 14, 16, 18 of the ground plane 12 (see FIGS. 2 and 3) obtained by the segmentation in step S18 are output in a subsequent step S20 for further processing by a driver assistance system. As part of the further processing, the output different areas are analyzed in subsequent steps S22 and S24 with regard to their trafficability.

In step S22, for example, obstacles 20 and 22 in the right and left side regions 16 and 18 of the road 14 (FIG. 2) and obstacles 30 and 32 in the right side region 16 of the road 14 (FIG. 3) are recognized, for example by a texture , Color or intensity analysis. If obstacles 20 and 22 or 30 and 32 are detected in one area, the corresponding area 16 and 18 with an obstacle 20 or 22 (FIG. 2) and the area 16 with obstacles 30 and 32 (FIG. not passable ", areas without obstacles are marked as" passable ".

In step S24 on the trafficability analysis, the trafficability assessment or analysis of areas around the vehicle contains information as to whether a driving activity has already been or will be perceived in the areas investigated. Such driving activity could be determined by a vehicle-own sensor, for. B. by a camera-based object recognition, or else by the merger with other sensors, eg. B. with the object list of a radar-based sensor.

If, as shown in FIG. 3, a driving activity 28 has been detected in a region 18 in the image, then this region can with high probability be drivable be considered. An observation of the area over a longer period of time can contribute to a correspondingly higher security of this assessment. In addition, it is possible to transfer an existing assessment to similar image regions without explicitly recognizing a driving activity there. If the analysis reveals that the area 18 is passable, this area is marked as "passable" as the result.

It is of course conceivable, based on the observation of driving activity, to explicitly exclude regions from trafficability, even though driving in the area would in principle be possible, but actual driving would be too great a risk or a danger, eg. B. because there was detected oncoming traffic. This could happen only temporarily, d. H. general release of trafficability but interim locking if danger was detected.

In general, the approach according to step S24 is not limited to camera-based systems, but the analysis and fused consideration of driving activity can be used in all traffic assessment systems.

The marked as "not passable" and "passable" areas can be further processed by the driver assistance system, in particular they can be used to determine a possible alternative route in an on-road obstacle 14. If an evasive maneuver is necessary, a determined alternative route can either be passively signaled to the driver, for example by a visual display or by a voice output similar to a navigation system, or a determined alternative route can be used for an active intervention in the vehicle control, for example Generate autonomous steering interventions for initiating and possibly performing an evasive maneuver. 4 shows a block diagram of a trafficability analysis device 100 according to the invention, which processes data from a stereo-vision camera with a first and second camera 102 or 104. The two cameras 102 and 104 provide image data of the surroundings in front of a vehicle. This image data is supplied to a stereo vision processing unit 106, which calculates the position of pixels in the space, that is, executes the above-explained process step S12. The calculated pixel positions in the space are transmitted to a ground level estimation unit 108, which estimates a ground level in the surrounding images based on the obtained space points in accordance with method step S14 explained above. A relevant pixel selection unit 1 1 0 determined on the basis of the unit 108 estimated ground level and the image data from the two cameras 1 02 and 1 04 relevant pixels for a segmentation of the ground plane (corresponding to the above-explained method step S1 6). Based on the relevant pixels determined by the unit 1 1 0, an image segmentation unit 1 1 2 performs a segmentation of the ground plane (method step S1 8). The different areas of the ground level determined by the unit 1 1 2 are output from a ground level area output unit 1 14 in a form suitable for further processing to a trafficability analysis unit 1 1 6 which analyzes each of the output different areas in terms of drivability (corresponding the method steps S22 and S24 explained above and outputs the result of the analysis, for example in the form of a list, as follows:

Fig. 1:

Area 14 -> passable

Area 1 6 -> not passable

Area 1 8 -> not passable

Fig. 2:

Area 14 -> passable

Area 1 6 -> not passable Area 1 8 -> passable.

The above lists can be further processed by a driver assistance system as described above.

The device shown in Fig. 4 may be implemented in hardware and / or software. For example, it may be implemented in the form of an Application Specific Integrated Circuit (ASIC) or Programmable Gate Array (FPGA) or a microprocessor or controller that executes a firmware implementing the method shown in FIG.

The present invention enables a computationally efficient audibility analysis, especially for use in driver assistance systems.

reference numeral

1 0 Environment image

1 2 ground level

14 asphalt road with two lanes

1 6 Meadow / right-hand side of the road 14 1 8 Meadow / left-hand side of the road 14 20 Obstacle in the right-hand side area 1 6 22 Obstacle in the left-hand side area 1 8 24 Horizontal area

28 Driving activity in the left-hand side area 30 Obstacle in the right-hand side area 32 Obstacle in the right-hand side area 1 00 Device for trafficability analysis 1 02 first camera

1 04 second camera

1 06 Stereovision processing unit

1 08 Ground plane estimation unit

1 10 Relevant pixel selector

1 12 image segmentation unit

1 14 Ground level area output unit 1 16 Trafficability analysis unit

Claims

claims

1 . Method for trafficability analysis using a computer with the following steps:

- Receiving image data of an environment (10) before one

Vehicle (S10),

Analyzing the image data to detect different regions (12, 14, 16, 18, 24) in a surrounding image (S12, S14, S16, S18, S20), and

- Analyzing recognized different areas (16, 18) with respect to the vehicle passability (S22, S24).

2. The method according to claim 1, characterized in that analyzing the image data for recognizing different regions comprises the following steps:

- Compute the position of pixels in the space of the

received image data of multiple images using a stereo vision method (S12),

Estimating a ground plane (12) based on the calculated positions of pixels in the space (S14),

Determine for a segmentation of the estimated

Ground plane (12) relevant pixels (S16),

Performing segmentation of the estimated ground plane on the relevant pixels (S18), and

- outputting recognized by the segmentation

different areas (14, 16, 18, 20, 22, S20).

3. The method according to claim 2, characterized in that as a stereo-vision method based on an environment images recorded with a plurality of camera optics or camera imagers based method or based on the recording with a camera optics at various positions by utilizing the proper motion

Method is used.

4. The method according to claim 2 or 3, characterized in that the segmentation of the estimated ground plane on the relevant pixels on the basis of color, saturation, intensity and / or texture information is performed.

5. The method of claim 2, 3 or 4, characterized in that the segmentation of the estimated ground plane on the relevant pixels on the basis of variance information of the calculated

Positions of pixels in space is performed.

6. The method according to any one of the preceding claims, characterized in that the analysis of recognized different areas

with regard to the trafficability by the vehicle, the detection of obstacles (20, 22, 30, 32, S22), in particular raised objects, and / or the recognition of driving activities (28, S24).

7. The method according to claim 6, characterized in that the detection of driving activities comprises receiving and evaluating data of a camera, radar and / or lidar-based object detection and / or receiving and evaluating an object list generated with a camera, radar and / or lidar-based object recognition.

8. The method of claim 6 or 7, characterized in that the detection of driving activities also a long-term observation of driving activities to increase the detection reliability, the transfer of a trafficability classification to similar

Image areas and / or a dynamic exclusion of

Trafficability when detected danger by driving activity observation has.

9. Device for trafficability analysis using a computer with

- First means (106, 1 10) for receiving image data of a

Environment (10) in front of a vehicle,

second means (108, 110, 112, 14) for analyzing the image data to recognize different regions (12, 14, 16, 18, 24) in an environmental image, and

- Third means (1 16) for analyzing recognized

different areas (16, 18) in terms of vehicle passability.

10. Apparatus according to claim 9 characterized in that the second means (108, 110, 112, 14) are adapted to carry out a method according to any one of claims 2 to 5, and

the third means (1 16) are formed, a method according to one of

Claims 6 to 7 execute.