[go: up one dir, main page]

WO2020127151A1 - Method for improved object detection - Google Patents

Method for improved object detection Download PDF

Info

Publication number
WO2020127151A1
WO2020127151A1 PCT/EP2019/085490 EP2019085490W WO2020127151A1 WO 2020127151 A1 WO2020127151 A1 WO 2020127151A1 EP 2019085490 W EP2019085490 W EP 2019085490W WO 2020127151 A1 WO2020127151 A1 WO 2020127151A1
Authority
WO
WIPO (PCT)
Prior art keywords
data points
dimensional array
vehicle
troublemakers
image
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/EP2019/085490
Other languages
French (fr)
Inventor
Vanessa Picron
Ashley Chacon-Alam
Camilo VEJARANO
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Valeo Schalter und Sensoren GmbH
Original Assignee
Valeo Schalter und Sensoren GmbH
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Valeo Schalter und Sensoren GmbH filed Critical Valeo Schalter und Sensoren GmbH
Publication of WO2020127151A1 publication Critical patent/WO2020127151A1/en
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S7/00—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00
    • G01S7/48—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S17/00
    • G01S7/4808—Evaluating distance, position or velocity data
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S13/00—Systems using the reflection or reradiation of radio waves, e.g. radar systems; Analogous systems using reflection or reradiation of waves whose nature or wavelength is irrelevant or unspecified
    • G01S13/86—Combinations of radar systems with non-radar systems, e.g. sonar, direction finder
    • G01S13/867—Combination of radar systems with cameras
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S13/00—Systems using the reflection or reradiation of radio waves, e.g. radar systems; Analogous systems using reflection or reradiation of waves whose nature or wavelength is irrelevant or unspecified
    • G01S13/88—Radar or analogous systems specially adapted for specific applications
    • G01S13/93—Radar or analogous systems specially adapted for specific applications for anti-collision purposes
    • G01S13/931—Radar or analogous systems specially adapted for specific applications for anti-collision purposes of land vehicles
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
    • G01S17/86—Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
    • G01S17/88—Lidar systems specially adapted for specific applications
    • G01S17/89—Lidar systems specially adapted for specific applications for mapping or imaging
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
    • G01S17/88—Lidar systems specially adapted for specific applications
    • G01S17/93—Lidar systems specially adapted for specific applications for anti-collision purposes
    • G01S17/931—Lidar systems specially adapted for specific applications for anti-collision purposes of land vehicles
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S7/00—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00
    • G01S7/02—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S13/00
    • G01S7/41—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S13/00 using analysis of echo signal for target characterisation; Target signature; Target cross-section
    • G01S7/417—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S13/00 using analysis of echo signal for target characterisation; Target signature; Target cross-section involving the use of neural networks
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S7/00—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00
    • G01S7/48—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S17/00
    • G01S7/483—Details of pulse systems
    • G01S7/486—Receivers
    • G01S7/487—Extracting wanted echo signals, e.g. pulse detection
    • G01S7/4876—Extracting wanted echo signals, e.g. pulse detection by removing unwanted signals
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/25—Fusion techniques
    • G06F18/251—Fusion techniques of input or preprocessed data
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
    • G06V20/58—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/60—Type of objects
    • G06V20/64—Three-dimensional objects
    • G06V20/647—Three-dimensional objects by matching two-dimensional images to three-dimensional objects
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S13/00—Systems using the reflection or reradiation of radio waves, e.g. radar systems; Analogous systems using reflection or reradiation of waves whose nature or wavelength is irrelevant or unspecified
    • G01S13/86—Combinations of radar systems with non-radar systems, e.g. sonar, direction finder
    • G01S13/865—Combination of radar systems with lidar systems
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S13/00—Systems using the reflection or reradiation of radio waves, e.g. radar systems; Analogous systems using reflection or reradiation of waves whose nature or wavelength is irrelevant or unspecified
    • G01S13/88—Radar or analogous systems specially adapted for specific applications
    • G01S13/93—Radar or analogous systems specially adapted for specific applications for anti-collision purposes
    • G01S13/931—Radar or analogous systems specially adapted for specific applications for anti-collision purposes of land vehicles
    • G01S2013/9323—Alternative operation using light waves

Definitions

  • the present invention refers to a method for improved object detection based on two types of environment sensors applied in a driving support system of a vehicle.
  • the present invention also refers to a driving support system for performing the above method.
  • Autonomous and semi-autonomous driving is becoming a more and more important issue in the automotive industry.
  • Prototypes for autonomous driving have already been developed and deployed and are currently being tested, in some places even under real driving situations.
  • Autonomous driving is considered as a disruptive technology in the automotive sector.
  • ego vehicle surrounding around the vehicle
  • Different types of environment sensors can be employed in the ego vehicle to monitor the surrounding thereof and to identify objects like third party vehicles, traffic signs or others.
  • the knowledge about the position of an object and the calculation of precise trajectories of moving objects in the surrounding of the ego vehicle, like third party vehicles, is an indispensable prerequisite. Therefore, in order to avoid accidents, the information about the object, its position, orientation and velocity need to be reliable.
  • the environment sensors used for monitoring the surroundings of the vehicle can be categorized in several categories according to the type of information that is obtained by the sensors.
  • some kind of sensors can measure a distance to an object in the surrounding of the ego vehicle, e.g. LiDAR or Radar based sensors, and are therefore referred to as range type sensors.
  • Other kinds of sensors e.g. optical cameras, cannot provide accurate information about the distance to an object. Instead, their strength is to provide a 2-dimensional image of the surrounding of the ego vehicle, which can be used to identify objects.
  • Range type sensors has the property that it is rather sparse data, meaning that the density of data points is low compared to image type sensors. Therefore, specific algorithms are used to retrieve information from the range type sensor data about the type of objects, their position, orientation and velocity. However, these specific algorithms can also produce wrong detection results, decreasing the overall system performance and increasing the risk of accidents.
  • Fig. 1 shows an example of a method of prior art, for object detection.
  • Fig. 1 a shows a vehicle 1 with a range type sensor 2 in a surrounding with a third party vehicle 3 and a bush 4.
  • Fig. 1 b) illustrates the range type sensor data 5 gathered by the range type sensor 2.
  • a method according to prior art can lead to wrong detection result.
  • the range type sensor data 5 stemming from the third party vehicle 3 is correctly detected as a first vehicle 6.
  • the range type sensor data 5 stemming from the bush 4 is misinterpreted to be a second vehicle 7. This detection error decreases the overall system performance and increases the risk of accidents.
  • the present invention provides a method for improved object detection based on two types of environment sensors applied in a driving support system of a vehicle, whereas the first type of environment sensor is an image type sensor having an image-field-of-view and the second type of environment sensor is a range type sensor having a range-field-of-view that at least partially overlaps the image-field-of-view, comprising the steps of providing a 2-dimensional array of data points representing the surrounding of the vehicle in the image-field-of-view by at least one image type sensor, identifying one or more troublemakers in the 2-dimensional array of data points, providing a 3-dimensional array of data points representing the surrounding of the vehicle in the range-field-of-view by at least one range-type-sensor, mapping the 3- dimensional array of data points into the 2-dimensional array of data points, selecting one or more 3D-sub-sets of data points in the 3-dimensional array of data points matching the one or more troublemakers, providing a revised 3-dimensional array of data points, considering the 3-dimensional array of data points and the one or more 3D
  • the present invention also provides a driving support system for performing the above method comprising at least one image type sensor for providing a 2-dimensional array of data points representing the surrounding of the vehicle in an image-field-of-view and one range type sensor for providing a 3-dimensional array of data points representing the surrounding of the vehicle in a range-field-of-view.
  • the basic idea of the invention is to use two different types of sensors, a range type sensor that gives accurate information about the position, size, velocity, and/or orientation of the object, and an image type sensor that gives accurate information about the nature or type of the object.
  • the data gathered by these sensors is analyzed interdependently allowing for a reliable determination of the position, size and/or orientation of the object.
  • a key aspect of the method is to use the 2-dimensional array of data points of the image type sensor for identifying troublemakers. This information gathered from the image type sensor is used for processing of the 3-dimensional array of data points provided by the range type sensor.
  • the 3-dimensional array of data points is mapped into the 2-dimensional array of data points and one or more 3D-sub-set of data points in the 3-dimensional array of data points is selected, which matches each of the identified troublemakers.
  • a revised 3-dimensional array of data points is provided, considering the 3-dimensional array of data points and the one or more 3D-sub-sets of data points. This revised 3-dimensional array of data points is used for detecting position, size, velocity and/or orientation of objects.
  • a troublemaker is an object in the surrounding of the ego vehicle that has a high likelihood of producing wrong and/or unreliable detection results.
  • the method uses a revised 3-dimensional array of data points to detect position, size, velocity and/or orientation of objects.
  • the revised 3-dimensional array of data points is provided by considering the 3-dimensional array of data points and the one or more 3D-sub-sets of data points. Since the one or more 3D-sub-set of data points are based on one or more troublemakers, the method has the advantages that the stability and reliability of the detection results are enhanced.
  • the method decreases wrong determination of position, size, velocity, and/or orientation of objects compared to a method where troublemakers are not identified. Therefore the method is less prone to errors, reliability and stability of object detection is improved and the risk for accidents is decreased.
  • the vehicle i.e. the ego vehicle, according to the present invention, can be any kind of vehicle, e.g. a car or truck.
  • the vehicle can be driven manually by a human driver.
  • the vehicle supports semi-autonomous or autonomous driving. It is possible that the vehicle transports passengers, including a driver, or is used for cargo handling.
  • An image type sensor is a device that detects and conveys information used to make a 2-dimensional array of data points, which in turn can be plotted as an image. It does so by converting the variable attenuation of light waves or electromagnetic waves into signals, preferably electric signals.
  • Light waves or electromagnetic waves can have different wavelengths. Depending on the wavelength different image type sensors can be used. E.g. for the visible spectrum a camera can be used. Alternatively image type sensors for electromagnetic waves in the infrared (around 1000 nm) or in the ultraviolet (around 200 nm) can be used.
  • the 2-dimensional array of data points can be in Cartesian coordinates or in polar coordinates. E.g.
  • the data captured by a camera can be in Cartesian coordinates determining the position of individual data points or pixels relative to the axis of the image.
  • the data points themselves can be annotated with more information.
  • a color camera provides a pixel based image as 2-dimensional data, with the individual pixels holding information in the three color channels RGB.
  • the individual pixels can hold information about the intensity of the signal and/or a brightness value.
  • the images captured by the camera may be individual still photographs or sequences of images constituting videos or movies.
  • an image type sensor can be an optical sensor, a camera, a thermal imaging device or a night vision sensor.
  • a range type sensor is a device that captures the three-dimensional structure of the environment from the viewpoint of the sensor, usually measuring the depth and/or distance to the nearest surfaces.
  • a range type sensor can be a LiDAR based sensor, a radar based sensor, an infrared based sensor, or an ultrasonic based sensor.
  • Radar sensors use radio waves to determine the range, angle, and/or velocity of objects.
  • Ultrasonic based sensors work on the principle of reflected sound waves.
  • LiDAR based sensors measure the distance to an object by illuminating the object with pulsed laser light and measuring the reflected pulses. Differences in laser return times, wavelengths and intensity can then be used to provide a 3-dimensional representation of the environment of the vehicle.
  • Infrared based sensors also work on the principle of reflected light waves.
  • the measurements of the distance to the objects can be performed at single points of the surrounding of the ego vehicle, across scanning planes, or the measurements can provide a full image with depth and or distance measurements at every point of the surrounding of the ego vehicle.
  • the data determined by the range type sensor is a 3-dimensional array of data points and can be in spherical coordinates, including the distance to an object (r) and the position of the object relative to the sensor position determined by the polar and the azimuth angle (theta, phi).
  • the data can be determined in or transformed to Cartesian coordinates, identifying the position of the object relative to the axis lines X, Y and Z and the origin of the coordinate system.
  • the individual data points of the 3-dimensional array of data points can be annotated with more information, e.g. intensity information of the reflected light pulses.
  • Objects in the surrounding of the ego vehicle can be any kind of objects. They can be static objects like houses, traffic signs, or parked cars. Furthermore, the objects can be moving, like third party vehicles or pedestrians. Troublemakers are also objects in the surrounding of the ego vehicle. However, range type data of troublemakers tends to produce wrong and/or unreliable detection results.
  • the range type sensor captures information about the three-dimensional structure of the environment by illuminating the objects in the surrounding with pulsed electromagnetic radiation and measuring the reflected pulses. Due to the pulsed radiation the data provided by range type sensors is rather sparse data, meaning that the density of data points is low compared to image type sensors. If the object has a continuous surface, e.g.
  • the data gathered by the range type sensor results in a reliable determination of the position, size, velocity and/or orientation of the object.
  • the object has a discontinuous surface, e.g. a wired fence, a grid, or vegetation, some of the
  • electromagnetic pulses are not reflected by the object itself but travel through the object and are eventually reflected by another object. Using this data for object detection often leads to wrong and/or unreliable results.
  • the field of view is a sector or in general part of the surrounding of the vehicle, from which the respective sensor captures information.
  • the field of view of a sensor may be influenced by the design of the sensor. For example, a camera gathers information via a lens, which focuses incoming light waves. The curvature of the lens influences the field of view of the camera. For example, the field-of-view of a camera having a fisheye lens is wider than the field-of-view of a camera having a conventional lens.
  • the field of view can also be influenced by the size or the dimensions of the detector that is used to convert the attenuation of light waves or electromagnetic waves into electrical signals. It is also possible that a sensor has a field-of-view that covers 360 degree of the surrounding of the vehicle. For example this can be achieved by a rotating sensor, or by using several interconnected sensors.
  • the two types of environment sensors applied in the driving support system of the vehicle have at least partially overlapping fields-of-view. Therefore the two types of sensors capture at least some information from the same sector or scope of the surrounding of the vehicle. It is also possible to have multiple environment sensors of one type that are interconnected to increase the field-of-view of this type of environment sensor.
  • the method comprises the steps of providing a 2-dimensional array of data points representing the surrounding of the vehicle in the image-field-of-view by at least one image type sensor, and identifying one or more troublemakers in the 2-dimensional array of data points.
  • the troublemaker can be identified by classifying objects in the surrounding of the vehicle.
  • an object in the surrounding of the vehicle can be a bush.
  • the object can be a third party vehicle, a wire fence or a pedestrian.
  • the different types of objects differ in their nature and characteristics, for example in their ability to move, in their velocity and in their vulnerability. Furthermore, the different types of objects differ in the likelihood of producing wrong and/or unreliable results, when detecting the object’s position, size, velocity, and/or orientation. For decreasing the risk of accidents, it is preferable to know what kinds or types of objects are located in the surrounding of the ego vehicle. Therefore, the one or more troublemakers can be identified by classifying the objects into different classes, and by considering the class of the object. By identifying the object, also a confidence value about the class of the object can be determined. The information about the class of the object and the confidence can be stored for later use.
  • identifying one or more troublemakers comprises detecting only objects that belong to a predefined class. This can be achieved by detecting instances of semantic objects of a certain class (such as bushes, or grids) in the 2-dimensional array of data points. Every object class has its own special features that help in classifying the object into the specific class. For example all circles are round. Object class detection uses these special features. For example, when looking for circles, objects that are at a particular distance from a point (i.e. the center) are sought. Identifying troublemakers can also comprise that the troublemaker is tagged with an unambiguous mark establishing therefore the possibility to follow the troublemaker in time.
  • the troublemaker is identified by analyzing the 2-dimensional array of data points.
  • troublemakers can be identified by pattern recognition algorithms, which automatically discover regularities or irregularities in the 2-dimensional array of data points.
  • the 2-dimensional array of data points can be partitioned in multiple segments.
  • the 2-dimensional array of data points can be partitioned by a thresholding method, where a threshold value is used for deciding to what segment an individual data point of the 2-dimensional array of data points belongs.
  • clustering methods can be used to partition the 2- dimensional array of data points into clusters/segments. It is also possible to use histogram-based methods, where a histogram is computed from all of the data points in the 2-dimensional array of data points.
  • Color or intensity values can be used as the measure for the histogram. Afterwards, peaks and valleys in the histogram are used to locate the segments in the 2-dimensional array of data points.
  • the goal of segmentation is to partition the 2-dimensional array of data points into segments, wherein the data points belonging to the same segment have one or more common features or the data points consist of an object with a semantic meaning (e.g. a bush). By this process objects and/or troublemakers and/or boundaries (lines, curves, etc.) of objects and/or troublemakers can be identified.
  • identifying one or more troublemakers comprises discovering one or more troublemakers coupled with segmenting the 2-dimensional array of data points.
  • object co-segmentation can be applied.
  • the troublemaker can be present sporadically in a set of images or the troublemaker can disappear intermittently throughout the video of interest.
  • multiple images or video frames are jointly segmented, based on semantically similar objects. Therefore information about the troublemaker can be shared among consecutive frames and information about motion and appearance of the troublemaker can be used to find the common regions in multiple images belonging to the troublemaker.
  • the step of identifying one or more troublemakers in the 2-dimensional array of data points comprises assigning a label to every data point in the 2-dimensional array of data points, such that data points with the same label share certain predefined characteristics.
  • the method also comprises the steps of providing a 3-dimensional array of data points representing the surrounding of the vehicle in the range-field-of-view by at least one range-type-sensor, and mapping the 3-dimensional array of data points into the 2- dimensional array of data points.
  • the mapping or projecting of the 3-dimensional array of data points into the 2-dimensional array of data points involves defining translation and rotation parameters to associate the 3-dimensional array of data points with the 2- dimensional array of data points. For this reason, the relative locations of the range type sensor and the image type sensor at the ego vehicle and their respective fields-of-view need to be known.
  • the revised 3-dimensional array of data points is used to detect the position, size, velocity, and/or orientation of the objects.
  • the position of the object is the position of the object with respect to the ego vehicle. Therefore from the position of the object the distance from the object to the ego vehicle can be determined. This information together with the velocity of the object is highly important for calculating the time of a possible collision with an object.
  • the size of an object includes the 3 dimensions width, length and height of an object.
  • the orientation of the object is the orientation of the object with respect to the ego vehicle.
  • a third party vehicle has a front side and a back side, determining an internal coordinate system of the object.
  • a pedestrian has also a front side and a back side determining a pedestrian based internal coordinate system.
  • the orientation of the object is the orientation of the object’s coordinate system with respect to the coordinate system defined by the ego vehicle. For example if a third party vehicle and the ego vehicle are both driving on a straight lane in the same direction, the orientation of the third party vehicle would be parallel to the ego vehicle. This is independent of the location of the third party vehicle, meaning it is regardless if the third party vehicle is driving in front of the ego vehicle or next to the ego vehicle. Flowever, if the third party vehicle is driving on a lane intersecting the lane of the ego vehicle, the orientation of the third party vehicle is different from being parallel.
  • the orientation of an object can be determined for static as well as for moving objects.
  • the position, size, velocity and/or orientation of the objects considering the revised 3- dimensional array of data points can be achieved by estimating parameters of a mathematical model from the revised 3-dimensional array of data points.
  • the step of identifying one or more objects in the 2-dimensional array of data points comprises determining an area in the 2-dimensional array of data points belonging to the one or more troublemakers, and the step of selecting one or more 3D-sub-sets of data points in the 3-dimensional array of data points matching the one or more troublemakers comprises selecting one or more 3D-sub-set of data points in the 3-dimensional array of data points that are mapped inside the area belonging to the one or more troublemakers.
  • the area in the 2-dimensional array of data points belonging to the troublemakers can have any shape.
  • it can be a rectangle or an ellipse surrounding the troublemaker.
  • a rectangular box, also called bounding box, surrounding the troublemaker can be used for determining the area in the 2-dimensional array of data points belonging to the troublemaker.
  • the area belonging to the troublemaker preferably has a shape retracing the contours of the troublemaker. This has the advantages that the results of the method are more precise than when using a rectangular box.
  • the troublemaker is identified by segmenting the 2- dimensional array of data points, the segment often already retraces the contour of the troublemaker. Therefore no additional computational resources are needed. If a data point of the 3-dimensional array of data points is mapped inside the area belonging to the troublemaker it is considered to match the troublemaker. Therefore this data point is selected for the 3D-sub-set of data points.
  • the step of providing a revised 3- dimensional array of data points, considering the 3-dimensional array of data points and the one or more 3D-sub-sets of data points comprises removing the one or more 3D- sub-sets of data points from the 3-dimensional array of data points.
  • the detection results have a high likelihood of being wrong and/or unreliable.
  • the revised 3-dimensional array of data points only consists of data points that do not stem from troublemakers. Therefore by removing the range type data stemming from the troublemaker from the 3- dimensional array of data points the reliability and stability of the detection results are enhanced.
  • the step of identifying one or more troublemakers in the 2-dimensional array of data points comprises identifying one or more objects having an inconsistent reflection surface and/or identifying one or more objects having a discontinuous reflection surface.
  • the troublemaker is an object having an inconsistent reflection surface and/or an object having a discontinuous reflection surface. Since the range type sensor captures information about the three-dimensional structure of the environment by illuminating the objects in the surrounding with pulsed electromagnetic radiation and measuring the reflected pulses, objects that have a discontinuous reflection surface, have a high likelihood to produce wrong and/or unreliable detection results. E.g. the data received by a range type sensor that is reflected from a bush does in general not show the outer contours of the bush.
  • the electromagnetic pulses may have travelled inside the bush and may be reflected by an inner part of the bush. Furthermore, the data points stemming from the bush can be unstable in the sense, that it appears that some part of the bush is moving relative to another part of the bush. For these reasons the range type data of troublemakers tends to produce wrong and/or unreliable detection results. A similar effect occurs with objects having an inconsistent reflection surface, meaning that the magnitude by which the electromagnetic pulse is reflected varies strongly with different locations on the reflection surface of the object. However, due to the sparseness of the range type data such situations are not easily detectable in the range type data itself. Therefore, the method uses the information gathered by the image type sensor to identify troublemakers.
  • the step of identifying one or more troublemakers in the 2-dimensional array of data points comprises identifying one or more objects of the category tree, bush, hedge, vegetation, grid and/or wire fence.
  • Using the 2-dimensional array of data points gathered by the image type sensor to identify the one or troublemakers has the advantage that an easy, fast and/or reliable identification of the troublemaker can be achieved.
  • the knowledge about what kind of object is a troublemaker is obtained by a heuristic technique.
  • the step of providing 2- dimensional array of data points representing the surrounding of the vehicle in the image-field-of-view by at least one image type sensor comprises providing 2- dimensional array of data points by at least one camera.
  • camera based systems are already integrated to detect the environment. Therefore, using a camera as image type sensor does not generate extra costs when employing the method.
  • cameras with special lenses e.g. fisheye lenses, which have a wide field-of-view up to 180 degrees or even more, can be used. Cameras provide accurate information about the type of an object. Also, the shape of an object can be determined in the image by established procedures.
  • night vision technology where parts of the electromagnetic spectrum not visible to a human are taken into account, like near-infrared or ultraviolet radiation, and very sensitive detectors are employed, information about objects and/or troublemakers can be gathered by the camera in situations where a human driver is limited.
  • the step of providing 3- dimensional array of data points representing the surrounding of the vehicle in the range-field-of-view by at least one range type sensor comprises providing 3-dimensional array of data points by at least one LiDAR sensor and/or by at least one radar sensor.
  • a LiDAR and/or a radar sensor is/are used in collision avoidance system, for example to measure the distance to a third party vehicle in front of the ego vehicle.
  • LiDAR and/or radar sensors give very accurate information about the position of an object, especially about its distance to the ego vehicle.
  • LiDAR and/or radar sensors can have field-of-views of up to 360 degrees e.g. by using rotating sensors. It is also possible to use several range type sensors, e.g. a combination of a LiDAR and a radar sensor.
  • the step of identifying one or more troublemakers in the 2-dimensional array of data points comprises identifying one or more troublemakers in the 2-dimensional array of data points by using an image recognition algorithm and/or by using a neural network.
  • Processing a 2-dimensional array of data points e.g. from cameras, in particular video data comprising a sequence of multiple frames per second, is very challenging. Huge amounts of data have to be processed in real time in order to reliably detect the surrounding of the vehicle without delays.
  • resources of the vehicle for processing the data are limited in respect to space for housing processing devices and also in respect to available computational and electrical power. Even when the technical issues are resolved, in order to provide vehicles at an affordable price, the resources keep limited to their price.
  • Neural networks comprise an input and an output layer, as well as one or multiple hidden layers.
  • the hidden layers of a deep neural network typically consist of convolutional layers, pooling layers, fully connected layers and normalization layers.
  • the convolutional layers apply a convolution operation to the input, passing the result to the next layer.
  • the convolution emulates the response of an individual neuron to visual stimuli.
  • an image recognition algorithm can be used.
  • Image recognition algorithms that can be used are for example genetic algorithms, approaches based on CAD-like object models, like Primal Sketch, appearance-based methods, e.g. edge matching or conquer and search, or other feature-based methods like Histogram of Oriented Gradients (HOG), Haar-like features, Scale-Invariant Feature Transform (SIFT) or Speeded UP Robust Feature (SURF).
  • HOG Histogram of Oriented Gradients
  • SIFT Scale-Invariant Feature Transform
  • SURF Speeded UP Robust Feature
  • segmentation techniques to partition the 2-dimensional array of data points into several parts, according to low-level cues such as color, texture and/or smoothness of boundary.
  • semantic segmentation techniques for example with deep neural networks, can be used, where the 2-dimensional array of data points is partitioned into semantically meaningful parts, and to classify each part into one of the pre-determined classes.
  • pixel-wise classification techniques can be used, where each data point of the 2-dimensional array of data points is classified rather than the entire 2-dimensional array of data points or segment.
  • the class of the troublemaker is known with a certain confidence value. Also the location of the troublemaker in the image and the area belonging to the troublemaker in the image are determined.
  • the driving support system comprises at least one camera as image type sensor for providing a 2-dimensional array of data points representing the surrounding of the vehicle in an image-field-of-view and at least one LiDAR sensor and/or at least one radar sensor as range type sensor for providing a 3-dimensional array of data points representing the surrounding of the vehicle in a range-field-of-view.
  • the camera and/or the LiDAR sensor and/or the radar sensor can be installed inside the vehicle or outside the vehicle.
  • the driving support system comprises at least one image type sensor for providing a 2-dimensional array of data points having an image-field-of-view of 360 degrees of the surrounding of the vehicle.
  • This can also be achieved by using multiple image type sensors.
  • the ego vehicle can use four cameras, one having a field-of-view that covers the sector of the surrounding in front of the ego-vehicle, one having a field-of-view that covers the sector of the surrounding behind the ego-vehicle, and two cameras having field-of-view that respectively cover the sectors of the surrounding on the sides of the ego-vehicle.
  • the driving support system comprises at least one range type sensor for providing a 3-dimensional array of data points having a range- field-of-view of 360 degrees of the surrounding of the vehicle. It is also possible to use multiple range type sensors that have fields-of-view that cover only part of the surrounding. Alternatively one or several rotating range type sensors can be used.
  • Fig. 1 shows a method known in the prior art, based on one type of environment sensor, producing a wrong detection result
  • Fig. 2 shows a vehicle with a driving support system for performing a method for improved object detection based on two types of environment sensors according to a first, preferred embodiment of the invention, together with a surrounding of the vehicle,
  • Fig. 3 shows a flow chart of the steps of the method for improved object
  • Fig. 4 illustrates the data representing the surrounding of the vehicle provided by the range type sensor and the image type sensor, which is the result of the steps S1 10 and S120 of the method for improved object detection, according to the first, preferred embodiment of the invention
  • Fig. 5 illustrates the results of step S210 of the method for improved object detection, which is identifying troublemakers on the data provided by the image type sensor of Fig. 4, according to the first, preferred embodiment of the invention
  • Fig. 6 illustrates the result of step S300, selecting a 3D-sub-set, according to the first, preferred embodiment of the invention.
  • Fig. 7 illustrates the result of steps S400 and S500 providing a revised 3- dimensional array of data points and detecting position, size, orientation and/or velocity of objects, according to the first, preferred embodiment of the invention.
  • Fig. 2 shows a vehicle 10 with a driving support system 12 for performing a method for improved object detection based on two types of environment sensors 14, 16 according to a first, preferred embodiment of the invention.
  • the driving support system 12 comprises two environment sensors 14, 16, whereby one environment sensor 14, 16 is an image type sensor 14, in the preferred embodiment of the invention a camera 14.
  • the other environment sensor 14, 16 is a range type sensor 16, in this preferred embodiment a LiDAR sensor 16.
  • the camera 14 has an image-field- of-view 18, defining a sector of the surrounding of the vehicle 10, from which the camera 14 is able to capture information.
  • the LiDAR sensor 16 has a range-field-of-view 20 defining another sector of the surrounding of the vehicle 10 from which the LiDAR sensor 16 is able to capture information. There is at least a partial overlap of the image- field-of-view 18 and the range-field-of-view 20.
  • the object 22, 24 In the surrounding of the vehicle 10, there are different objects 22, 24, whereby some of the objects 22, 24 are troublemakers 24 having a high likelihood of leading to wrong detection results, when only using the LiDAR sensor 16.
  • the camera 14 provides a 2-dimensional array of data points 26 representing the surrounding of the vehicle 10 in the image-field-of-view 18 and the LiDAR sensor 16 provides a 3-dimensional array of data points 28 representing the surrounding of the vehicle 10 in the range-field-of-view 20.
  • a neural network is employed, to identify troublemakers 24 in the 2-dimensional array of data points 26.
  • Figure 3 shows a flowchart of the method for improved object detection based on the two types of environment sensors 14, 16, according to the first, preferred embodiment of the invention.
  • the method is performed using the driving support system 12 in the vehicle 10 of the first embodiment of the invention.
  • the method starts with providing data from the environment sensors 14, 16.
  • step S1 10 a 2-dimensional array of data points 26 representing the surrounding of the vehicle 10 in the image-field-of-view 18 is provided by the camera 14.
  • step S120 a 3-dimensional array of data points 28 representing the surrounding of the vehicle 10 in the range-field-of-view 20 is provided by the LiDAR sensor 16.
  • the method provides data from both sensors 14, 16 in a parallel and continuous manner.
  • Figure 4 illustrates the data representing the surrounding of the vehicle 10 provided by the camera 14 and the LiDAR sensor 16.
  • Figure 4a) illustrates the 2-dimensional array of data points 26 provided by the camera 14, with objects 22 and troublemakers 24.
  • Figure 4b) illustrates the 3-dimensional array of data points 28, provided by the LiDAR sensor 16.
  • a further step S210 of the method troublemakers 24 are identified in the 2- dimensional array of data points 26 and the area 32 belonging to the identified troublemakers 24 is determined in the 2-dimensional array of data points 26.
  • Fig. 5 illustrates the result of this step.
  • the troublemaker 24, here a bush, and the corresponding area 32 are determined by a semantic segmentation technique and the area 32 has a shape retracing the contours of the troublemaker 24.
  • step S220 the 3-dimensional array of data points 28 is mapped into the 2-dimensional array of data points 26.
  • step S300 of the method a 3D-sub-set of data points 27 in the 3-dimensional array of data points 28 that are mapped inside the area 32 belonging to the identified troublemakers 24 are selected.
  • Figure 6 illustrates the 3-dimensional array of data points 28 and the selected 3D-sub- set of data points 27. All data points in the selected 3D-sub-set of data points 27 have been mapped on the area 32 of the bush in Figure 5.
  • step S400 of the method these selected 3D-sub-set of data points 27 are removed from the 3-dimensional array of data points 28, therefore providing a revised 3- dimensional array of data points 34.
  • the result of step S400 is shown in figure 7. All the data points belong to the selected 3D-sub-set of data points 27 after step S300 coming from the troublemaker 24 (the bush) have been removed from the 3-dimensional array of data points 28, resulting in the revised 3-dimensional array of data points 34 (figure 7a).
  • the next step S500 - detecting the position, size, velocity and/or orientation of the object 22 - is more reliable. No wrong detection results occur, therefore improving the determination of the position, size, velocity and/or orientation of the objects 22 (figure 7b).

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Remote Sensing (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Theoretical Computer Science (AREA)
  • Multimedia (AREA)
  • Electromagnetism (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Computation (AREA)
  • Data Mining & Analysis (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Evolutionary Biology (AREA)
  • General Engineering & Computer Science (AREA)
  • Image Analysis (AREA)
  • Traffic Control Systems (AREA)

Abstract

The invention provides a method for improved object (22) detection based on two types of environment sensors (14, 16) applied in a driving support system (12) of a vehicle (10), whereas the first type of environment sensor is an image type sensor (14) having an image-field-of-view (18) and the second type of environment sensor is a range type sensor (16) having a range-field-of-view (20) that at least partially overlaps the image- field-of-view (18), comprising the steps of providing a 2-dimensional array of data points (26) representing the surrounding of the vehicle (10) in the image-field-of-view (18) by at least one image type sensor (14), identifying one or more troublemakers (24) in the 2- dimensional array of data points (26), providing a 3-dimensional array of data points (28) representing the surrounding of the vehicle (10) in the range-field-of-view (20) by at least one range-type-sensor (16), mapping the 3-dimensional array of data points (28) into the 2-dimensional array of data points (26), selecting one or more 3D-sub-sets of data points (27) in the 3-dimensional array of data points (28) matching the one or more troublemakers (24), providing a revised 3-dimensional array of data points (34), considering the 3-dimensional array of data points (28) and the one or more 3D-sub-sets of data points (27), and detecting position, size, velocity, and/or orientation of objects (22), considering the revised 3-dimensional array of data points (34).

Description

Method for improved object detection
The present invention refers to a method for improved object detection based on two types of environment sensors applied in a driving support system of a vehicle.
The present invention also refers to a driving support system for performing the above method.
Autonomous and semi-autonomous driving is becoming a more and more important issue in the automotive industry. Prototypes for autonomous driving have already been developed and deployed and are currently being tested, in some places even under real driving situations. Autonomous driving is considered as a disruptive technology in the automotive sector.
Autonomous and semi-autonomous driving depends on knowledge about the
surrounding around the vehicle, further referred to as ego vehicle. Different types of environment sensors can be employed in the ego vehicle to monitor the surrounding thereof and to identify objects like third party vehicles, traffic signs or others. For autonomous and semi-autonomous driving systems, the knowledge about the position of an object and the calculation of precise trajectories of moving objects in the surrounding of the ego vehicle, like third party vehicles, is an indispensable prerequisite. Therefore, in order to avoid accidents, the information about the object, its position, orientation and velocity need to be reliable.
The environment sensors used for monitoring the surroundings of the vehicle can be categorized in several categories according to the type of information that is obtained by the sensors. For example, some kind of sensors can measure a distance to an object in the surrounding of the ego vehicle, e.g. LiDAR or Radar based sensors, and are therefore referred to as range type sensors. Other kinds of sensors, e.g. optical cameras, cannot provide accurate information about the distance to an object. Instead, their strength is to provide a 2-dimensional image of the surrounding of the ego vehicle, which can be used to identify objects.
Data provided by range type sensors has the property that it is rather sparse data, meaning that the density of data points is low compared to image type sensors. Therefore, specific algorithms are used to retrieve information from the range type sensor data about the type of objects, their position, orientation and velocity. However, these specific algorithms can also produce wrong detection results, decreasing the overall system performance and increasing the risk of accidents.
Fig. 1 shows an example of a method of prior art, for object detection. Fig. 1 a) shows a vehicle 1 with a range type sensor 2 in a surrounding with a third party vehicle 3 and a bush 4. Fig. 1 b) illustrates the range type sensor data 5 gathered by the range type sensor 2. As shown in Fig. 1c) a method according to prior art can lead to wrong detection result. The range type sensor data 5 stemming from the third party vehicle 3 is correctly detected as a first vehicle 6. However, the range type sensor data 5 stemming from the bush 4 is misinterpreted to be a second vehicle 7. This detection error decreases the overall system performance and increases the risk of accidents.
Therefore, methods for reliable detection of objects are highly important.
It is an object of the present invention to provide a method for improved object detection to avoid wrong detection results and to decrease the risk of accidents. This object is achieved by the independent claims. Advantageous embodiments are given in the dependent claims.
In particular, the present invention provides a method for improved object detection based on two types of environment sensors applied in a driving support system of a vehicle, whereas the first type of environment sensor is an image type sensor having an image-field-of-view and the second type of environment sensor is a range type sensor having a range-field-of-view that at least partially overlaps the image-field-of-view, comprising the steps of providing a 2-dimensional array of data points representing the surrounding of the vehicle in the image-field-of-view by at least one image type sensor, identifying one or more troublemakers in the 2-dimensional array of data points, providing a 3-dimensional array of data points representing the surrounding of the vehicle in the range-field-of-view by at least one range-type-sensor, mapping the 3- dimensional array of data points into the 2-dimensional array of data points, selecting one or more 3D-sub-sets of data points in the 3-dimensional array of data points matching the one or more troublemakers, providing a revised 3-dimensional array of data points, considering the 3-dimensional array of data points and the one or more 3D- sub-sets of data points, and detecting position, size, velocity, and/or orientation of objects, considering the revised 3-dimensional array of data points.
The present invention also provides a driving support system for performing the above method comprising at least one image type sensor for providing a 2-dimensional array of data points representing the surrounding of the vehicle in an image-field-of-view and one range type sensor for providing a 3-dimensional array of data points representing the surrounding of the vehicle in a range-field-of-view.
The basic idea of the invention is to use two different types of sensors, a range type sensor that gives accurate information about the position, size, velocity, and/or orientation of the object, and an image type sensor that gives accurate information about the nature or type of the object. The data gathered by these sensors is analyzed interdependently allowing for a reliable determination of the position, size and/or orientation of the object. A key aspect of the method is to use the 2-dimensional array of data points of the image type sensor for identifying troublemakers. This information gathered from the image type sensor is used for processing of the 3-dimensional array of data points provided by the range type sensor. On this account, the 3-dimensional array of data points is mapped into the 2-dimensional array of data points and one or more 3D-sub-set of data points in the 3-dimensional array of data points is selected, which matches each of the identified troublemakers. A revised 3-dimensional array of data points is provided, considering the 3-dimensional array of data points and the one or more 3D-sub-sets of data points. This revised 3-dimensional array of data points is used for detecting position, size, velocity and/or orientation of objects.
A troublemaker is an object in the surrounding of the ego vehicle that has a high likelihood of producing wrong and/or unreliable detection results. The method uses a revised 3-dimensional array of data points to detect position, size, velocity and/or orientation of objects. The revised 3-dimensional array of data points is provided by considering the 3-dimensional array of data points and the one or more 3D-sub-sets of data points. Since the one or more 3D-sub-set of data points are based on one or more troublemakers, the method has the advantages that the stability and reliability of the detection results are enhanced. The method decreases wrong determination of position, size, velocity, and/or orientation of objects compared to a method where troublemakers are not identified. Therefore the method is less prone to errors, reliability and stability of object detection is improved and the risk for accidents is decreased.
The vehicle, i.e. the ego vehicle, according to the present invention, can be any kind of vehicle, e.g. a car or truck. The vehicle can be driven manually by a human driver. Alternatively, the vehicle supports semi-autonomous or autonomous driving. It is possible that the vehicle transports passengers, including a driver, or is used for cargo handling.
An image type sensor is a device that detects and conveys information used to make a 2-dimensional array of data points, which in turn can be plotted as an image. It does so by converting the variable attenuation of light waves or electromagnetic waves into signals, preferably electric signals. Light waves or electromagnetic waves can have different wavelengths. Depending on the wavelength different image type sensors can be used. E.g. for the visible spectrum a camera can be used. Alternatively image type sensors for electromagnetic waves in the infrared (around 1000 nm) or in the ultraviolet (around 200 nm) can be used. Depending on the image type sensor, the 2-dimensional array of data points can be in Cartesian coordinates or in polar coordinates. E.g. the data captured by a camera can be in Cartesian coordinates determining the position of individual data points or pixels relative to the axis of the image. The data points themselves can be annotated with more information. For example, a color camera provides a pixel based image as 2-dimensional data, with the individual pixels holding information in the three color channels RGB. Alternatively the individual pixels can hold information about the intensity of the signal and/or a brightness value. Furthermore, the images captured by the camera may be individual still photographs or sequences of images constituting videos or movies. For example an image type sensor can be an optical sensor, a camera, a thermal imaging device or a night vision sensor.
A range type sensor is a device that captures the three-dimensional structure of the environment from the viewpoint of the sensor, usually measuring the depth and/or distance to the nearest surfaces. A range type sensor can be a LiDAR based sensor, a radar based sensor, an infrared based sensor, or an ultrasonic based sensor. Radar sensors use radio waves to determine the range, angle, and/or velocity of objects. Ultrasonic based sensors work on the principle of reflected sound waves. LiDAR based sensors measure the distance to an object by illuminating the object with pulsed laser light and measuring the reflected pulses. Differences in laser return times, wavelengths and intensity can then be used to provide a 3-dimensional representation of the environment of the vehicle. Infrared based sensors also work on the principle of reflected light waves. The measurements of the distance to the objects can be performed at single points of the surrounding of the ego vehicle, across scanning planes, or the measurements can provide a full image with depth and or distance measurements at every point of the surrounding of the ego vehicle. The data determined by the range type sensor is a 3-dimensional array of data points and can be in spherical coordinates, including the distance to an object (r) and the position of the object relative to the sensor position determined by the polar and the azimuth angle (theta, phi).
Alternatively, the data can be determined in or transformed to Cartesian coordinates, identifying the position of the object relative to the axis lines X, Y and Z and the origin of the coordinate system. The individual data points of the 3-dimensional array of data points can be annotated with more information, e.g. intensity information of the reflected light pulses.
Objects in the surrounding of the ego vehicle can be any kind of objects. They can be static objects like houses, traffic signs, or parked cars. Furthermore, the objects can be moving, like third party vehicles or pedestrians. Troublemakers are also objects in the surrounding of the ego vehicle. However, range type data of troublemakers tends to produce wrong and/or unreliable detection results. The range type sensor captures information about the three-dimensional structure of the environment by illuminating the objects in the surrounding with pulsed electromagnetic radiation and measuring the reflected pulses. Due to the pulsed radiation the data provided by range type sensors is rather sparse data, meaning that the density of data points is low compared to image type sensors. If the object has a continuous surface, e.g. the surface of a third party vehicle, the data gathered by the range type sensor results in a reliable determination of the position, size, velocity and/or orientation of the object. However, if the object has a discontinuous surface, e.g. a wired fence, a grid, or vegetation, some of the
electromagnetic pulses are not reflected by the object itself but travel through the object and are eventually reflected by another object. Using this data for object detection often leads to wrong and/or unreliable results.
The field of view is a sector or in general part of the surrounding of the vehicle, from which the respective sensor captures information. The field of view of a sensor may be influenced by the design of the sensor. For example, a camera gathers information via a lens, which focuses incoming light waves. The curvature of the lens influences the field of view of the camera. For example, the field-of-view of a camera having a fisheye lens is wider than the field-of-view of a camera having a conventional lens. The field of view can also be influenced by the size or the dimensions of the detector that is used to convert the attenuation of light waves or electromagnetic waves into electrical signals. It is also possible that a sensor has a field-of-view that covers 360 degree of the surrounding of the vehicle. For example this can be achieved by a rotating sensor, or by using several interconnected sensors.
The two types of environment sensors applied in the driving support system of the vehicle have at least partially overlapping fields-of-view. Therefore the two types of sensors capture at least some information from the same sector or scope of the surrounding of the vehicle. It is also possible to have multiple environment sensors of one type that are interconnected to increase the field-of-view of this type of environment sensor.
The method comprises the steps of providing a 2-dimensional array of data points representing the surrounding of the vehicle in the image-field-of-view by at least one image type sensor, and identifying one or more troublemakers in the 2-dimensional array of data points.
The troublemaker can be identified by classifying objects in the surrounding of the vehicle. For example an object in the surrounding of the vehicle can be a bush.
Alternatively, the object can be a third party vehicle, a wire fence or a pedestrian. The different types of objects differ in their nature and characteristics, for example in their ability to move, in their velocity and in their vulnerability. Furthermore, the different types of objects differ in the likelihood of producing wrong and/or unreliable results, when detecting the object’s position, size, velocity, and/or orientation. For decreasing the risk of accidents, it is preferable to know what kinds or types of objects are located in the surrounding of the ego vehicle. Therefore, the one or more troublemakers can be identified by classifying the objects into different classes, and by considering the class of the object. By identifying the object, also a confidence value about the class of the object can be determined. The information about the class of the object and the confidence can be stored for later use. Preferably identifying one or more troublemakers comprises detecting only objects that belong to a predefined class. This can be achieved by detecting instances of semantic objects of a certain class (such as bushes, or grids) in the 2-dimensional array of data points. Every object class has its own special features that help in classifying the object into the specific class. For example all circles are round. Object class detection uses these special features. For example, when looking for circles, objects that are at a particular distance from a point (i.e. the center) are sought. Identifying troublemakers can also comprise that the troublemaker is tagged with an unambiguous mark establishing therefore the possibility to follow the troublemaker in time.
Preferably the troublemaker is identified by analyzing the 2-dimensional array of data points. For example troublemakers can be identified by pattern recognition algorithms, which automatically discover regularities or irregularities in the 2-dimensional array of data points. Furthermore the 2-dimensional array of data points can be partitioned in multiple segments. For example the 2-dimensional array of data points can be partitioned by a thresholding method, where a threshold value is used for deciding to what segment an individual data point of the 2-dimensional array of data points belongs. Alternatively or additionally clustering methods can be used to partition the 2- dimensional array of data points into clusters/segments. It is also possible to use histogram-based methods, where a histogram is computed from all of the data points in the 2-dimensional array of data points. Color or intensity values can be used as the measure for the histogram. Afterwards, peaks and valleys in the histogram are used to locate the segments in the 2-dimensional array of data points. The goal of segmentation is to partition the 2-dimensional array of data points into segments, wherein the data points belonging to the same segment have one or more common features or the data points consist of an object with a semantic meaning (e.g. a bush). By this process objects and/or troublemakers and/or boundaries (lines, curves, etc.) of objects and/or troublemakers can be identified.
Preferably identifying one or more troublemakers comprises discovering one or more troublemakers coupled with segmenting the 2-dimensional array of data points. When using sequences of images constituting videos or movies object co-segmentation can be applied. The troublemaker can be present sporadically in a set of images or the troublemaker can disappear intermittently throughout the video of interest. For object co- segmentation multiple images or video frames are jointly segmented, based on semantically similar objects. Therefore information about the troublemaker can be shared among consecutive frames and information about motion and appearance of the troublemaker can be used to find the common regions in multiple images belonging to the troublemaker.
Preferably the step of identifying one or more troublemakers in the 2-dimensional array of data points, comprises assigning a label to every data point in the 2-dimensional array of data points, such that data points with the same label share certain predefined characteristics.
The method also comprises the steps of providing a 3-dimensional array of data points representing the surrounding of the vehicle in the range-field-of-view by at least one range-type-sensor, and mapping the 3-dimensional array of data points into the 2- dimensional array of data points. The mapping or projecting of the 3-dimensional array of data points into the 2-dimensional array of data points involves defining translation and rotation parameters to associate the 3-dimensional array of data points with the 2- dimensional array of data points. For this reason, the relative locations of the range type sensor and the image type sensor at the ego vehicle and their respective fields-of-view need to be known.
After mapping the 3-dimensional array of data points into the 2-dimensional array of data points one or more 3D-sub-sets of data points in the 3-dimensional array of data points matching the one or more troublemakers in the 2-dimensional array of data points are selected. Now the specific data points that belong to troublemakers are known. In a next step a revised 3-dimensional array of data points is provided, considering the 3- dimensional array of data points and the one or more 3D-sub-sets of data points.
Preferably only data points that do not belong to troublemakers are retained in the revised 3-dimensional array of data points. The revised 3-dimensional array of data points is used to detect the position, size, velocity, and/or orientation of the objects.
The position of the object is the position of the object with respect to the ego vehicle. Therefore from the position of the object the distance from the object to the ego vehicle can be determined. This information together with the velocity of the object is highly important for calculating the time of a possible collision with an object. The size of an object includes the 3 dimensions width, length and height of an object.
The orientation of the object is the orientation of the object with respect to the ego vehicle. For example a third party vehicle has a front side and a back side, determining an internal coordinate system of the object. A pedestrian has also a front side and a back side determining a pedestrian based internal coordinate system. The orientation of the object is the orientation of the object’s coordinate system with respect to the coordinate system defined by the ego vehicle. For example if a third party vehicle and the ego vehicle are both driving on a straight lane in the same direction, the orientation of the third party vehicle would be parallel to the ego vehicle. This is independent of the location of the third party vehicle, meaning it is regardless if the third party vehicle is driving in front of the ego vehicle or next to the ego vehicle. Flowever, if the third party vehicle is driving on a lane intersecting the lane of the ego vehicle, the orientation of the third party vehicle is different from being parallel. The orientation of an object can be determined for static as well as for moving objects.
The position, size, velocity and/or orientation of the objects considering the revised 3- dimensional array of data points can be achieved by estimating parameters of a mathematical model from the revised 3-dimensional array of data points.
According to a modified embodiment of the invention, the step of identifying one or more objects in the 2-dimensional array of data points, comprises determining an area in the 2-dimensional array of data points belonging to the one or more troublemakers, and the step of selecting one or more 3D-sub-sets of data points in the 3-dimensional array of data points matching the one or more troublemakers comprises selecting one or more 3D-sub-set of data points in the 3-dimensional array of data points that are mapped inside the area belonging to the one or more troublemakers.
The area in the 2-dimensional array of data points belonging to the troublemakers can have any shape. For example it can be a rectangle or an ellipse surrounding the troublemaker. E.g. a rectangular box, also called bounding box, surrounding the troublemaker can be used for determining the area in the 2-dimensional array of data points belonging to the troublemaker. This has the advantage that it has a simple form and is therefore inexpensive to process. Flowever, the area belonging to the troublemaker preferably has a shape retracing the contours of the troublemaker. This has the advantages that the results of the method are more precise than when using a rectangular box. Furthermore, if the troublemaker is identified by segmenting the 2- dimensional array of data points, the segment often already retraces the contour of the troublemaker. Therefore no additional computational resources are needed. If a data point of the 3-dimensional array of data points is mapped inside the area belonging to the troublemaker it is considered to match the troublemaker. Therefore this data point is selected for the 3D-sub-set of data points.
According to a modified embodiment of the invention, the step of providing a revised 3- dimensional array of data points, considering the 3-dimensional array of data points and the one or more 3D-sub-sets of data points comprises removing the one or more 3D- sub-sets of data points from the 3-dimensional array of data points. When using range type data of troublemakers for detecting the position, size, velocity, and/or orientation, the detection results have a high likelihood of being wrong and/or unreliable. After removing the one or more 3D-sub-sets of data points, the revised 3-dimensional array of data points, only consists of data points that do not stem from troublemakers. Therefore by removing the range type data stemming from the troublemaker from the 3- dimensional array of data points the reliability and stability of the detection results are enhanced.
According to a modified embodiment of the invention, the step of identifying one or more troublemakers in the 2-dimensional array of data points comprises identifying one or more objects having an inconsistent reflection surface and/or identifying one or more objects having a discontinuous reflection surface. The troublemaker is an object having an inconsistent reflection surface and/or an object having a discontinuous reflection surface. Since the range type sensor captures information about the three-dimensional structure of the environment by illuminating the objects in the surrounding with pulsed electromagnetic radiation and measuring the reflected pulses, objects that have a discontinuous reflection surface, have a high likelihood to produce wrong and/or unreliable detection results. E.g. the data received by a range type sensor that is reflected from a bush does in general not show the outer contours of the bush. Some of the electromagnetic pulses may have travelled inside the bush and may be reflected by an inner part of the bush. Furthermore, the data points stemming from the bush can be unstable in the sense, that it appears that some part of the bush is moving relative to another part of the bush. For these reasons the range type data of troublemakers tends to produce wrong and/or unreliable detection results. A similar effect occurs with objects having an inconsistent reflection surface, meaning that the magnitude by which the electromagnetic pulse is reflected varies strongly with different locations on the reflection surface of the object. However, due to the sparseness of the range type data such situations are not easily detectable in the range type data itself. Therefore, the method uses the information gathered by the image type sensor to identify troublemakers.
In this regard, according to a modified embodiment of the invention the step of identifying one or more troublemakers in the 2-dimensional array of data points comprises identifying one or more objects of the category tree, bush, hedge, vegetation, grid and/or wire fence. Using the 2-dimensional array of data points gathered by the image type sensor to identify the one or troublemakers has the advantage that an easy, fast and/or reliable identification of the troublemaker can be achieved. The knowledge about what kind of object is a troublemaker is obtained by a heuristic technique.
According to a modified embodiment of the invention, the step of providing 2- dimensional array of data points representing the surrounding of the vehicle in the image-field-of-view by at least one image type sensor comprises providing 2- dimensional array of data points by at least one camera. In a lot of vehicles camera based systems are already integrated to detect the environment. Therefore, using a camera as image type sensor does not generate extra costs when employing the method. Furthermore, cameras with special lenses, e.g. fisheye lenses, which have a wide field-of-view up to 180 degrees or even more, can be used. Cameras provide accurate information about the type of an object. Also, the shape of an object can be determined in the image by established procedures. Furthermore, by using night vision technology, where parts of the electromagnetic spectrum not visible to a human are taken into account, like near-infrared or ultraviolet radiation, and very sensitive detectors are employed, information about objects and/or troublemakers can be gathered by the camera in situations where a human driver is limited.
According to a modified embodiment of the invention, the step of providing 3- dimensional array of data points representing the surrounding of the vehicle in the range-field-of-view by at least one range type sensor comprises providing 3-dimensional array of data points by at least one LiDAR sensor and/or by at least one radar sensor. In many vehicles a LiDAR and/or a radar sensor is/are used in collision avoidance system, for example to measure the distance to a third party vehicle in front of the ego vehicle. LiDAR and/or radar sensors give very accurate information about the position of an object, especially about its distance to the ego vehicle. LiDAR and/or radar sensors can have field-of-views of up to 360 degrees e.g. by using rotating sensors. It is also possible to use several range type sensors, e.g. a combination of a LiDAR and a radar sensor.
According to a modified embodiment of the invention, the step of identifying one or more troublemakers in the 2-dimensional array of data points comprises identifying one or more troublemakers in the 2-dimensional array of data points by using an image recognition algorithm and/or by using a neural network. Processing a 2-dimensional array of data points e.g. from cameras, in particular video data comprising a sequence of multiple frames per second, is very challenging. Huge amounts of data have to be processed in real time in order to reliably detect the surrounding of the vehicle without delays. However, resources of the vehicle for processing the data are limited in respect to space for housing processing devices and also in respect to available computational and electrical power. Even when the technical issues are resolved, in order to provide vehicles at an affordable price, the resources keep limited to their price.
A powerful possibility to process 2-dimensional array of data points is to use a neural network. State of the Art applications of neural networks for image processing are typically based on deep neural networks. Usage of such network types has shown promising results at an affordable price. Neural networks comprise an input and an output layer, as well as one or multiple hidden layers. The hidden layers of a deep neural network typically consist of convolutional layers, pooling layers, fully connected layers and normalization layers. The convolutional layers apply a convolution operation to the input, passing the result to the next layer. The convolution emulates the response of an individual neuron to visual stimuli.
Alternatively or additionally to the use of a neural network, an image recognition algorithm can be used. Image recognition algorithms that can be used are for example genetic algorithms, approaches based on CAD-like object models, like Primal Sketch, appearance-based methods, e.g. edge matching or conquer and search, or other feature-based methods like Histogram of Oriented Gradients (HOG), Haar-like features, Scale-Invariant Feature Transform (SIFT) or Speeded UP Robust Feature (SURF).
It is possible to use segmentation techniques to partition the 2-dimensional array of data points into several parts, according to low-level cues such as color, texture and/or smoothness of boundary. Alternatively or additionally semantic segmentation techniques for example with deep neural networks, can be used, where the 2-dimensional array of data points is partitioned into semantically meaningful parts, and to classify each part into one of the pre-determined classes. Furthermore pixel-wise classification techniques can be used, where each data point of the 2-dimensional array of data points is classified rather than the entire 2-dimensional array of data points or segment.
Preferably a combination of the techniques is used.
After one or more troublemakers are identified in the 2-dimensional array of data points, the class of the troublemaker is known with a certain confidence value. Also the location of the troublemaker in the image and the area belonging to the troublemaker in the image are determined.
According to a modified embodiment of the invention the driving support system comprises at least one camera as image type sensor for providing a 2-dimensional array of data points representing the surrounding of the vehicle in an image-field-of-view and at least one LiDAR sensor and/or at least one radar sensor as range type sensor for providing a 3-dimensional array of data points representing the surrounding of the vehicle in a range-field-of-view. The camera and/or the LiDAR sensor and/or the radar sensor can be installed inside the vehicle or outside the vehicle.
In a modified embodiment of the invention, the driving support system comprises at least one image type sensor for providing a 2-dimensional array of data points having an image-field-of-view of 360 degrees of the surrounding of the vehicle. This can also be achieved by using multiple image type sensors. For example the ego vehicle can use four cameras, one having a field-of-view that covers the sector of the surrounding in front of the ego-vehicle, one having a field-of-view that covers the sector of the surrounding behind the ego-vehicle, and two cameras having field-of-view that respectively cover the sectors of the surrounding on the sides of the ego-vehicle.
Alternatively one or several rotating image type sensors can be used. In a modified embodiment of the invention the driving support system comprises at least one range type sensor for providing a 3-dimensional array of data points having a range- field-of-view of 360 degrees of the surrounding of the vehicle. It is also possible to use multiple range type sensors that have fields-of-view that cover only part of the surrounding. Alternatively one or several rotating range type sensors can be used.
These and other aspects of the invention will be apparent from and elucidated with reference to the embodiments described hereinafter. Individual features disclosed in the embodiments can constitute alone or in combination an aspect of the present invention. Features of the different embodiments can be carried over from one embodiment to another embodiment.
In the drawings:
Fig. 1 shows a method known in the prior art, based on one type of environment sensor, producing a wrong detection result,
Fig. 2 shows a vehicle with a driving support system for performing a method for improved object detection based on two types of environment sensors according to a first, preferred embodiment of the invention, together with a surrounding of the vehicle,
Fig. 3 shows a flow chart of the steps of the method for improved object
detection based on two types of environment sensors, according to the first, preferred embodiment of the invention,
Fig. 4 illustrates the data representing the surrounding of the vehicle provided by the range type sensor and the image type sensor, which is the result of the steps S1 10 and S120 of the method for improved object detection, according to the first, preferred embodiment of the invention,
Fig. 5 illustrates the results of step S210 of the method for improved object detection, which is identifying troublemakers on the data provided by the image type sensor of Fig. 4, according to the first, preferred embodiment of the invention,
Fig. 6 illustrates the result of step S300, selecting a 3D-sub-set, according to the first, preferred embodiment of the invention, and
Fig. 7 illustrates the result of steps S400 and S500 providing a revised 3- dimensional array of data points and detecting position, size, orientation and/or velocity of objects, according to the first, preferred embodiment of the invention.
Fig. 2 shows a vehicle 10 with a driving support system 12 for performing a method for improved object detection based on two types of environment sensors 14, 16 according to a first, preferred embodiment of the invention.
The driving support system 12 comprises two environment sensors 14, 16, whereby one environment sensor 14, 16 is an image type sensor 14, in the preferred embodiment of the invention a camera 14. The other environment sensor 14, 16 is a range type sensor 16, in this preferred embodiment a LiDAR sensor 16. The camera 14 has an image-field- of-view 18, defining a sector of the surrounding of the vehicle 10, from which the camera 14 is able to capture information. The LiDAR sensor 16 has a range-field-of-view 20 defining another sector of the surrounding of the vehicle 10 from which the LiDAR sensor 16 is able to capture information. There is at least a partial overlap of the image- field-of-view 18 and the range-field-of-view 20. In the surrounding of the vehicle 10, there are different objects 22, 24, whereby some of the objects 22, 24 are troublemakers 24 having a high likelihood of leading to wrong detection results, when only using the LiDAR sensor 16. The camera 14 provides a 2-dimensional array of data points 26 representing the surrounding of the vehicle 10 in the image-field-of-view 18 and the LiDAR sensor 16 provides a 3-dimensional array of data points 28 representing the surrounding of the vehicle 10 in the range-field-of-view 20. In the driving support system 12, a neural network is employed, to identify troublemakers 24 in the 2-dimensional array of data points 26.
Figure 3 shows a flowchart of the method for improved object detection based on the two types of environment sensors 14, 16, according to the first, preferred embodiment of the invention. The method is performed using the driving support system 12 in the vehicle 10 of the first embodiment of the invention.
Hereinafter, the individual steps of the method for improved object detection based on the two types of environment sensors 14, 16 are described with reference to the flowchart in figure 3 and the examples in figures 4 to 7.
The method starts with providing data from the environment sensors 14, 16. In step S1 10 a 2-dimensional array of data points 26 representing the surrounding of the vehicle 10 in the image-field-of-view 18 is provided by the camera 14.
Parallel to this, in step S120 a 3-dimensional array of data points 28 representing the surrounding of the vehicle 10 in the range-field-of-view 20 is provided by the LiDAR sensor 16. The method provides data from both sensors 14, 16 in a parallel and continuous manner.
Figure 4 illustrates the data representing the surrounding of the vehicle 10 provided by the camera 14 and the LiDAR sensor 16. Figure 4a) illustrates the 2-dimensional array of data points 26 provided by the camera 14, with objects 22 and troublemakers 24. Figure 4b) illustrates the 3-dimensional array of data points 28, provided by the LiDAR sensor 16.
In a further step S210 of the method, troublemakers 24 are identified in the 2- dimensional array of data points 26 and the area 32 belonging to the identified troublemakers 24 is determined in the 2-dimensional array of data points 26. Fig. 5 illustrates the result of this step. In this case the troublemaker 24, here a bush, and the corresponding area 32 are determined by a semantic segmentation technique and the area 32 has a shape retracing the contours of the troublemaker 24.
Parallel to this step, in step S220 the 3-dimensional array of data points 28 is mapped into the 2-dimensional array of data points 26. In a further step S300 of the method, a 3D-sub-set of data points 27 in the 3-dimensional array of data points 28 that are mapped inside the area 32 belonging to the identified troublemakers 24 are selected. Figure 6 illustrates the 3-dimensional array of data points 28 and the selected 3D-sub- set of data points 27. All data points in the selected 3D-sub-set of data points 27 have been mapped on the area 32 of the bush in Figure 5.
In a further step S400 of the method, these selected 3D-sub-set of data points 27 are removed from the 3-dimensional array of data points 28, therefore providing a revised 3- dimensional array of data points 34. The result of step S400 is shown in figure 7. All the data points belong to the selected 3D-sub-set of data points 27 after step S300 coming from the troublemaker 24 (the bush) have been removed from the 3-dimensional array of data points 28, resulting in the revised 3-dimensional array of data points 34 (figure 7a).
By using the revised 3-dimensional array of data points 34, the next step S500 - detecting the position, size, velocity and/or orientation of the object 22 - is more reliable. No wrong detection results occur, therefore improving the determination of the position, size, velocity and/or orientation of the objects 22 (figure 7b).
Reference signs
1 vehicle
2 range type sensor
3 third party vehicle
4 bush
5 data from range type sensor
6 correct detection result, first vehicle
7 wrong detection result, second vehicle
10 vehicle
12 driving support system
14 image type sensor, camera
16 range type sensor, LiDAR sensor
18 image-field-of-view
20 range-field-of-view
22 object
24 troublemaker
26 2-dimensional array of data points
27 3D-sub-sets of data points
28 3-dimensional array of data points
32 area belonging to troublemaker,
34 revised 3-dimensional array of data points

Claims

Patent claims
1. Method for improved object (22) detection based on two types of environment sensors (14, 16) applied in a driving support system (12) of a vehicle (10), whereas the first type of environment sensor is an image type sensor (14) having an image-field-of-view (18) and the second type of environment sensor is a range type sensor (16) having a range-field-of-view (20) that at least partially overlaps the image-field-of-view (18), comprising the steps of
providing a 2-dimensional array of data points (26) representing the surrounding of the vehicle (10) in the image-field-of-view (18) by at least one image type sensor (14),
identifying one or more troublemakers (24) in the 2-dimensional array of data points (26),
providing a 3-dimensional array of data points (28) representing the surrounding of the vehicle (10) in the range-field-of-view (20) by at least one range-type-sensor (16),
mapping the 3-dimensional array of data points (28) into the 2-dimensional array of data points (26),
selecting one or more 3D-sub-sets of data points (27) in the 3-dimensional array of data points (28) matching the one or more troublemakers (24),
providing a revised 3-dimensional array of data points (34), considering the 3-dimensional array of data points (28) and the one or more 3D-sub-sets of data points (27), and
detecting position, size, velocity, and/or orientation of objects (22), considering the revised 3-dimensional array of data points (34).
2. Method according to any preceding claim, characterized in that
the step of identifying one or more troublemakers (24) in the 2-dimensional array of data points (28) comprises
determining an area (32) in the 2-dimensional array of data points (26) belonging to the one or more troublemakers (24), and the step of selecting one or more 3D-sub-sets of data points (27) in the 3-dimensional array of data points (28) matching the one or more troublemakers (24) comprises
selecting one or more 3D-sub-set of data points (27) in the 3-dimensional array of data points (28) that are mapped inside the area (32) belonging to the one or more troublemakers (24).
3. Method according to any preceding claim, characterized in that
the step of identifying one or more troublemakers (24) in the 2-dimensional array of data points (28), comprises
discovering one or more troublemakers (24) coupled with segmenting the 2-dimensional array of data points (28).
4. Method according to any preceding claim, characterized in that
the step of providing a revised 3-dimensional array of data points (34), considering the 3-dimensional array of data points (28) and the one or more 3D- sub-sets of data points (27) comprises
removing the one or more 3D-sub-sets of data points (27) from the 3- dimensional array of data points (28).
5. Method according to any preceding claim, characterized in that
the step of identifying one or more troublemakers (24) in the 2-dimensional array of data points (26) comprises
identifying one or more objects (22) having an inconsistent reflection surface and/or one or more objects (22) having a discontinuous reflection surface.
6. Method according to any preceding claim, characterized in that
the step of identifying one or more troublemakers (24) in the 2-dimensional array of data points (26) comprises
identifying one or more objects (22) of the category tree, bush, hedge, vegetation, grid and/or wire fence.
7. Method according to any preceding claim, characterized in that the step of providing a 2-dimensional array of data points (26) representing the surrounding of the vehicle (10) in the image-field-of-view (18) by at least one image type sensor (14) comprises
providing a 2-dimensional array of data points (26) by at least one camera
(14).
8. Method according to any preceding claim, characterized in that
the step of providing a 3-dimensional array of data points (28) representing the surrounding of the vehicle (10) in the range-field-of-view (20) by at least one range type sensor (16) comprises
providing a 3-dimensional array of data points (28) by at least one LiDAR sensor (16) and/or by at least one radar sensor.
9. Method according to any preceding claim, characterized in that
the step of identifying one or more troublemakers (24) in the 2-dimensional array of data points (26)
comprises identifying one or more troublemakers (24) in the 2-dimensional array of data points (26) by an image recognition algorithm and/or by a neural network.
10. Driving support system (12) for performing the method according to any of
preceding claims 1 to 9,
comprising at least one image type sensor (14) for providing a 2-dimensional array of data points (26) representing the surrounding of the vehicle (10) in an image-field-of-view (18) and one range type sensor (16) for providing a 3- dimensional array of data points (28) representing the surrounding of the vehicle (10) in a range-field-of-view (20).
1 1. Driving support system (12) according to claim 10, characterized in that
the image type sensor (14) for providing a 2-dimensional array of data points (26) is a camera (14) and the range type sensor (16) for providing a 3-dimensional array of data points (28) is a LiDAR sensor (16) and/or a radar sensor.
12. Driving support system (12) according to claim 10 or 1 1 , characterized in that the at least one image type sensor (14) for providing a 2-dimensional array of data points (26) has an image-field-of-view (18) of 360 degrees of the surrounding of the vehicle (10).
13. Driving support system (12) according to any of claims 10 to 12, characterized in that the at least one range type sensor (16) for providing a 3-dimensional array of data points (28) has a range-field-of-view (20) of 360 degrees of the surrounding of the vehicle (10).
PCT/EP2019/085490 2018-12-19 2019-12-17 Method for improved object detection Ceased WO2020127151A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
DE102018132805.2A DE102018132805A1 (en) 2018-12-19 2018-12-19 Procedure for improved object detection
DE102018132805.2 2018-12-19

Publications (1)

Publication Number Publication Date
WO2020127151A1 true WO2020127151A1 (en) 2020-06-25

Family

ID=69061331

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/EP2019/085490 Ceased WO2020127151A1 (en) 2018-12-19 2019-12-17 Method for improved object detection

Country Status (2)

Country Link
DE (1) DE102018132805A1 (en)
WO (1) WO2020127151A1 (en)

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114518118A (en) * 2020-11-19 2022-05-20 财团法人资讯工业策进会 System and method for generating basic information for positioning and self-positioning judgment device
CN114897684A (en) * 2022-04-25 2022-08-12 深圳信路通智能技术有限公司 Vehicle image splicing method and device, computer equipment and storage medium
EP4180837A1 (en) * 2021-11-14 2023-05-17 Faro Technologies, Inc. Removing reflection from scanned data
CN116400301A (en) * 2023-03-31 2023-07-07 南昌大学 Design Method of Orthogonal Sparse Frequency Waveform Sequence Set Based on CCM Algorithm for MIMO Radar
US12379487B2 (en) 2022-06-07 2025-08-05 Caterpillar Sarl System and method to reliably detect objects using camera and radar

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10802122B1 (en) * 2020-01-15 2020-10-13 Ike Robotics, Inc. Methods and systems for calibration of multiple lidar devices with non-overlapping fields of view
DE102020210380B4 (en) 2020-08-14 2024-10-24 Continental Autonomous Mobility Germany GmbH Method for determining a movement of an object
DE102020210816A1 (en) 2020-08-27 2022-03-03 Robert Bosch Gesellschaft mit beschränkter Haftung Method for detecting three-dimensional objects, computer program, machine-readable storage medium, control unit, vehicle and video surveillance system

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20170371348A1 (en) * 2017-09-07 2017-12-28 GM Global Technology Operations LLC Ground reference determination for autonomous vehicle operations
US20180307921A1 (en) * 2017-04-25 2018-10-25 Uber Technologies, Inc. Image-Based Pedestrian Detection

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8027029B2 (en) * 2007-11-07 2011-09-27 Magna Electronics Inc. Object detection and tracking system
US20180288320A1 (en) * 2017-04-03 2018-10-04 Uber Technologies, Inc. Camera Fields of View for Object Detection

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180307921A1 (en) * 2017-04-25 2018-10-25 Uber Technologies, Inc. Image-Based Pedestrian Detection
US20170371348A1 (en) * 2017-09-07 2017-12-28 GM Global Technology Operations LLC Ground reference determination for autonomous vehicle operations

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
PAN WEI ET AL: "LiDAR and Camera Detection Fusion in a Real Time Industrial Multi-Sensor Collision Avoidance System", ARXIV.ORG, CORNELL UNIVERSITY LIBRARY, 201 OLIN LIBRARY CORNELL UNIVERSITY ITHACA, NY 14853, 11 July 2018 (2018-07-11), XP081251605 *

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114518118A (en) * 2020-11-19 2022-05-20 财团法人资讯工业策进会 System and method for generating basic information for positioning and self-positioning judgment device
TWI768548B (en) * 2020-11-19 2022-06-21 財團法人資訊工業策進會 System and method for generating basic information for positioning and self-positioning determination device
CN114518118B (en) * 2020-11-19 2024-05-28 财团法人资讯工业策进会 Basic information generation system and method for positioning and self-positioning judgment device
EP4180837A1 (en) * 2021-11-14 2023-05-17 Faro Technologies, Inc. Removing reflection from scanned data
CN114897684A (en) * 2022-04-25 2022-08-12 深圳信路通智能技术有限公司 Vehicle image splicing method and device, computer equipment and storage medium
US12379487B2 (en) 2022-06-07 2025-08-05 Caterpillar Sarl System and method to reliably detect objects using camera and radar
CN116400301A (en) * 2023-03-31 2023-07-07 南昌大学 Design Method of Orthogonal Sparse Frequency Waveform Sequence Set Based on CCM Algorithm for MIMO Radar

Also Published As

Publication number Publication date
DE102018132805A1 (en) 2020-06-25

Similar Documents

Publication Publication Date Title
WO2020127151A1 (en) Method for improved object detection
KR102109941B1 (en) Method and Apparatus for Vehicle Detection Using Lidar Sensor and Camera
US11556745B2 (en) System and method for ordered representation and feature extraction for point clouds obtained by detection and ranging sensor
CN110325818B (en) Joint 3D object detection and orientation estimation via multimodal fusion
Rezaei et al. Robust vehicle detection and distance estimation under challenging lighting conditions
US10366310B2 (en) Enhanced camera object detection for automated vehicles
JP5297078B2 (en) Method for detecting moving object in blind spot of vehicle, and blind spot detection device
CN113119963B (en) Intelligent ultrasonic system, vehicle rear collision warning device and control method thereof
US20200082560A1 (en) Estimating two-dimensional object bounding box information based on bird's-eye view point cloud
US20130034298A1 (en) Image-based crack detection
US12243321B2 (en) Method for determining a semantic free space
Chen et al. A visual method tor the detection of available parking slots
Lin et al. CNN-based classification for point cloud object with bearing angle image
CN115100628B (en) Obstacle recognition method and device for automatic driving vehicle
JP7226565B2 (en) Object recognition method and object recognition device
Fu et al. Camera-based semantic enhanced vehicle segmentation for planar lidar
KR20230109406A (en) Apparatus and method for detecting an object based on fusion of thermal imaging camera and lidar sensor
Hwang et al. Vision-based vehicle detection and tracking algorithm design
CN114424254A (en) Object detection circuit and object detection method
CN113902043B (en) Target recognition method, device and equipment
Xiao et al. Parking spot classification based on surround view camera system
US20230267749A1 (en) System and method of segmenting free space based on electromagnetic waves
FR2899363A1 (en) Movable/static object`s e.g. vehicle, movement detecting method for assisting parking of vehicle, involves carrying out inverse mapping transformation on each image of set of images of scene stored on charge coupled device recording camera
Mohottala et al. Fusion of a camera and a laser range sensor for vehicle recognition
Li Object detection and instance segmentation of cables

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19829085

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19829085

Country of ref document: EP

Kind code of ref document: A1