[go: up one dir, main page]

EP3687839B1 - Attelage de remorque automatique utilisant des coordonnées d'image - Google Patents

Attelage de remorque automatique utilisant des coordonnées d'image Download PDF

Info

Publication number
EP3687839B1
EP3687839B1 EP18786555.5A EP18786555A EP3687839B1 EP 3687839 B1 EP3687839 B1 EP 3687839B1 EP 18786555 A EP18786555 A EP 18786555A EP 3687839 B1 EP3687839 B1 EP 3687839B1
Authority
EP
European Patent Office
Prior art keywords
trailer
vehicle
tow vehicle
hitch
path
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
EP18786555.5A
Other languages
German (de)
English (en)
Other versions
EP3687839A1 (fr
Inventor
Kyle CARPENTER
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Continental Autonomous Mobility US LLC
Original Assignee
Continental Autonomous Mobility US LLC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Continental Autonomous Mobility US LLC filed Critical Continental Autonomous Mobility US LLC
Publication of EP3687839A1 publication Critical patent/EP3687839A1/fr
Application granted granted Critical
Publication of EP3687839B1 publication Critical patent/EP3687839B1/fr
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60D—VEHICLE CONNECTIONS
    • B60D1/00—Traction couplings; Hitches; Draw-gear; Towing devices
    • B60D1/24—Traction couplings; Hitches; Draw-gear; Towing devices characterised by arrangements for particular functions
    • B60D1/36—Traction couplings; Hitches; Draw-gear; Towing devices characterised by arrangements for particular functions for facilitating connection, e.g. hitch catchers, visual guide means, signalling aids
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60D—VEHICLE CONNECTIONS
    • B60D1/00—Traction couplings; Hitches; Draw-gear; Towing devices
    • B60D1/58—Auxiliary devices
    • B60D1/62—Auxiliary devices involving supply lines, electric circuits, or the like
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B62—LAND VEHICLES FOR TRAVELLING OTHERWISE THAN ON RAILS
    • B62D—MOTOR VEHICLES; TRAILERS
    • B62D15/00—Steering not otherwise provided for
    • B62D15/02—Steering position indicators ; Steering position determination; Steering aids
    • B62D15/027—Parking aids, e.g. instruction means
    • B62D15/0285—Parking performed automatically
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B62—LAND VEHICLES FOR TRAVELLING OTHERWISE THAN ON RAILS
    • B62D—MOTOR VEHICLES; TRAILERS
    • B62D6/00—Arrangements for automatically controlling steering depending on driving conditions sensed and responded to, e.g. control circuits
    • B62D6/001—Arrangements for automatically controlling steering depending on driving conditions sensed and responded to, e.g. control circuits the torque NOT being among the input parameters
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
    • G05D1/0088—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
    • G05D1/02—Control of position or course in two dimensions
    • G05D1/021—Control of position or course in two dimensions specially adapted to land vehicles
    • G05D1/0212—Control of position or course in two dimensions specially adapted to land vehicles with means for defining a desired trajectory
    • G05D1/0221—Control of position or course in two dimensions specially adapted to land vehicles with means for defining a desired trajectory involving a learning process
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
    • G05D1/02—Control of position or course in two dimensions
    • G05D1/021—Control of position or course in two dimensions specially adapted to land vehicles
    • G05D1/0212—Control of position or course in two dimensions specially adapted to land vehicles with means for defining a desired trajectory
    • G05D1/0223—Control of position or course in two dimensions specially adapted to land vehicles with means for defining a desired trajectory involving speed control of the vehicle
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00—Machine learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G06T7/74—Determining position or orientation of objects or cameras using feature-based methods involving reference images or patches
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30248—Vehicle exterior or interior
    • G06T2207/30252—Vehicle exterior; Vicinity of vehicle

Definitions

  • This disclosure relates to an automotive vehicle, and more particularly to a driver assistance system for automotive vehicles to aid in hitching the vehicle to a trailer.
  • Trailers are usually unpowered vehicles that are pulled by a powered tow vehicle.
  • a trailer may be a utility trailer, a popup camper, a travel trailer, livestock trailer, flatbed trailer, enclosed car hauler, and boat trailer, among others.
  • the tow vehicle may be a car, a crossover, a truck, a van, a sports-utility-vehicle (SUV), a recreational vehicle (RV), or any other vehicle configured to attach to the trailer and pull the trailer.
  • the trailer may be attached to a powered vehicle using a trailer hitch.
  • a receiver hitch mounts on the tow vehicle and connects to the trailer hitch to form a connection.
  • the trailer hitch may be a ball and socket, a fifth wheel and gooseneck, or a trailer jack.
  • the trailer is electrically connected to the tow vehicle.
  • the electrical connection allows the trailer to take the feed from the powered vehicle's rear light circuit, allowing the trailer to have taillights, turn signals, and brake lights that are in sync with the powered vehicle's lights.
  • Some of the challenges that face tow vehicle drivers are connecting the tow vehicle to the trailer, because more than one person is needed. For example, one person drives the vehicle, e.g., the driver, and another one or more people are needed to view the tow vehicle and the trailer and provide the driver with direction regarding the path the tow vehicle has to take to align with the hitch. If the people providing directions to the driver are not accustomed to hitching a tow vehicle to a trailer, then they may have difficulty providing efficient instructions for directing the path of the tow vehicle.
  • Such driver assistance systems use sensors located on the vehicle to detect an ongoing collision.
  • the system may warn the driver of one or more driving situations to prevent or minimize collisions.
  • sensors and cameras may also be used to alert a driver of possible obstacles when the vehicle is traveling in a forward direction. Therefore, it is desirable to provide a system that includes sensors to overcome the challenges faced by drivers of tow vehicles.
  • EP 3 081 405 A2 describes an automated hitch assist system for a vehicle which provides a vehicle operation to move the vehicle into an appropriate location to align a vehicle hitch ball with a hitch of a trailer.
  • the hitch assist system includes a driver input device and one or more cameras to provide a rear view of the vehicle, wherein the respective images are analyzed by a controller. If multiple trailers are detected, an operator can select the appropriate trailer via the driver input device.
  • the hitch assist system then calculates a path from an initial position of the vehicle to a final position in which the hitch ball aligns with the trailer hitch, wherein the vehicle will be aligned in a generally axial manner in the final position.
  • the controller calculates the location of the vehicle hitch ball and the location of the trailer hitch based on information from the camera. Thereby, also an initial orientation between the vehicle and trailer is determined, wherein an initial trailer orientation is determined by the controller by performing image analysis and determining trailer orientation using relative distances between multiple measured locations on the trailer. Once the path is calculated, the controller sends instructions to the other vehicle systems to move the vehicle.
  • One general aspect includes a method of maneuvering a vehicle in reverse for attachment to a trailer, including: detecting, by a neural network of the vehicle, one or more trailers within one or more images, detecting a trailer hitch receiver in the image.
  • the method of maneuvering also includes determining, at a computing device in communication with the neural network, a pixel angular difference in the image between a tow vehicle fore-aft axis and a trailer fore-aft axis, determining a pixel distance between a tow vehicle hitch ball and a hitch receiver, determining a vehicle path from an initial position to a final position adjacent the trailer.
  • the vehicle path including maneuvers configured to move the vehicle along the vehicle path from the initial position to the final position.
  • Also including determining vehicle path including maneuvers configured to move the vehicle along the vehicle path from the initial position to the final position and then autonomously following, at a drive system in communication with the computing device, the vehicle path from the initial position to the final position.
  • One general aspect includes a method of maneuvering a vehicle in reverse for attachment to a trailer, including: receiving a trailer hitch receiver image location selected by a driver using a user interface, where the trailer hitch receiver is shown within one or more images from at least one vehicle camera.
  • the method of maneuvering also includes determining, at a computing device, a pixel angular difference in the image between a tow vehicle fore-aft axis and a trailer fore-aft axis, determining a pixel distance between a tow vehicle hitch ball and a hitch receiver, determining a vehicle path from an initial position to a final position adjacent the trailer.
  • the vehicle path including maneuvers configured to move the vehicle along the vehicle path from the initial position to the final position.
  • Also including determining vehicle path including maneuvers configured to move the vehicle along the vehicle path from the initial position to the final position and then autonomously following, at a drive system in communication with the computing device, the vehicle path from the initial position to the final position.
  • a tow vehicle such as, but not limited to a car, a crossover, a truck, a van, a sports-utility-vehicle (SUV), and a recreational vehicle (RV) may be configured to tow a trailer.
  • the tow vehicle connects to the trailer by way of a trailer hitch. It is desirable to have a tow vehicle that is capable to having a more automated system and method for maneuvering towards a trailer and attaching to the trailer, thus reducing the need for a driver to drive the tow vehicle in a rearward direction while another one or more people provide the driver with directions regarding the path that the tow vehicle has to take to align with the trailer and ultimately a hitch of the trailer.
  • a tow vehicle with an assistance device for rearward driving provides a driver with a safer and faster experience when hitching the tow vehicle to the trailer.
  • a driver of a tow vehicle 100 wants to tow a trailer 200.
  • the tow vehicle 100 may be configured with a driver assistance system 310 to provide guidance to the driver to drive towards the selected trailer 200.
  • the tow vehicle 100 may include a drive system 110 that maneuvers the tow vehicle 100 across a road surface based on drive commands having x, y, and z components, for example.
  • the drive system 110 includes a front right wheel 112, 112a, a front left wheel 112, 112b, a rear right wheel 112, 112c, and a rear left wheel 112, 112d.
  • the drive system 110 may include other wheel configurations as well.
  • the drive system 110 may also include a brake system 120 that includes brakes associated with each wheel 112, 112a-d, and an acceleration system 130 that is configured to adjust a speed and direction of the tow vehicle 100.
  • the drive system 110 may include a suspension system 132 that includes tires associates with each wheel 112, 112a-d, tire air, springs, shock absorbers, and linkages that connect the tow vehicle 100 to its wheels 112, 112a-d and allows relative motion between the tow vehicle 100 and the wheels 112, 112a-d.
  • the suspension system 132 improves the road handling of the tow vehicle 100 and provides a better ride quality by isolating road noise, bumps, and vibrations.
  • the suspension system 132 is configured to adjust a height of the tow vehicle 100 allowing the tow vehicle hitch 160 to align with the trailer hitch 210, which aids in connection between the tow vehicle 100 and the trailer 200.
  • the tow vehicle 100 may move across the road surface by various combinations of movements relative to three mutually perpendicular axes defined by the tow vehicle 100: a transverse axis X, a fore-aft axis Y, and a central vertical axis Z.
  • the transverse axis x extends between a right side R and a left side of the tow vehicle 100.
  • a forward drive direction along the fore-aft axis Y is designated as F, also referred to as a forward motion.
  • an aft or rearward drive direction along the fore-aft direction Y is designated as R, also referred to as rearward motion.
  • the suspension system 132 adjusts the suspension of the tow vehicle 100, the tow vehicle 100 may tilt about the X axis and or Y axis, or move along the central vertical axis Z.
  • a driver assistance system 310 may include a user interface 140.
  • the user interface 140 may be a display that is incorporated into the vehicle or may be provided on a separate device, such as a personal wireless device.
  • the user interface 140 receives one or more user commands from the driver via one or more input mechanisms or a touch screen display 142 and/or displays one or more notifications to the driver.
  • the user interface 140 is in communication with a vehicle controller 300, which is in turn in communication and sensor system 400 and a drive system 110.
  • the user interface 140 displays an image of an environment of the tow vehicle 100 leading to one or more commands being received by the user interface 140 (from the driver) that initiate execution of one or more behaviors.
  • the vehicle controller 300 includes a computing device (or processor) 302 (e.g., central processing unit having one or more computing processors) in communication with non-transitory memory 304 (e.g., a hard disk, flash memory, random-access memory) capable of storing instructions executable on the computing processor(s)).
  • a computing device or processor
  • non-transitory memory 304 e.g., a hard disk, flash memory, random-access memory
  • the vehicle controller 300 executes a driver assistance system 310, which in turn includes a path following sub-system 320.
  • the path following sub-system 320 receives a planned path 552 ( FIGS. 4A and 4B ) from a path planning system 550 and executes behaviors 322-330 that send commands 301 to the drive system 110, leading to the tow vehicle 100 autonomously driving about the planned path 552 in a rearward direction R.
  • the path following sub-system 320 includes, a braking behavior 322, a speed behavior 324, a steering behavior 326, a hitch connect behavior 328, and a suspension adjustment behavior 330.
  • Each behavior 322-330 cause the tow vehicle 100 to take an action, such as driving backward, turning at a specific angle, breaking, speeding, slowing down, among others.
  • the vehicle controller 300 may maneuver the tow vehicle 100 in any direction across the road surface by controlling the drive system 110, more specifically by issuing commands 301 to the drive system 110. For example, the vehicle controller 300 may maneuver the tow vehicle 100 from an initial position (as shown in Fig. 4A ) to a final position (as shown in FIG. 5C ). In the final position, a hitch ball 162 of the tow vehicle 100 aligns with a hitch coupler 212 of the trailer 200 connecting the tow vehicle 100 and the selected trailer 200.
  • the tow vehicle 100 may include a sensor system 400 to provide reliable and robust autonomous driving.
  • the sensor system 400 may include different types of sensors that may be used separately or with one another to create a perception of the tow vehicle's environment that is used for the tow vehicle 100 to autonomously drive and make intelligent decisions based on objects and obstacles detected by the sensor system 400.
  • the sensors may include, but not limited to, one or more imaging devices (such as cameras) 410, and sensors 420 such as, but not limited to, radar, sonar, LIDAR (Light Detection and Ranging, which can entail optical remote sensing that measures properties of scattered light to find range and/or other information of a distant target), LADAR (Laser Detection and Ranging), etc.
  • the camera(s) 410 and the sensor(s) 420 may be used to alert the driver of possible obstacles when the tow vehicle 100 is traveling in the forward direction F or in the rearward direction R, by way of audible alerts and/or visual alerts via the user interface 140. Therefore, the sensor system 400 is especially useful for increasing safety in tow vehicles 100 which operate under semi-autonomous or autonomous conditions.
  • the tow vehicle 100 includes a rear camera 410, 410a that is mounted to provide a view of a rear driving path for the tow vehicle 100. Additionally, in some examples, the tow vehicle 100 includes a front camera 410, 410b to provide a view of a front driving path for the tow vehicle 100, a right camera 410, 410c positioned on the right side of the tow vehicle 100, and a left camera 410, 410d positioned on the left side of the tow vehicle 100. The left and right cameras 410, 410c, 410d provide additional side views of the tow vehicle 100.
  • the tow vehicle 100 may detect object and obstacles positioned on either side of the tow vehicle 100, in addition to the objects and obstacle detected along the front and rear driving paths.
  • the camera(s) 410, 410a-d may be a monocular camera, binocular camera, or another type of sensing device capable of providing a view of the rear travelling path of the tow vehicle 100.
  • the tow vehicle 100 includes one or more Neural Networks (NN) 500, for example, Deep Neural Networks (DNN) to improve the autonomous driving of the tow vehicle 100.
  • DNNs 500 are computational approaches used in computer science, among other disciplines, and are based on a large collection of neural unites, loosely imitating the way a biological brain solves problems with large clusters of biological neurons connected by axons.
  • DNNs 500 are self-learning and trained, rather than programed, and excel in areas where the solution feature detection is difficult to express in a traditional computer program. In other words, DNNs 500 are a set of algorithms that are designed to recognize patterns.
  • DNNs 500 interpret sensor system data 402 (e.g., from the sensor system 400) through a machine perception, labeling or clustering raw input.
  • the recognized patters are numerical, vectors, into which all-real-world data, such as images, text, sound, or time series is translates.
  • the DNN 500 includes multiple layers of nonlinear processing units 502 in communication with DNN non-transitory memory 504.
  • the DNN non-transitory memory 504 stores instructions that when executed on the nonlinear processing units 502 cause the DNN 500 to provide an output 506, 508.
  • Each nonlinear processing unit 502 is configured to transform an input or signal (e.g., sensor system data 402) using parameters that are learned through training.
  • the DNN 500 is capable of determining the location based on images 412 or sensor data 422 eliminating the need to have a DGPS or a GPS.
  • the DNN 500 receives sensor system data 402 (including images 412 and/or sensor data 422) and based on the received data 420 provides an image output 506 to the user interface 140 and/or a data output 508 to the vehicle controller 300.
  • the DNN 500 receives image(s) 412 of a rear view of the tow vehicle 100 from the camera 410 in communication with the DNN 500.
  • the DNN 500 analyzes the image 412 and identifies a trailer 200 and a trailer hitch receiver 212 in the received image 412.
  • the DNN 500 may also receive sensor data 420 from the sensors 420 in communication with the DNN 500, and analyze the received sensor data 420.
  • the DNN 500 Based on the analyzed images 412 (or the analyzed images 412 and the sensor data 422), the DNN 500 identifies the location of the trailer hitch receiver 212 relative to the tow vehicle hitch ball 162, as described in further detail below. As such, the DNN 500 displays on the user interface 140 the received images 412 displaying representations 146 of the trailer 200 located at a distance behind the tow vehicle 100.
  • the user interface is a touch screen display 142. In other examples, the user interface 140 is not a touchscreen and the driver may use an input device, such as, but not limited to, a rotary knob or a mouse.
  • the location of the trailer hitch receiver 212 is determined by the a Hitch Location System 520.
  • the Hitch Location System 520 uses the DNN 500 or controller 300 to calculate image coordinate of the location of the trailer hitch receiver 212 (X T , Y T ) and for the vehicle hitch ball location (X V , Y V ) in the image 144.
  • a boundary for detecting selectable trailers is schematically shown.
  • the boundary may be limited by the field of view of the camera(s) 410, 410a or by limited by an angular difference between the vehicle 100 and the trailer 200 for which the DNN 500 can perform the image analysis and calculations, or may be limited by the by an angular difference between the vehicle 100 and the trailer 200 for which the path following sub-system 320 can perform.
  • the boundary and the Hitch Location System 520 calculations are schematically illustrated on image 144 but may be entirely partially or may not actually be displayed in the image 144.
  • the DNN 500 or controller 300 can be taught to recognize the hitch receiver 212 in the image 144 as well as other trailer information, such as at least one horizontal trailer edge 220. Based on the detected trailer information the DNN 500 can calculate angular difference ⁇ between the tow vehicle fore-aft axis Y and a trailer fore-aft axis T.
  • a path planning system 550 plans a path 552 ( FIG. 4A , 4B ) between the tow vehicle 100 and the trailer 200 based on the location of the selected trailer 200 (determined by the DNN 500 or controller 300 from the received sensor system data 402) relative to a position of the tow vehicle 100 (e.g., orientation and distance). As the tow vehicle 100 is autonomously backing up towards the selected trailer 200. The planned path 552 allows the tow vehicle 100 to autonomously drive and connect to the trailer 200.
  • the path planning system 550 plans the path 552 for the tow vehicle 100 to autonomously maneuver such that the tow vehicle 100, in an intermediate position within a predetermined distance D from the trailer 200, is in an orientation aligned generally parallel with the trailer 200 and facing away from the trailer 200, where the hitch 160 of the tow vehicle 100 is substantially aligned with the hitch 210 of the trailer 200.
  • the angular difference ⁇ between the tow vehicle fore-aft axis Y and the trailer fore-aft axis T and the pixel distance ( ⁇ ) between the hitch ball 162 and the hitch receiver 212 are input into the control strategy shown to provide a steer request. Further details on the calculation of the steer request by the path planning system 550, 550B are described below.
  • the path planning system 550 is part of the vehicle controller 300 as shown in FIG. 2A ; while in other examples, the path planning system 550 is part of the DNN 500 as shown in FIG. 2B .
  • the DNN 500a sends the controller 300 data output 508 including the location of the selected trailer 200 with respect to the tow vehicle 100, as received from the user input device 140.
  • FIG. 2C illustrates an exemplary embodiment of the tow vehicle 100 wherein the path planning system 550 is part of a controller and a DNN500 is not utilized.
  • the driver of the tow vehicle 100 would be responsible for selecting the trailer receiver location 212 in the image.
  • the path planning system 550a plans the path 552 between the tow vehicle 100 and the selected trailer 200.
  • the path planning system 550a may use several methods to determine the path 552.
  • FIGS. 4A and 4B provide a method for path planning.
  • the path planning system 550a extends its fore-aft axis Y in the rearward direction R while the trailer 200 extends a fore-aft axis about the length of the trailer 200 in a forward direction.
  • the path planning system 550a draws a first circle 554 tangent at a first tangent point 555 to the tow vehicle fore-aft axis Y facing the trailer fore-aft axis, and a second circle 556 tangent at a second tangent point 557 to the trailer fore-aft axis facing the tow vehicle fore-aft axis Y.
  • the first and second circles 554, 556 intersect at an intersection point 558.
  • the size of the first and second circles 554, 556 may be adjusted and manipulated based on the distance between the tow vehicle 100 and the trailer 200, obstacles and object positioned between the tow vehicle 100 and the trailer 200, and any other considerations.
  • the path planning system 550a determines the path 552 by following the tow vehicle fore-aft axis Y until the first tangent point 555, then moving along an arc of the first circle 554 until the intersection point 558, then moving along an arc of the second circle 556 until the second tangent point 557, then following the trailer fore-aft axis.
  • the planned path 552 positions the tow vehicle 100 in an orientation aligned generally parallel with the trailer 200, where the hitch 160 of the tow vehicle 100 is substantially aligned with the hitch 210 of the trailer 200.
  • FIG. 4A shows an example of the path 552 where the fore-aft axis Y of the tow vehicle 100 is substantially parallel to the fore-aft axis T of the trailer 200.
  • FIG. 4B shows an example of the path 552 where the fore-aft axis Y of the tow vehicle 100 is not substantially parallel to the fore-aft axis T of the trailer 200.
  • the DNN 500 continuously sends the vehicle controller 300 the location of the selected trailer 200 with respect to the tow vehicle 100 as the tow vehicle 100 moves along the planned path 552 based on the received sensor system data 402, i.e., images 412.
  • the DNN 500 identifies one or more objects along the planned path 552 and sends the path planning system 550a data relating to the position of the one or more objects.
  • the path planning system 550a may recalculate the planned path 552 to avoid the one or more objects.
  • the path planning system 550a determined a probability of collision and if the probability of collision exceeds a predetermined threshold, the path planning system 550a adjusts the path and sends it to the path following sub-system 320.
  • the path following sub-system 320 includes one or more behaviors 322-330 that once executed allow for the autonomous driving of the tow vehicle 100 along the planned path 552.
  • the behaviors 322-330 may include, but are not limited to a braking behavior 322, a speed behavior 324, a steering behavior 326, a hitch connect behavior 328, and a suspension adjustment behavior 330.
  • the braking behavior 322 may be executed to either stop the tow vehicle 100 or to slow down the tow vehicle based on the planned path 552.
  • the braking behavior 322 sends a signal or command 301 to the drive system 110, e.g., the brake system 120, to either stop the tow vehicle 100 or reduce the speed of the tow vehicle 100.
  • the speed behavior 324 may be executed to change the speed of the tow vehicle 100 by either accelerating or decelerating based on the planned path 552.
  • the speed behavior 324 sends a signal or command 301 to the brake system 120 for decelerating or the acceleration system 130 for accelerating.
  • the steering behavior 326 may be executed to change the direction of the tow vehicle 100 based on the planned path. As such, the steering behavior 326 sends the acceleration system 130 a signal or command 301 indicative of an angle of steering causing the drive system 110 to change direction.
  • FIGS. 4A-5C show the tow vehicle 100 at an initial position Pi ( FIG. 4A ), an intermediate position P M ( FIG. 4B ), and a final position P F ( FIG. 5C ) or a connected position, with respect to the selected trailer 200.
  • the tow vehicle 100 is at the initial position Pr relative to the trailer 200 prior to initiating autonomous maneuvering towards the selected trailer 200.
  • an initial distance D I between the tow vehicle 100 and the selected trailer 200 is about 15 meters.
  • the tow vehicle 100 autonomously maneuvers along the planned path 552 until the tow vehicle 100 reaches an intermediate position P M being an intermediate distance D M from the selected trailer 200, as shown in FIG. 4B .
  • the tow vehicle hitch 160 is in an orientation aligned generally parallel with the selected trailer 200 and the tow vehicle hitch 160 is substantially aligned with the hitch 210 of trailer hitch 210.
  • the vehicle fore-aft Y defines a plane that extends along the vehicle vertical axis Z and along the trailer fore-aft T along a trailer vertical axis.
  • the intermediate distance D M is about 1 meter.
  • the hitch connect behavior 328 executes to connect the vehicle hitch 160 with the trailer hitch 210.
  • the DNN 500 determines a relative height H R between a top portion of the tow vehicle hitch ball 162 and a bottom portion of the trailer hitch coupler 212.
  • the trailer hitch coupler 212 releasably receives the tow vehicle hitch ball 162.
  • the relative height H R has to equal zero allowing the tow vehicle hitch ball 162 to move under and be inserted in the trailer hitch coupler 212. Therefore, when the hitch connect behavior 328 receives the relative height H R that is greater than zero between the tow vehicle hitch ball 162 and the trailer hitch coupler 212 from the DNN 500, the hitch connect behavior 328 sends a command to the suspension adjustment behavior 330 to execute and issue a command 301 to the suspension system 132 to adjust the height of the tow vehicle 100 reducing the relative height H R based on the measurements from the DNN 500.
  • the hitch connect behavior 328 When the hitch connect behavior 328 receives the relative height H R that is equal to zero, then the hitch connect behavior 328 issues a command 301 to the drive system 110 to maneuver along the remainder of the path 552, i.e., from the intermediate position P M to the final position P F ( FIG. 5C ), connecting the tow vehicle 100 to the selected trailer 200.
  • this may be completed by the controller 300 rather than a DNN 500.
  • FIG. 7 illustrates an example arrangement of operations for a method 700 of autonomously maneuvering a tow vehicle 100 (as shown in FIGS. 1-5 ) towards a selected trailer 200.
  • the method 700 includes receiving an indication that a driver wants to autonomously hitch the tow vehicle 100 to a trailer 200.
  • the indication may be by way of a selection on the user interface 140 of the tow vehicle 100, putting the tow vehicle in reverse (without reversing), or any other indication.
  • the controller 300/DNN 500 detects a trailer and identifies a trailer hitch receiver and other trailer data, including at least one trailer edge.
  • the method 700 determines the the angular difference ⁇ between the tow vehicle fore-aft axis Y and the trailer fore-aft axis T and the pixel distance ( ⁇ ) between the hitch ball 162 and the hitch receiver 212 with the hitch location system 520.
  • the method 700 includes planning a path 552 from an initial position P I of the tow vehicle 100 to a final position P F with respect to the selected trailer 200.
  • the path planning system plans the path 552.
  • the path planning system 550 may be part of the controller 300 or part of the DNN 500.
  • the method 700 includes executing the path following sub-system 320.
  • the method 700 determines if the tow vehicle 100 is within a predetermined distance from the selected trailer 200, i.e., the method 700 determines if the tow vehicle 100 has reached the intermediate position P M .
  • the method 700 at decision block 712 determines a relative height H R between a top portion of the hitch ball 162 of the tow vehicle 100 and a bottom portion of the hitch coupler 212 of the selected trailer 200 and determines if the hitch coupler 212 can releasably receive the hitch ball 162 based on the relative height H R . In other words, the method 700 determines if the relative height H R equals to zero.
  • the method 700 adjusts the suspension of the tow vehicle 100 and then determines the relative height H R and checks if the relative height H R equals zero at block 714. Once relative height H R is equal to zero, then the method 700 at block 618 continues maneuvering about the path 552 from the intermediate position P M to a final position P F connecting the hitch ball 162 of the tow vehicle 100 with the hitch coupler 212 of the selected trailer 200.
  • the above implementations may include one or more of the following features.
  • the final position places the tow vehicle hitch coupler 162 within a distance from the center of the trailer hitch coupler 212 that is less than or equal to the size of the trailer hitch coupler 212.
  • the final position places the tow vehicle hitch 162 within a distance from the center of the trailer hitch coupler 212 that is less than or equal to half the size of the trailer hitch coupler 212.
  • the final position places the tow vehicle hitch 162 within a distance from the center of the trailer hitch coupler 212 is sufficiently close to couple the tow vehicle hitch 162 and the trailer hitch coupler 212 to one another without further maneuvers of the tow vehicle 100.
  • implementations of the systems and techniques described here can be realized in digital electronic circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and/or combinations thereof.
  • ASICs application specific integrated circuits
  • These various implementations can include implementation in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.
  • Implementations of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them.
  • subject matter described in this specification can be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a computer readable medium for execution by, or to control the operation of, data processing apparatus.
  • the computer readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a composition of matter effecting a machine-readable propagated signal, or a combination of one or more of them.
  • data processing apparatus encompass all apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers.
  • the apparatus can include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.
  • a propagated signal is an artificially generated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal that is generated to encode information for transmission to suitable receiver apparatus.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Transportation (AREA)
  • Mechanical Engineering (AREA)
  • Theoretical Computer Science (AREA)
  • Aviation & Aerospace Engineering (AREA)
  • Automation & Control Theory (AREA)
  • Remote Sensing (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Combustion & Propulsion (AREA)
  • Chemical & Material Sciences (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Software Systems (AREA)
  • Medical Informatics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Multimedia (AREA)
  • Health & Medical Sciences (AREA)
  • General Engineering & Computer Science (AREA)
  • Mathematical Physics (AREA)
  • Data Mining & Analysis (AREA)
  • Computing Systems (AREA)
  • Game Theory and Decision Science (AREA)
  • Business, Economics & Management (AREA)
  • Control Of Driving Devices And Active Controlling Of Vehicle (AREA)
  • Vehicle Body Suspensions (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)

Claims (9)

  1. Procédé de manoeuvre d'un véhicule de remorquage (100) en marche arrière pour l'attacher à une remorque (200), le procédé comprenant :
    la détection, par un réseau neuronal (500) du véhicule de remorquage (100), d'une ou de plusieurs remorques (200) au sein d'une ou de plusieurs images (144, 412) ;
    la détection, par le réseau neuronal (500), d'un récepteur d'attelage (212) de remorque dans l'image (144, 412) ;
    la détermination, au niveau d'un dispositif informatique (300) en communication avec le réseau neuronal (500), d'une différence angulaire en pixels (θ) dans l'image (144, 412) entre un axe avant-arrière de véhicule de remorquage (100) dans l'image (144, 412) et un axe avant-arrière de remorque dans l'image (144, 412) ;
    la détermination, au niveau du dispositif informatique, d'une distance en pixels (λ) dans l'image (144, 412) entre une coordonnée d'image d'un emplacement d'une boule d'attelage (162) de véhicule de remorquage et une coordonnée d'image d'un emplacement du récepteur d'attelage (212) de remorque ;
    la détermination, au niveau du dispositif informatique (300), d'une trajectoire (552) de véhicule depuis une position initiale (PI) jusqu'à une position finale (PF) adjacente à la remorque (200), le trajectoire (552) de véhicule comprenant des manoeuvres configurées pour déplacer le véhicule de remorquage (100) le long de la trajectoire (552) de véhicule depuis la position initiale (PI) jusqu'à la position finale (PF) ; et
    le suivi autonome, au niveau d'un système de conduite (110) en communication avec le dispositif informatique (300), de la trajectoire (552) de véhicule depuis la position initiale (PI) jusqu'à la position finale (PF),
    la trajectoire de véhicule étant déterminée par prolongement d'un axe avant-arrière (Y) de véhicule de remorquage dans la direction arrière (R) et par prolongement d'un axe avant-arrière (T) de remorque sur la longueur de la remorque (200, 200a-c) dans une direction avant, la trajectoire de véhicule étant déterminée comme empruntant des arcs d'un premier cercle (554) tangent à l'axe avant-arrière (Y) de véhicule de remorquage prolongé faisant face à l'axe avant-arrière (T) de remorque et d'un deuxième cercle (556) tangent au premier cercle (554) et à l'axe avant-arrière (T) de remorque prolongé faisant face à l'axe avant-arrière (Y) de véhicule de remorquage, et les premier et deuxième cercles (554, 556) se coupant en un point d'intersection (558).
  2. Procédé selon la revendication 1, comprenant en outre l'arrêt ou l'immobilisation, au niveau du système de conduite (110), du véhicule de remorquage (100) en une position intermédiaire (PM) avant qu'il n'atteigne la position finale (PF), la position intermédiaire (PM) étant plus proche de la position finale (PF) que de la position initiale (PI) ;
    la modification, au niveau du système de conduite (110), d'une ou de plusieurs suspensions de véhicule associées au véhicule de remorquage (100) afin d'aligner un attelage (160) de véhicule avec un attelage (210) de remorque ;
    le suivi autonome, au niveau du système de conduite (110), de la trajectoire (552) de véhicule depuis la position initiale (PI) jusqu'à la position finale (PF) ; et
    le rattachement, au niveau du système de conduite (110), de la boule d'attelage (163) de véhicule au récepteur d'attelage (212) de remorque.
  3. Procédé selon la revendication 1, dans lequel le rattachement de la boule d'attelage (162) de véhicule de remorquage au récepteur d'attelage (212) de remorque comprend la modification d'une ou de plusieurs suspensions de véhicule (100) associées au véhicule de remorquage (100) afin d'aligner un attelage (160) de véhicule avec un attelage (210) de remorque.
  4. Procédé selon la revendication 1, dans lequel les manoeuvres comportent un braquage, un freinage et une accélération.
  5. Procédé selon la revendication 1, comprenant en outre :
    la détection continue, au niveau du réseau neuronal (500), d'un ou de plusieurs objets au sein de la trajectoire (552) de véhicule à mesure que le véhicule de remorquage (100) se déplace le long de la trajectoire (552) de véhicule ; et
    lors de la détection d'un objet, le changement de la trajectoire (552) de véhicule au niveau du dispositif informatique (300).
  6. Procédé selon la revendication 1, dans lequel la détection d'une ou de plusieurs remorques (200) comprend :
    la capture, au niveau d'un ou de plusieurs dispositifs d'imagerie (400, 410) en communication avec le réseau neuronal (500), d'une ou de plusieurs images (144, 412), au moins un des un ou plusieurs dispositifs d'imagerie (400, 410) étant positionné à l'arrière de la remorque (200) en étant orienté dans une direction arrière ; et
    la détermination, au niveau du réseau neuronal (500), du récepteur d'attelage (212) de remorque au sein des une ou plusieurs images (144, 412).
  7. Procédé selon la revendication 1, dans lequel la position finale (PF) place l'attelage (160) de véhicule de remorquage à une distance d'un centre d'un accouplement d'attelage (212) de remorque qui est inférieure ou égale à un diamètre de l'accouplement d'attelage (212) de remorque.
  8. Procédé selon la revendication 7, dans lequel la position finale place l'attelage (160) de véhicule de remorquage à une distance du centre de l'accouplement d'attelage (212) de remorque qui est inférieure ou égale à la moitié du diamètre de l'accouplement d'attelage (212) de remorque.
  9. Procédé selon la revendication 1, dans lequel la position finale (PF) place l'attelage (160) de véhicule de remorquage à une distance d'un centre d'un accouplement d'attelage (212) de remorque qui est suffisamment proche pour permettre d'accoupler la boule d'attelage (162) de véhicule de remorquage et l'accouplement d'attelage (212) de remorque sans autres manoeuvres du véhicule de remorquage (100).
EP18786555.5A 2017-09-25 2018-09-25 Attelage de remorque automatique utilisant des coordonnées d'image Active EP3687839B1 (fr)

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
US201762562751P 2017-09-25 2017-09-25
US16/139,531 US10987984B2 (en) 2017-09-25 2018-09-24 Automated trailer hitching using image coordinates
PCT/US2018/052661 WO2019060893A1 (fr) 2017-09-25 2018-09-25 Attelage de remorque automatique utilisant des coordonnées d'image

Publications (2)

Publication Number Publication Date
EP3687839A1 EP3687839A1 (fr) 2020-08-05
EP3687839B1 true EP3687839B1 (fr) 2023-08-23

Family

ID=65807165

Family Applications (1)

Application Number Title Priority Date Filing Date
EP18786555.5A Active EP3687839B1 (fr) 2017-09-25 2018-09-25 Attelage de remorque automatique utilisant des coordonnées d'image

Country Status (5)

Country Link
US (1) US10987984B2 (fr)
EP (1) EP3687839B1 (fr)
JP (1) JP6938793B2 (fr)
CN (1) CN111372795B (fr)
WO (1) WO2019060893A1 (fr)

Families Citing this family (25)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10427716B2 (en) * 2017-09-12 2019-10-01 Ford Global Technologies, Llc Hitch assist system and method
US20190129429A1 (en) * 2017-10-26 2019-05-02 Uber Technologies, Inc. Systems and Methods for Determining Tractor-Trailer Angles and Distances
US11518204B2 (en) * 2018-05-01 2022-12-06 Continental Autonomous Mobility US, LLC Trailer detection and autonomous hitching
US11385651B2 (en) * 2018-06-26 2022-07-12 Ford Global Technologies, Llc System and methods for detection and response to interference between trailer coupler and hitch ball
JP7169121B2 (ja) * 2018-08-23 2022-11-10 日立Astemo株式会社 車両連結支援装置、車両連結支援方法、及び車両連結支援システム
US20200097021A1 (en) * 2018-09-19 2020-03-26 Continental Automotive Systems, Inc. Autonomous Farm Equipment Hitching To A Tractor
US11030476B2 (en) * 2018-11-29 2021-06-08 Element Ai Inc. System and method for detecting and tracking objects
US20200238912A1 (en) * 2019-01-30 2020-07-30 AISIN Technical Center of America, Inc. Backup camera system for a vehicle having a trailer hitch
US10933914B2 (en) * 2019-01-31 2021-03-02 Ford Global Technologies, Llc Trailer hitching aid
US11584177B2 (en) * 2019-09-13 2023-02-21 Ford Global Technologies, Llc Fixed offset method for hitch detection in hitch assist operation
US11347968B2 (en) * 2020-02-25 2022-05-31 Ford Global Technologies, Llc Image enhancement for realism
CN111854782A (zh) * 2020-06-22 2020-10-30 北京九曜智能科技有限公司 一种可实现自动脱挂钩脱挂的路径规划的方法
JP7528367B2 (ja) * 2020-09-11 2024-08-05 コンチネンタル オートノマス モビリティ ユーエス，エルエルシー 牽引ボール位置検出システム及び方法
US11403767B2 (en) 2020-09-15 2022-08-02 Connaught Electronics, Ltd. Method and apparatus for detecting a trailer, tow-ball, and coupler for trailer hitch assistance and jackknife prevention
US11721108B2 (en) 2020-11-10 2023-08-08 Ford Global Technologies, Llc System and method for training trailer detection systems
US20220203786A1 (en) * 2020-12-30 2022-06-30 Continental Automotive Systems, Inc. Method and assembly for positioning a vehicle relative to a trailer
CN112904363B (zh) * 2021-01-19 2023-04-25 北京九曜智能科技有限公司 一种自动驾驶车辆对接拖车自动托挂钩的方法
DE102021002956B3 (de) * 2021-06-10 2022-08-11 Jost-Werke Deutschland Gmbh Verfahren zum Bewegen eines Fahrzeugs an eine Komponente eines hierzu beabstandeten Objekts (Koordinatentransformation)
DE102021002955B3 (de) * 2021-06-10 2022-08-11 Jost-Werke Deutschland Gmbh Verfahren zum Bewegen eines Fahrzeugs an eine Komponente eines hierzu beabstandeten Objekts (Vorpositionierungspunkt)
CN113771561A (zh) * 2021-09-23 2021-12-10 广州小鹏汽车科技有限公司 车辆拖挂方法、电子设备、牵引车辆及存储介质
CN114211926B (zh) * 2021-12-31 2022-08-05 重庆工业职业技术学院 一种颠簸路面的汽车悬架控制系统
US12400448B2 (en) 2022-02-02 2025-08-26 Ford Global Technologies, Llc Systems and methods for a step and scan detection network for hitchball location estimation
CN116945826B (zh) * 2023-08-17 2024-01-30 合肥马格勒斯汽车科技发展有限公司 一种供电拖车控制方法、系统、存储介质及智能终端
WO2025173111A1 (fr) * 2024-02-14 2025-08-21 三菱電機株式会社 Dispositif d'actionnement à distance et système d'actionnement à distance
JP2025154784A (ja) * 2024-03-29 2025-10-10 株式会社アイシン 走行制御装置

Family Cites Families (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7777615B2 (en) 2008-03-20 2010-08-17 Toyota Motor Engineering & Manufacturing North America, Inc. System for assisting the attachment of a trailer to a vehicle
US9926008B2 (en) * 2011-04-19 2018-03-27 Ford Global Technologies, Llc Trailer backup assist system with waypoint selection
DE102012001380A1 (de) 2012-01-24 2012-08-02 Daimler Ag Assistenzverfahren und Fahrerassistenzsystem für ein Ankuppelmanöver eines Kraftfahrzeugs
US10150505B2 (en) * 2015-04-14 2018-12-11 Continental Automotive Systems, Inc. Automated hitching assist system
US9731568B2 (en) * 2015-12-01 2017-08-15 GM Global Technology Operations LLC Guided tow hitch control system and method
EP3400556A1 (fr) * 2016-01-05 2018-11-14 Mobileye Vision Technologies Ltd. Systèmes et procédés d'estimation de futures trajectoires
JP6795379B2 (ja) * 2016-03-10 2020-12-02 パナソニック インテレクチュアル プロパティ コーポレーション オブ アメリカＰａｎａｓｏｎｉｃ Ｉｎｔｅｌｌｅｃｔｕａｌ Ｐｒｏｐｅｒｔｙ Ｃｏｒｐｏｒａｔｉｏｎ ｏｆ Ａｍｅｒｉｃａ 運転制御装置、運転制御方法及び運転制御プログラム
US10906583B2 (en) 2017-03-03 2021-02-02 Continental Automotive Systems, Inc. Autonomous trailer hitching using neural network

Also Published As

Publication number Publication date
CN111372795A (zh) 2020-07-03
JP2020535077A (ja) 2020-12-03
EP3687839A1 (fr) 2020-08-05
WO2019060893A1 (fr) 2019-03-28
US20190092109A1 (en) 2019-03-28
US10987984B2 (en) 2021-04-27
JP6938793B2 (ja) 2021-09-22
CN111372795B (zh) 2023-07-04

Similar Documents

Publication Publication Date Title
EP3687839B1 (fr) Attelage de remorque automatique utilisant des coordonnées d'image
EP3589506B1 (fr) Attelage autonome de remorque à l'aide d'un réseau neuronal
EP3853047B1 (fr) Attelage autonome d'équipement agricole à un tracteur
EP3787913B1 (fr) Détection de remorque et attelage autonome
US10884425B2 (en) Autonomous trailer hitching using neural network
US20190086927A1 (en) Automated Trailer Hitching Using GPS
EP3829960B1 (fr) Inversion automatisée par choix d'emplacement cible
EP3081405B1 (fr) Système d'assistance d'attelage automatisé
US10899384B2 (en) Trailer reverse assist with follow-me system
US11155133B2 (en) System and methods for vehicle alignment control
WO2021141833A1 (fr) Système et procédé de génération de carte et de stationnement de véhicule
EP3820723B1 (fr) Système et procédé d'étalonnage d'algorithme d'estimation de mouvement à l'aide d'une caméra de véhicule
WO2022246477A1 (fr) Système et procédé d'aide à l'attelage de remorque
WO2023070011A1 (fr) Système et procédé de réglage de paramètres d'aide à la marche arrière d'une remorque sur la base d'une position estimée de remorque
WO2019005976A1 (fr) Amarrage autonome d'un véhicule à un quai de chargement
US20250232674A1 (en) Trailer Assist System and Method for a Tow Vehicle

Legal Events

Date Code Title Description
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: UNKNOWN

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE

PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20200428

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

AX Request for extension of the european patent

Extension state: BA ME

RAP1 Party data changed (applicant data changed or rights of an application transferred)

Owner name: CONTINENTAL AUTOMOTIVE SYSTEMS, INC.

DAV Request for validation of the european patent (deleted)
DAX Request for extension of the european patent (deleted)
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

17Q First examination report despatched

Effective date: 20210518

RAP1 Party data changed (applicant data changed or rights of an application transferred)

Owner name: CONTINENTAL AUTONOMOUS MOBILITY US, LLC

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

INTG Intention to grant announced

Effective date: 20230327

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE PATENT HAS BEEN GRANTED

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602018056006

Country of ref document: DE

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG9D

REG Reference to a national code

Ref country code: NL

Ref legal event code: MP

Effective date: 20230823

REG Reference to a national code

Ref country code: AT

Ref legal event code: MK05

Ref document number: 1602174

Country of ref document: AT

Kind code of ref document: T

Effective date: 20230823

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20231124

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20231223

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230823

Ref country code: RS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230823

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20231226

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20231123

Ref country code: NL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230823

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230823

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230823

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20231223

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230823

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20231124

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230823

Ref country code: AT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230823

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230823

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230823

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SM

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230823

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230823

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230823

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230823

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230823

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230823

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230823

REG Reference to a national code

Ref country code: CH

Ref legal event code: PL

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20230925

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 602018056006

Country of ref document: DE

REG Reference to a national code

Ref country code: BE

Ref legal event code: MM

Effective date: 20230930

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20230925

Ref country code: IT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230823

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230823

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

REG Reference to a national code

Ref country code: IE

Ref legal event code: MM4A

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20230925

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CH

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20230930

GBPC Gb: european patent ceased through non-payment of renewal fee

Effective date: 20231123

26N No opposition filed

Effective date: 20240524

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20230925

Ref country code: CH

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20230930

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230823

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20230930

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: GB

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20231123

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: GB

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20231123

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230823

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230823

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CY

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO

Effective date: 20180925

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: HU

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO

Effective date: 20180925

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: DE

Payment date: 20250930

Year of fee payment: 8

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: FR

Payment date: 20250922

Year of fee payment: 8

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: TR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230823