JP7593935B2 - Attribute-based pedestrian prediction - Google Patents

Description

This disclosure relates to attribute-based pedestrian prediction.

This patent application claims priority to U.S. utility patent application Ser. No. 16/363541, filed March 25, 2019, and U.S. utility patent application Ser. No. 16/363627, both of which are incorporated herein by reference in their entireties.

Prediction techniques can be used to determine the future state of an entity within an environment. That is, predictive techniques can be used to determine how a particular entity is likely to behave in the future. Current predictive techniques often require physics-based modeling or rule-of-the-road simulations to predict the future state of an entity within an environment.

The detailed description will be set forth with reference to the accompanying drawings. In the drawings, the leftmost digit of a reference number identifies the drawing in which the reference number first appears. Use of the same reference number in different drawings indicates similar or identical components or functions.

FIG. 1 is a pictorial flow diagram of an example process for acquiring sensor data, determining attributes associated with an object, determining a predicted position based on the attributes, and controlling control of a vehicle based on the predicted position. FIG. 11 is a diagram showing examples of object attributes. 1 is an exemplary diagram of determining a destination associated with an object in an environment. FIG. 1 illustrates another example of determining a destination associated with an object in an environment. FIG. 13 is a diagram illustrating determining a predicted location of an object based on attributes of the object over time. FIG. 13 illustrates updating a reference frame used in determining a predicted position. FIG. 13 is a pictorial flow diagram illustrating a process of acquiring sensor data, determining that a first object and a second object are within an environment, determining attributes associated with the second object, determining a predicted position based on the attributes and the reference line, and controlling the vehicle based on the predicted position. FIG. 11 is a diagram showing examples of object attributes. FIG. 1 illustrates an example of determining a predicted location of a first object based on attributes of a second object over time. FIG. 1 is a block diagram illustrating an example system for implementing the techniques described herein. FIG. 1 illustrates a process of acquiring sensor data, determining attributes associated with an object, determining a predicted position based on the attributes, and controlling a vehicle based on the predicted position. FIG. 1 illustrates a process of acquiring sensor data, determining that a first object and a second object are within an environment, determining attributes associated with the second object, determining a predicted position based on the attributes and a reference line, and controlling a vehicle based on the predicted position.

The present disclosure is directed to techniques for predicting the location of an object based on attributes of the object and/or attributes of other objects proximate the object. In a first example, the techniques discussed herein can be implemented to predict the location of a pedestrian proximate a pedestrian crossing area in an environment as the pedestrian crosses or prepares to cross the pedestrian crossing area. In a second example, the techniques discussed herein can be implemented to predict the location of an object (e.g., a vehicle) as the vehicle crosses an environment. For example, the predicted location of the vehicle can be based on attributes of the vehicle and attributes of other vehicles proximate the vehicle in the environment. The attributes can include information about the object, such as, but not limited to, location, speed, acceleration, bounding box, etc. The attributes can be determined over time (e.g., times T _-M , ..., T _-2 , T _-1 , T ₀ ) for the object such that, when input to a prediction component (e.g., a machine learning model such as a neural network), the prediction component can output a prediction (e.g., a predicted position of the object) at future times (e.g., times T ₁ , T ₂ , T ₃ , ..., T _N ). A vehicle, such as an autonomous vehicle, can be controlled to traverse an environment based at least in part on the predicted position of the object.

As mentioned above, in a first example, the techniques discussed herein can be implemented to predict the location of a pedestrian proximate a crosswalk area in an environment when the pedestrian crosses the crosswalk area or prepares to cross the crosswalk area. For example, sensor data can be acquired in the environment and objects can be identified and classified as pedestrians. Furthermore, a crosswalk area can be identified in the environment based on map data and/or based on sensor data (e.g., identifying a crosswalk area directly from the sensor data by observing visual indications of a crosswalk area (stripes, crosswalk signs, etc.) or indirectly by detecting a history of pedestrians crossing the road at such locations). At least one destination can be associated with the crosswalk area. For example, if the pedestrian is on a sidewalk proximate to the crosswalk, the destination can represent the opposite side of the road to the crosswalk area. If the pedestrian is on the road (either inside or outside the crosswalk area), the destination can be selected or otherwise determined based on the pedestrian's attributes (e.g., position, speed, acceleration, path, etc.). For multiple crosswalk areas in close proximity to one another, a score associated with the likelihood that a pedestrian will cross a particular crosswalk can be determined based on the pedestrian's attributes (e.g., location, speed, acceleration, path, etc.). The crosswalk area associated with the highest score can be selected or otherwise determined to be the target crosswalk associated with the pedestrian.

In some examples, such as when running a red light or crossing a road where a crosswalk area is not easily identifiable, a destination associated with a pedestrian can be determined based on many factors. For example, the destination can be determined based at least in part on one or more of a linear extrapolation of the pedestrian's speed, the closest location of a sidewalk area associated with the pedestrian, a gap between parked vehicles, an open door associated with a vehicle, and the like. In some examples, sensor data can be obtained about the environment to determine the likelihood that these exemplary candidate destinations are present in the environment. In some examples, a score can be associated with each candidate destination, and the likely destinations can be used in accordance with the techniques discussed herein.

If the crosswalk area (or other location) is determined to be the pedestrian's destination, the technique can include predicting the location of the pedestrian crossing the crosswalk area over time. In some examples, the attributes of the object can be determined over time (e.g., times T _-M , . . ., T _-2 , T _-1 , T ₀ ), whereby the attributes can be expressed in a frame of reference associated with the object at time T ₀ . That is, the position of the object at T ₀ can be considered as the origin (e.g., coordinate (0,0) in an x-y coordinate system), whereby a first axis can be defined by the origin and the destination associated with the crosswalk area. In some examples, other points can be considered as the origin, relative to another frame of reference. As mentioned above, if the pedestrian is on a first side of the road, the destination associated with the crosswalk area can be selected as a point on a second side of the road opposite the first side of the road, although any destination can also be selected. A second axis of the frame of reference can be perpendicular to the first axis and, in at least some instances, lies along a plane that includes the crosswalk area.

In some examples, pedestrian attributes can be determined based on sensor data captured over time, including the pedestrian's position at a time (e.g., the position can be expressed in a frame of reference as described above), the pedestrian's speed at that time (e.g., magnitude and/or angle relative to a first axis (or other reference line)), the pedestrian's acceleration at that time, an indication of whether the pedestrian is in a drivable area (e.g., whether the pedestrian is on the sidewalk or on the road), an indication of whether the pedestrian is in a crosswalk area, the state of indicators controlling the area (e.g., whether the intersection is signal-controlled and/or whether the crosswalk is signal-controlled (e.g., walk/no walk) and/or the state of the signal), vehicle context (e.g., the presence of vehicles in the environment and attributes associated with the vehicles), flux through the crosswalk area over a period of time (e.g., the number of objects (e.g., vehicles) that have passed through the crosswalk area over a period of time), object associations (e.g., whether the pedestrian is in a crosswalk area with multiple pedestrians), and/or attributes associated with the pedestrian. (e.g., whether the pedestrian is moving in a group of pedestrians), distance to the crosswalk in a first direction (e.g., distance in a global x-direction or an x-direction based on a frame of reference), distance to the crosswalk in a second direction (e.g., distance in a global y-direction or a y-direction based on a frame of reference), distance to the road in the crosswalk area (e.g., minimum distance to the road in the crosswalk area), pedestrian hand gestures, pedestrian gaze detection, indication of whether the pedestrian is standing, walking, running, etc., whether other pedestrians are in the crosswalk, pedestrian crosswalk flux (e.g., the number of pedestrians crossing the crosswalk (e.g., drivable area) and passing through the crosswalk in a certain period of time), the ratio of the number of first pedestrians on the sidewalk (or non-drivable area) to the number of second pedestrians on the crosswalk area (or drivable area), the variance, confidence, and/or probability associated with each attribute, etc.

The attributes can be determined over time (e.g., but not limited to, times T _-M , ..., T -2 , T _-1 , T ₀ (where M is an integer) representing any time before the current time and/or including the current time, such as, but not limited to, _{0.01 seconds} , 0.1 seconds, 1 second, 2 seconds, etc.) and input to a prediction component to determine a predicted location of the pedestrian. In some examples, the prediction component is a machine learning model such as a neural network, a fully connected neural network, a convolutional neural network, a recurrent neural network, etc.

In some examples, the prediction component can output information associated with a future pedestrian. For example, the prediction component can output prediction information associated with a future time (e.g., times _T1 , _T2 , _T3 , ..., _TN (where N is an integer) representing any time after the current time). In some examples, the prediction information can comprise a predicted position of the pedestrian at the future time. For example, the predicted position can be represented in a frame of reference as a distance (e.g., distance s) between an origin (e.g., the position of the pedestrian at _T0 ) and the pedestrian at _T1 , and/or a lateral offset (e _y ) relative to a first axis (e.g., relative to a reference line). In some examples, the distance s and/or the lateral offset e _y can be represented as rational numbers (e.g., 0.1 meters, 1 meter, 1.5 meters, etc.). In some examples, the distance s and/or the lateral offset can be binned (e.g., input to a binning algorithm) to discretize the original data values into one or many discrete intervals. In some examples, the bins of distance s may be 0 to 1 meter, 1 to 2 meters, 3 to 4 meters, etc., although any regular or irregular interval for such bins may be used.

In some examples, a vehicle, such as an autonomous vehicle, can be controlled to traverse an environment based at least in part on the predicted positions of pedestrians.

As mentioned above, in a second example, the techniques discussed herein may be implemented to predict the location of an object (e.g., a vehicle) as the vehicle passes through an environment. For example, sensor data may be acquired within the environment, and an object may be identified and classified as a vehicle. Further, a reference line may be identified and associated with the vehicle based on map data (e.g., identifying drivable areas, such as lanes) and/or sensor data (e.g., identifying drivable areas, or lanes, from the sensor data). As can be appreciated, the environment may include any number of objects. For example, an object of interest, or a vehicle of interest (e.g., a vehicle that is the subject of such prediction techniques) may be moving through an environment in which there are other vehicles in close proximity to the vehicle of interest. In some examples, the techniques may include identifying K objects closest to the object of interest (where K is an integer). For example, the techniques may include identifying the five vehicles or other objects closest to the vehicle of interest, although any number of vehicles or other objects may be identified or otherwise determined. In some examples, the techniques may include identifying objects within a threshold distance to the object of interest. In some examples, the vehicle capturing the sensor data is identified as one of the objects proximate to the vehicle of interest. In at least some examples, additional features are used to determine which objects to consider. As non-limiting examples, objects traveling in the opposite direction, objects on the other side of a divided road, objects with certain classifications (e.g., non-vehicle), etc. are ignored when considering the K closest objects.

In some examples, attributes may be determined for the object of interest and/or other objects proximate to the object of interest. For example, the attributes may include, but are not limited to, one or more of the following: the speed of the object at a given time; the acceleration of the object at a given time; the position of the object at a given time (e.g., in global or local coordinates); the bounding box of the object at a given time (e.g., representing the range, roll, pitch, and/or yaw of the object); the state of the lights associated with the object at an initial time (headlights, brake lights, hazard lights, turn signals, backlights, etc.); the orientation of the vehicle's wheels; the distance between the object and a map element at a given time (distance to a stop line, speed bump, yield line, intersection, roadway, etc.); the classification of the object (car, vehicle, animal, truck, bicycle, etc.); features associated with the object (whether the object is changing lanes, whether it is a double-parked vehicle, etc.); lane type (lane direction, parking lane, etc.); road signs (e.g., indicating whether passing or lane changing is permitted, etc.);

In some examples, attribute information associated with an object of interest and/or other objects proximate the object of interest can be captured over time and input to a prediction component to determine predictive information associated with the object of interest. In some examples, the predictive information can represent predicted locations of the object at various time intervals (e.g., predicted locations at times _T1 , _T2 , _T3 , ..., _TN ).

In some examples, the predicted location can be compared to candidate reference lines in the environment to determine a reference line associated with the object of interest. For example, the environment includes two lanes that are eligible (e.g., legal) drivable areas for the vehicle of interest to cross. Furthermore, such drivable areas are associated with representative reference lines (e.g., the center of the lanes or drivable areas). In some examples, the predicted location can be compared to the reference lines to determine a similarity score between the predicted location and the reference line candidates. In some examples, the similarity score can be based at least in part on the distance between the predicted location and the reference lines, etc. In some examples, attributes associated with the object (e.g., times T _-M , T _-1 , T ₀ ) can be input to a reference line prediction component that can output a reference line that is likely to be associated with the object. The technique can include receiving, selecting, or otherwise determining a reference line and representing a predicted location with respect to the reference line in the environment. That is, the predicted position may be represented as a distance s along a reference line that represents the distance between the object's position at time _T0 and the object's predicted position at a future time (e.g., time _T1 ). The lateral offset e _y may represent the distance between the reference line and a point that intersects a line perpendicular to a tangent associated with the reference line.

The prediction techniques can be repeated iteratively or in parallel to determine predicted positions associated with objects in the environment. That is, a first object of interest is associated with a first subset of objects in the environment and a second object of interest is associated with a second subset of objects in the environment. In some examples, the first object of interest is included in the second subset of objects, while the second object of interest is included in the first subset of objects. In this manner, predicted positions can be determined for multiple objects in the environment. In some cases, the predicted positions can be determined substantially simultaneously within technical tolerances.

In some examples, a vehicle, such as an autonomous vehicle, can be controlled to traverse an environment based at least in part on a predicted position of an object. For example, such predicted positions can be input to a planning component of the vehicle to understand the predicted positions of objects in the environment and traverse the environment.

The techniques discussed herein can improve the capabilities of a computing device, such as a computing device of an autonomous vehicle, in many additional ways. In some examples, determining attributes and inputting the attributes into a predictive component, such as a machine learning component, can avoid hard-coded rules that would otherwise represent the environment inflexibly. In some cases, determining a predicted position associated with an object (such as a pedestrian or vehicle) in the environment can provide other vehicles or objects with the ability to better plan a trajectory for safe and comfortable travel through the environment. For example, a predicted position that indicates a potential collision or near collision can provide the autonomous vehicle with the ability to modify its trajectory (e.g., change lanes, stop, etc.) to safely traverse the environment. Such and other improvements in the capabilities of computing devices are discussed herein.

The techniques described herein can be implemented in many ways. Exemplary implementations are provided with reference to the following figures. Although discussed in the context of an autonomous vehicle, the methods, apparatus, and systems described herein can be applied to a variety of systems (e.g., sensor systems or robotic platforms) and are not limited to autonomous vehicles. In one example, similar techniques are utilized in driver-controlled vehicles where such systems provide indications of whether it is safe to perform various maneuvers. In another example, the techniques can be utilized in the context of a manufacturing assembly line or in the context of aerial surveying. Additionally, the techniques described herein can be used with real data (e.g., captured through the use of sensors), simulated data (e.g., generated by a simulator), or any combination of the two.

FIG. 1 is a pictorial flow diagram of an example process 100 for capturing sensor data, determining attributes associated with an object, determining a predicted position based on the attributes, and controlling a vehicle based on the predicted position.

In operation 102, the process may include capturing sensor data of the environment. In some examples, the sensor data may be captured by one or more sensors on the vehicle (autonomous or otherwise). For example, the sensor data may include data captured by a LIDAR sensor, an image sensor, a RADAR sensor, a time of flight sensor, a sonar sensor, etc. In some examples, operation 102 may include determining a classification of the object (e.g., determining that the object is a pedestrian in the environment).

At operation 104, the process may include determining a destination associated with the object (e.g., a pedestrian). Example 106 shows a vehicle 108 and an object 110 (e.g., a pedestrian) in an environment. In some examples, the vehicle 108 may perform the operations discussed in process 100.

The operation 104 may include determining attributes of the object 110 to determine the location, speed, path, etc. of the object 110. Additionally, the operation 104 may include accessing map data to determine whether a crosswalk area (e.g., the crosswalk area 112) exists in the environment. In some examples, the crosswalk area 112 may represent the perimeter of a crosswalk in the environment. In some examples, the operation 104 may include determining that the object is within a threshold distance (e.g., 5 meters) of a portion of the crosswalk area 112. In some examples, the threshold distance is considered to be the shortest distance from the object to any portion of the crosswalk area. If the object 110 is within the threshold distance of various crosswalk areas in the environment, the operation 104 may include determining a probability, or score, associated with a pedestrian (e.g., the object 110) crossing each crosswalk area and selecting the most likely crosswalk area. In some examples, the destination 114 may be associated with the crosswalk area 112. In some examples, the destination 114 may represent any point in the environment associated with the crosswalk area 112, but may represent the center or midpoint of the side of the crosswalk area 112 opposite the location of the object 110. Additional details of determining the destination are discussed in connection with Figures 3A and 3B, as well as throughout this disclosure.

At operation 116, the process may include determining attributes associated with the object. As illustrated in example 118, attributes may be determined for object 110 at various instances in time (e.g., times T _-M , . . ., T _-2 , T _-1 , T ₀ ) up to and including the most recent time associated with the attribute. Object 110 may be referenced as object 120 (e.g., at time T _-2 ), as object 122 (e.g., at time T _-1 ), and as object 124 (e.g., at time T ₀ ). In some examples, time T ₀ represents a time when data is input into a prediction component (described below), time T _-1 represents one second before time T ₀ , and time T _-2 represents two seconds before time T ₀ . However, it may be understood that times T ₀ , T _-1 , and T _-2 may represent any time instance and/or period of time. For example, time T _-1 represents 0.1 seconds before time T ₀ and time T _-2 represents 0.2 seconds before time T _0. In some examples, the attributes determined in operation 116 may include, but are not limited to, information about objects 120, 122, and/or 124. For example, a speed attribute associated with object 120 represents the speed of object 120 at time T _-2 . A speed attribute associated with object 122 represents the speed of the object at time T _-1 . And a speed attribute associated with object 124 represents the speed of the object at time T _0. In some examples, some or all of the attributes are expressed in a frame of reference relative to object 124 (e.g., object 110 at time T ₀ ) and destination 114. In such examples, there are three unique frames of reference associated with each preceding time step (T _-M through T ₀ ), and each attribute is associated with its particular time frame of reference. Additional details of the attributes are discussed throughout this disclosure, as well as in connection with FIG. 2 .

In act 126, the process may include determining a predicted position associated with the object based on the attributes. Example 128 illustrates predicted position 130 (e.g., a predicted position of object 110 at time _T1, which is a time after _T0 ). In some examples, act 126 may be performed at or near time _T0 , so that predicted position 130 at time _T1 may represent a location of object 110 in the future. As can be appreciated, in some examples, act 126 may include determining predicted positions for multiple times associated with future object 124. For example, act 126 may include determining predicted positions of the object at times _T1 , _T2 , _T3 , ..., _TN , where N is an integer representing a time in the future, e.g., 1 second, 2 seconds, 3 seconds, etc. In some examples, the predicted position may be represented as a distance s along a reference line and a lateral offset _ey from the reference line. In at least some examples, the distance s and offset e _y are relative to a relative coordinate system defined at each time step and/or relative to the last determined reference frame. Additional details regarding determining predicted positions are discussed in connection with Figures 4 and 5 and throughout this disclosure.

In some examples, operations 102, 104, 116, and/or 126 may be performed iteratively or repeatedly (e.g., at each time step, at a frequency of 10 Hz, etc.), although process 100 may be performed at any interval or for any length of time.

In operation 132, the process may include controlling the vehicle based at least in part on the predicted position. In some examples, operation 132 may include generating a trajectory for the vehicle 108 to follow (e.g., stopping before an intersection and/or before a pedestrian crossing area 112 and allowing the pedestrian 110 to cross through the pedestrian crossing area 112 to the destination 114).

2 illustrates example attributes 200 of an object. In some examples, the attributes 202 may represent various information about or associated with an object in an environment (e.g., object 110 in FIG. 1 ). In some examples, the attributes 202 may be determined for one or more time instances associated with the object. For example, object 120 represents object 110 at time T ₋₂ , object 122 represents object 110 at time T ₋₁ , and object 124 represents object 110 at time T ₀ . Attributes may be determined for the object at each of the time instances T ₋₂ , T ₋₁ , and T ₀ , for example.

Examples of attributes 202 include, but are not limited to, distance between the object and the road, x-distance (or first distance) to the region, y-distance (or second distance) to the region, distance to destination, speed (magnitude), speed (angle), x-position, y-position, region flux, state of indicator controlling the region, vehicle context (or generally, object context), object association, etc. In at least some examples, the attributes discussed herein are relative to a relative coordinate system defined at each time step (e.g., associated with each of the objects 120, 122, 124), relative to a last determined reference frame, relative to a reference frame defined with respect to the vehicle 108 (e.g., at various time steps (s)), relative to a global coordinate reference frame, etc.

Example 204 illustrates various attributes associated with object 124. For example, example 204 illustrates attributes related to crosswalk area 112 and destination 114. In some examples, an x-distance to the area can correspond to distance 206. That is, distance 206 can represent a distance in a first direction (which can be a global or local reference frame) between object 124 and an edge of crosswalk area 112 that is closest to object 124. In some examples, a y-distance to the area can correspond to distance 208. That is, distance 208 can represent a distance in a second direction between object 124 and an edge of crosswalk area 112. In at least some examples, the simplest distance between object 124 and crosswalk area is determined and then decomposed into respective x- and y-components as x-distance and y-distance.

As shown in example 204, object 124 is at a location on sidewalk region 210 (or generally, non-drivable region 210). In some examples, crosswalk region 112 provides a path across road 212 (or generally, drivable region 212). In some examples, the distance to the road may correspond to distance 214, which may correspond to a minimum or minimum distance between object 124 and the portion of road 212 within crosswalk region 112.

In some examples, the distance to the destination may correspond to distance 216. As shown, distance 216 represents the distance between object 124 and destination 114.

As mentioned above, in some examples, the attribute 202 can be expressed in a frame of reference. As discussed herein, a frame of reference is defined with respect to the position of the object at each time step, relative to a last reference frame, a global coordinate system, etc. In some examples, an origin corresponding to the frame of reference can correspond to the position of the object 124. Example 218 illustrates a frame of reference 220 (also referred to as reference frame 220). In some examples, a first axis of the frame of reference 220 is defined by a unit vector from the position of the object 124 in the direction of the destination 114. The first axis is labeled as the x-axis in example 218. In some examples, a second axis can be perpendicular to the second axis and lie in a plane that includes the crosswalk. The second axis is labeled as the y-axis in example 218. In some examples, the first axis can represent a reference line from which a distance s can be determined, while a lateral offset e _y can be determined with respect to a second direction (e.g., the y-axis).

Example 222 shows a velocity vector 224 associated with object 124 and an angle 226 representing the angle between velocity vector 224 and a reference line. In some examples, the reference line may correspond to a first axis of frame of reference 220, although any reference line may be selected or otherwise determined.

As discussed herein, attributes associated with objects 124, 122, and 120 may be expressed with respect to a frame of reference 220. That is, at time T0, the x- and y-positions of object 124 may be expressed as (0,0) (e.g., object 124 represents the origin of frame of reference 220). Additionally, with respect to frame of reference 220, the x- and y-positions of object 122 (at time _T0 ) may be expressed as ( _-x1 , _-y1 ), and the x- and y-positions of object 120 (at time _T0 ) may be expressed as ( _-x2 , _-y2 ). While in at least some examples, a single coordinate frame is used, in other examples, a relative coordinate frame is associated with all points and attributes are defined for each relative coordinate frame.

As mentioned above, the attributes 202 can include area flux. In some examples, the area flux can represent the number of objects passing through the crosswalk area 112 within a period of time. For example, the area flux can correspond to J cars (and/or other objects, such as other pedestrians) passing through the crosswalk area 112 (or any area) within K seconds (e.g., 5 cars in the time between T _-2 and T ₀ ). In some examples, the area flux can represent any period of time. Additionally, the area flux can include information such as speed, acceleration, velocity, etc., for such vehicles passing through the crosswalk area 112 within the period of time.

Further, attributes 202 can include indicators controlling the area. In some examples, the indicators controlling the area can correspond to the state of a signal or indicator controlling pedestrian traffic within crosswalk area 112. In some examples, the indicators controlling the area can indicate whether a signal is present, the state of the signal (e.g., green, yellow, red, etc.), and/or the state of a crosswalk indicator (e.g., walk, no walk, unknown, etc.).

In some examples, attributes 202 may include vehicle context indicating whether a vehicle or other object is in proximity to object (e.g., 124) and attributes associated with any such vehicle or object. In some examples, vehicle context may include, but is not limited to, speed, direction, acceleration, bounding box, position (e.g., within frame of reference 220), distance between the object and object 124, etc.

In some examples, the attributes 202 can include object associations. For example, the object associations can indicate whether the object 124 is associated with other objects (e.g., whether the object 124 is in a group of pedestrians). In some examples, the object association attributes 202 can include associated attributes for the associated objects.

Attributes 202 further include, but are not limited to, information associated with acceleration, yaw, pitch, roll, relative velocity, relative acceleration, whether the object is on the road 212, whether the object is on the sidewalk 210, whether the object is in a crosswalk area 112, whether the destination has changed (e.g., whether the object has turned back at an intersection), the height of the object, whether the object is riding a bicycle, etc.

Attributes 202 may further include, but are not limited to, pedestrian hand gestures, pedestrian gaze detection, an indication of whether a pedestrian is standing, walking, running, etc., whether other pedestrians are in a crosswalk, pedestrian crosswalk flux (e.g., the number of pedestrians moving through a crosswalk (e.g., across a drivable area) in a period of time), the ratio of a first number of pedestrians on the sidewalk (or non-drivable area) to a second number of pedestrians in the crosswalk area (or drivable area), a variance, confidence, and/or probability associated with each attribute, etc.

Figures 3A and 3B show examples of determining destinations associated with objects in an environment. Generally, Figure 3A illustrates choosing between two crosswalk areas, while Figure 3B illustrates choosing between two destinations associated with a single crosswalk area.

FIGURE 3A illustrates an example 300 of determining a destination associated with an object in an environment. As discussed above, and generally, FIGURE 3A illustrates selecting between two pedestrian crossing areas. Example 302 illustrates an object 304 corresponding to a pedestrian at time T _-1 and an object 306 corresponding to a pedestrian at time _T0 . For example, a vehicle, such as vehicle 108, can capture sensor data of the environment and can determine that a pedestrian is in the environment.

Further, based at least in part on the objects 304 and 306, the computing system can determine that the objects 304 and/or 306 are proximate to one or more pedestrian crossing areas in the environment. For example, the computing device can access map data that includes map elements that indicate the location and extent (e.g., length and width) of such pedestrian crossing areas. Example 302 illustrates the environment as including a first pedestrian crossing area 308 (also referred to as area 308) and a second pedestrian crossing area 310 (also referred to as area 310).

In some examples, region 308 can be associated with threshold region 312 (also referred to as threshold 312) and region 310 can be associated with threshold region 314 (also referred to as threshold 314). As shown, objects 304 and 306 are within thresholds 312 and 314. Based at least in part on objects 304 and/or 306 being within thresholds 312 and 314, a computing device can determine that objects 304 and/or 306 are associated with regions 308 and 310, respectively.

In some examples, threshold 312 may represent any region or area associated with region 308. As illustrated, threshold 312 may represent a 5 meter threshold surrounding region 308, although any distance or shape of threshold 312 may be associated with region 308. Similarly, threshold 314 may include any distance or shape associated with region 310.

In some examples, the region 308 can be associated with a destination 316. Further, in some examples, the region 310 can be associated with a destination 318. In some examples, the location of the destination 316 and/or 318 is located across a road from the object 304 and/or 306. That is, the destination associated with the crosswalk region can be selected based at least in part on the location of the pedestrian relative to the crosswalk region.

Objects 304 and/or 306 can be associated with attributes as discussed herein. That is, techniques can include determining the position, velocity, path, acceleration, etc. of objects 304 and 306, respectively.

Additionally, the information depicted in example 302 (e.g., attributes associated with objects 304 and/or 306, locations of regions 308 and/or 310, locations of thresholds 312 and/or 314, locations of destinations 316 and/or 318, etc.) can be input to a destination prediction component 320. In some examples, the destination prediction component 320 can output a score, or probability, that object 306 will cross region 308 and/or region 310. Although example 302 shows object information associated with two time steps (e.g., T ₋₁ and T ₀ ), any object information over time can be used to determine the destination.

In some examples, attributes associated with objects 302 and 306 can be input to destination prediction component 320 using one or more frames of reference. For example, to evaluate destination 316, attributes associated with objects 304 and 306 can be input to destination prediction component 320 using a frame of reference based at least in part on destination 316. Further, to evaluate destination 318, attributes associated with objects 304 and 306 can be input to destination prediction component 320 using a frame of reference based at least in part on destination 318.

In some examples, such as in the case of a pedestrian running a red light or crossing a road where a crosswalk area is not easily identifiable, a destination associated with the pedestrian can be determined based on a number of factors. For example, the destination can be determined based at least in part on one or more of a linear extrapolation of the pedestrian's speed, the closest location of a sidewalk area associated with the pedestrian, a gap between parked vehicles, an open door associated with a vehicle, and the like. In some examples, sensor data of the environment can be captured to identify possible destinations within the environment. Additionally, attributes associated with the object can be represented in a frame of reference based at least in part on the determined destination, and the attributes can be input to the destination prediction component 320 for evaluation, as discussed herein.

Example 322 illustrates output of destination prediction component 320. For example, based at least in part on attributes of object 304 and/or 306, destination prediction component 320 predicts that object 304 and/or 306 are heading toward destination 318.

Figure 3B illustrates another example 324 of determining destinations associated with objects in the environment. As discussed above, Figure 3B illustrates selecting between two destinations associated with a single crosswalk area.

Example 324 shows object 326 corresponding to a pedestrian at time T ₋₁ and object 328 corresponding to a pedestrian at time T _0. In some examples, because objects 326 and 328 are on road 330 (or drivable area 330) (as opposed to being located on sidewalk 332 (or non-drivable area 332)), the computing device identifies two destinations 334 and 336 associated with area 338. In some examples, attributes associated with objects 326 and 328 can be input to destination prediction component 320 to determine which of destinations 334 and 336 is most likely (as well as information about destinations 334 and 336 and area 338, as well as other information). While depicted as entering and leaving a crosswalk in this FIG. 3B for illustrative purposes, such a crosswalk area is not required. As a non-limiting example, such a destination prediction component 320 would typically determine that the pedestrian is about to run a red light or otherwise cross an area that is not a crosswalk area, and output the corresponding destination. In such instances, attributes associated with the region are not determined (because the region does not exist), however, in some instances, a fixed region perpendicular to the road segment and having a certain width is used as the region for determining such parameters.

As discussed above, in some examples, region 338 is associated with object 326 and/or 328 at a time when object 326 and/or 328 is within a threshold distance of region 338.

Figure 4 shows an example 400 of determining a predicted location of an object based on attributes of the object over time.

Example 402 illustrates object 120 (e.g., a pedestrian at time T ₋₂ ), object 122 (e.g., a pedestrian at time T ₋₁ ), and object 124 (e.g., a pedestrian at time T ₀ ). As discussed herein, objects 120, 122, and 124 may be represented in a frame of reference that has object 124 as the origin (and/or one or more frames of reference associated with any one or more times). Additionally, example 402 illustrates objects 120, 122, and 124 associated with pedestrian crossing area 112 and destination 114.

Data associated with example 402 can be input to a location prediction component 404, which can output a predicted location associated with objects 120, 122, and/or 124.

Example 406 illustrates a predicted position based on objects 120, 122, and/or 124. For example, location prediction component 404 can output a predicted position 408, which represents the position of the object at time _T. In some examples, predicted position 408 is expressed as a distance (e.g., s) 410 and a lateral offset 412 (e.g., e _y ) based at least in part on a frame of reference defined by object 124 (e.g., origin) and destination 114.

As shown, the location prediction component 404 may output five predicted positions corresponding to each of times _T1 , _T2 , _T3 , _T4 , and _T5 , respectively, although it may be understood that the location prediction component 404 may output any number of predicted positions associated with any future time. In some examples, such additional predicted positions may be defined with respect to a global coordinate frame, a local coordinate frame, a relative reference frame associated with a previous predicted point, etc.

In some examples, the location prediction component 404 can include functionality for binning output values, such as distance s or lateral offset e _y . That is, the location prediction component 404 can include a binning functionality that replaces values that fall into a bin with a value representative of that bin. For example, the distance s that falls into a bin can be replaced with a value representative of the binned value. For example, if distance s=0.9 meters and a first bin from 0.0 meters to 1.0 meters corresponds to a bin value of 0.5 meters, then the binned output for distance s=0.9 meters corresponds to 0.5 meters. Any number of bins can be used across any range. Of course, in some examples, the original value can be output without binning. In such examples, an additional value indicating an offset from the center of the bin is associated with the output bin. As a non-limiting example, an output indicating that the next predicted location falls into the first bin (e.g., between 0 and 1 m) and an associated offset of 0.2 m can be used to indicate that the predicted location is likely to be 0.7 m (e.g., 0.5 m + 0.2 m).

In general, the predicted location shown in example 406 can be referred to as predicted location 414.

In some examples, the location prediction component 404 can output a variance, covariance, probability, or certainty associated with each predicted location 414 that indicates the certainty that the object 124 will be located at each predicted location at each time.

Figure 5 shows an example 500 of updating the frame of reference used in determining a predicted position.

5 to represent a time _T corresponding to time _T represented by example 406. As shown, objects 120, 122, and 124 are represented in a frame of reference 220 that is defined in part by the position of object 124 and the position of destination 114.

In some examples, the example 406 is updated for the next time step and an updated predicted position can be determined (e.g., in operation 502).

Such an updated example is shown as example 504, which shows an environment that corresponds to example 406, but at a time T that occurs after time _T. An object 506 in example 504 represents time _T with respect to a frame of reference 508. Similarly, example 504 includes an object ₅₁₀ that represents an object at time _T. Additionally, object 512 represents an object at time _T.

In some examples, object 510 (e.g., object at time T ₋₁ in frame of reference 508) may correspond to object 124 (e.g., object at time T ₀ in frame of reference 220). Similarly, object 512 (e.g., object at time T ₋₂ in frame of reference 508) may correspond to object 122 (e.g., object at time T ₋₁ in frame of reference 220). For comparison, example 504 illustrates object 120, whereby object 120 (and/or attributes associated with object 120) are used or object 120 is not used when determining the updated predicted position in example 504.

As can be appreciated, the frame of reference 508 can be defined by, or at least in part based on, the positions of the object 506 and the destination 114. In this manner, a relative frame of reference can be defined with respect to the most recently determined positions of the destination 114 and the object 124 (e.g., such a coordinate reference frame can change with changes in the objects in the environment).

Thus, information associated with the example 504 (including or not including information associated with the object 120) can be input to the location prediction component 404 to determine an updated predicted location 514. As discussed herein, the updated predicted location 514 is based at least in part on the frame of reference 508.

In some examples, the updated predicted position may be determined at a frequency of 10 Hz, but the predicted position may be determined at any frequency or at any regular or irregular time interval.

FIG. 6 is a pictorial flow diagram illustrating a process 600 of capturing sensor data, determining that a first object and a second object are in an environment, determining attributes associated with the second object, determining a predicted position based on the attributes and the reference line, and controlling the vehicle based on the predicted position.

Although discussed in the context of determining attributes of a first and second object to determine a predicted location associated with a first object, in some examples, no attributes are determined for one or more second objects, and the predicted location of the first object can be determined based on the attributes associated with the first object.

At operation 602, the process may include capturing sensor data of the environment. In some examples, the sensor data may be captured by one or more sensors on the vehicle (autonomous or otherwise). For example, the sensor data may include data captured by a LIDAR sensor, an image sensor, a RADAR sensor, a time of flight sensor, a sonar sensor, etc. In some examples, operation 602 may include determining a classification of the object (e.g., determining that the object is a vehicle in the environment).

Example 604 illustrates a vehicle 606 capturing sensor data in operation 602. The environment further includes objects 608, 610, 612, 614, 616, and 618. In some examples, object 618 may be referred to as a target object 618 because it is the subject (e.g., the target) of a predictive operation as discussed herein.

In some examples, the vehicle 606 traverses the environment via a trajectory 620. As can be understood in the context of FIG. 6, the object 608 can be traveling in the same direction as the vehicle 606 (e.g., in the same lane as the vehicle 606), while in some examples, the objects 610-618 and the target object 618 can travel in the opposite direction (e.g., the target object 618 can represent an oncoming vehicle with respect to the vehicle 606). Of course, the process 600 can be used in any environment and is not limited to the particular objects and/or geometries shown in FIG. 6.

At operation 622, the process may include determining attributes associated with the target object and objects proximate to the target object. Example 624 shows vehicle 606, objects 606-616, and target object 618. In some examples, operation 622 may include determining attributes associated with the target object without determining attributes of other objects. For example, no such other objects are present in the environment, or such attributes of the other objects are not needed, desired, or required for determining a predicted location of target object 618 in accordance with implementations of the techniques discussed herein.

For purposes of illustration, the outline of object 612 is shown as a dotted line, and elements 626, 628, and 630 corresponding to object 612 are represented as dots. In some examples, element 626 represents a position associated with object 612 at time T _-2 . In some examples, element 628 represents a position associated with object 612 at time T _-1 . And, in some examples, element 630 represents a position associated with object 612 at time T ₀ .

As further shown, vehicle 606, objects 608-616, and target object 618 are associated with elements, even though such elements are not labeled in Figure 6. It can be understood in the context of this disclosure that such elements can represent positions associated with the vehicles and/or objects at the respective times (e.g., times T _-2 , T _-1 , and _T0 ) and/or represent attributes associated with the objects at the respective times.

In some examples, the attributes determined in operation 622 can represent information about each object. For example, such attributes can include, but are not limited to, the object's position (e.g., a global position and/or a position relative to any frame of reference), velocity, acceleration, bounding box, lighting conditions, lane attributes, reference lines, or offset from a predicted path, etc. Additional details of such attributes are discussed in connection with FIG. 7 as well as throughout this disclosure.

In some examples, operation 622 may include determining or identifying an object based at least in part on the proximity of the object to the target object. For example, operation 622 may include determining the N closest objects in proximity to target object 618, where N is an integer. Additionally or alternatively, operation 622 may include identifying or selecting an object based on the object being within a threshold distance of target object 618. In at least some examples, such selection excludes certain objects based on one or more characteristics such as, but not limited to, object classification (e.g., considering only vehicles), direction of motion (e.g., considering only objects moving in the same direction), or location relative to the map (e.g., considering only vehicles in one or more lanes of a road).

At operation 632, the process may include determining a predicted position associated with the target object based at least in part on the attributes, the predicted position relative to a reference line in the environment (which in some examples comprises a centerline of a lane associated with the object). Example 634 illustrates a predicted position 636 associated with target object 618 in the environment. In some examples, predicted position 636 may be defined by and/or based at least in part on reference line 638. That is, predicted position 636 may be represented by a distance s along reference line 638 and a lateral offset e _y from reference line 638.

In some examples, the reference lines 638 may be based at least in part on map data within the environment. Further, in some examples, the reference lines 638 may correspond to centerlines of lanes of a road or other drivable area.

In some examples, operation 632 may include receiving a reference line associated with the target object 618, such as from a reference line prediction component. In some examples, the reference line prediction component may comprise a machine learning model trained to output a most likely reference line based at least in part on map data, attributes of objects in the environment, etc. In some examples, the reference line prediction component may be integrated with other machine learning models discussed herein, and in some examples, the reference line prediction component may be a separate component.

In some examples, operation 632 may include selecting a reference line 638 from a plurality of candidate reference lines. In some examples, the reference line 638 may be selected based at least in part on a similarity score that represents a similarity of the predicted position 636 to the reference line 638. Some examples include the predicted position 636 relative to a predicted path and/or trajectory, a previously predicted waypoint, and/or the like. Additional examples of predicted positions, reference lines, and similarity scores are discussed throughout this disclosure, as well as in connection with FIG. 8.

In operation 640, the process may include controlling the vehicle based at least in part on the predicted position. In some examples, operation 640 may include generating a trajectory for the vehicle 608 to follow, or an updated trajectory 642 (e.g., biasing the vehicle 606 away from the predicted position 636 associated with the vehicle 618 if the object 618 crosses closely relative to the expected path of the vehicle 608).

Figure 7 shows example object attributes 700. In some examples, the attributes 702 can represent various information about or associated with objects in an environment (e.g., object 612 of Figure 6 and target object 618, represented by example 604 reproduced in Figure 7).

In some examples, attributes 702 may be determined for one or more time instances of the object. Example 704 shows object 612 at time instances _T2 , T _-1 , and _T0 . For example, element 626 represents object 612 at time T _-2 , element 628 represents object 612 at time T- ₁ , and element 630 represents object 612 at time T0.

Additionally, attributes may be determined for any type and/or number of objects in example 704 and are not limited to object 612. For example, attributes may be determined for element 706 (e.g., representing object of interest 618 at time T _-2 ), element 708 (e.g., representing object of interest 618 at time T _-1 ), and element 710 (e.g., representing object of interest 618 at time T ₀ ). Additionally, attributes may be determined for any number of time instances and are not limited to T _-2 , T _-1 , and T ₀ .

Examples of attributes 702 include, but are not limited to, an object's speed, an object's acceleration, an object's x-position (e.g., relative to a global, local, and/or other frame of reference), an object's y-position (e.g., relative to a local, global, and/or other frame of reference), a bounding box associated with the object (e.g., range (length, width, and/or height), yaw, pitch, roll, etc.), lighting status (e.g., brake lights, blinker lights, hazard lights, headlights, backlights, etc.), wheel direction of the object, map elements (e.g., distance between the object and a stop light, stop sign, speed bump, intersection, yield sign, etc.), object classification (e.g., vehicle, car, truck, bicycle, motorcycle, pedestrian, animal, etc.), object characteristics (e.g., whether a lane is being changed, whether the object is a double-parked vehicle, etc.), proximity to one or more objects (in any coordinate frame), lane type (lane direction, parking lane, etc.), road signs (e.g., indicating whether passing or lane changes are permitted, etc.), etc.

In some examples, object attributes may be determined with respect to a local frame of reference, global coordinates, etc. For example, a frame of reference may be determined to have an origin that corresponds to the position of target object 618 (e.g., object 710) at time _T0 .

Figure 8 shows an example 800 of determining a predicted location of a first object based on attributes of a second object over time.

As shown, information associated with example 704 of FIG. 7 can be input to a location prediction component 802, which can output a predicted location associated with the target object. For example, attribute information associated with vehicle 606, objects 608-616, and/or target object 618 at various times (e.g., T ₋₂ , T ₋₁ , and T ₀ ) can be input to location prediction component 802.

Example 804 illustrates a predicted location 806 associated with the target object 618. That is, the location prediction component 802 can receive attribute information associated with the target object 618 as well as attribute information associated with objects proximate the target object 618 and output a predicted location 806 representing the target object 618 in the future.

Object 808 represents object of interest 618 at time T _-2 , object 810 represents object of interest 618 at time T _-1 , and object 812 represents object of interest at time T ₀ .

The location prediction component 802 can determine a predicted location 806 based on the attribute information discussed herein. In some examples, the predicted location can be initially represented in a global coordinate system, with a frame of reference originating from the target object, etc. Additionally, the predicted location can be represented relative to a reference line in the environment.

In some examples, the environment represents multiple reference lines, such as reference line 814 and reference line 816. As depicted in FIG. 8 for illustrative purposes, reference line 816 corresponds to, for example, a lane change of an object. In some examples, reference line 814 represents a centerline of a first road segment, and reference line 816 represents a centerline of a second road segment (and/or a transition therebetween). In some examples, such as a single lane road, the environment represents a single reference line. However, in some examples, the environment represents multiple reference lines.

In some examples, the location prediction component 802 can receive as input an indication of a most likely reference line (e.g., 814). In some examples, the location prediction component 802 can determine the likely reference line based at least in part on one or more attributes of the target object 618, of other objects, and/or of the environment, as described herein.

In some examples, the location prediction component 802 can determine a similarity score 818 that represents a similarity between the predicted location 806 and the reference line 814. Additionally, the location prediction component 802 can determine a similarity score 820 that represents a similarity between the predicted location 806 and the reference line 816. In some examples, the similarity score can be based at least in part on individual or cumulative lateral offsets between the predicted location and each reference line, although other factors can be used to determine the similarity score.

In some examples, the location prediction component 802 may determine that the similarity score 818 is lower than the similarity score 820 and, accordingly, select the reference line 814 as a basis for partially defining the predicted location 806. However, in other examples, each potential reference line is input to the location prediction component 802 along with previously calculated attributes such that the location prediction component 802 selects the appropriate reference line and/or trajectory to use as a basis based on machine-learned parameters.

Predicted location 806 may include predicted locations 822, 824, 826, 828, and/or 830. In some examples, predicted location 822 may represent a first distance s and a first lateral offset (e.g., (s ₁ , e _y1 )) relative to reference line 814. Predicted location 824 may represent a second distance s and a second lateral offset (e.g., (s ₂ , e _y2 )) relative to reference line 814. Predicted location 826 may represent a third distance s and a third lateral offset (e.g., (s ₃ , e _y3 )) relative to reference line 814. Predicted location 828 may represent a fourth distance s and a fourth lateral offset (e.g., (s ₄ , e _y4 )) relative to reference line 814. And predicted position 830 may represent a fifth distance s and a fifth lateral offset (e.g., ( _s5 , _ey5 )) relative to reference line 814. Of course, position prediction component 802 may determine fewer or more predicted positions, as discussed herein.

FIG. 9 illustrates a block diagram of an example system 900 for implementing the techniques described herein. In at least one example, the system 900 can include a vehicle 902 that can correspond to the vehicle 108 of FIG. 1 and the vehicle 606 of FIG. 6.

An exemplary vehicle 902 may be a driverless vehicle, such as an autonomous vehicle configured to operate according to a Level 5 classification issued by the National Highway Traffic Safety Administration, which describes a vehicle that can perform all safety-critical functions throughout a trip, with no driver (or passenger) expected to control the vehicle at any time. In such an example, the vehicle 902 may be configured to control all functions from the beginning to the completion of a trip, including all parking functions, and thus does not include a driver and/or controls for operating the vehicle 902, such as a steering wheel, accelerator pedal, and/or brake pedal. This is merely an example, and the systems and methods described herein may be incorporated into any ground-based, air-based, or water-based vehicle, including vehicles that must be manually controlled by a driver at all times, to vehicles that are partially or fully autonomously controlled.

The vehicle 902 may include a vehicle computing device 904, one or more sensor systems 906, one or more emitters 908, one or more communication connections 910, at least one direct connection 912, and one or more drive systems 914.

The vehicle computing device 904 may include one or more processors 916 and a memory 918 communicatively coupled to the one or more processors 916. In the illustrated example, the vehicle 902 is an autonomous vehicle, but the vehicle 902 may be another type of vehicle or a robotic platform. In the illustrated example, the memory 918 of the vehicle computing device 904 stores a localization component 920, a perception component 922, one or more maps 924, one or more system controllers 926, an attribute component 930, a prediction component 928 including a destination prediction component 932, and a location prediction component 934, and a planning component 936. While depicted in FIG. 9 as residing in memory 918 for purposes of illustration, it is contemplated that the localization component 920, the perception component 922, the one or more maps 924, the one or more system controllers 926, the prediction component 928, the attribute component 930, the destination prediction component 932, the location prediction component 934, and the planning component 936 may additionally or alternatively be accessible to the vehicle 902 (e.g., stored in memory separate from the vehicle 902 or otherwise accessible to the vehicle 902).

In at least one example, the localization component 920 can include functionality for receiving data from the sensor system 906 to determine the position and/or orientation of the vehicle 902 (e.g., one or more of x position, y position, z position, roll, pitch, or yaw). For example, the localization component 920 can include and/or request/receive a map of the environment and continuously determine the position and/or orientation of the autonomous vehicle within the map. In some examples, the localization component 920 can receive image data, LIDAR data, RADAR data, time of flight data, IMU data, GPS data, wheel encoder data, etc., using SLAM (simultaneous localization and mapping), CLAMS (simultaneous calibration, localization, and mapping), relative SLAM, bundle adjustment, nonlinear least squares optimization, etc., to accurately determine the position of the autonomous vehicle. In some examples, the localization component 920 can provide data to various components of the vehicle 902 to determine an initial position of the autonomous vehicle, to generate a trajectory, and/or to determine that an object is in proximity to one or more pedestrian crossing areas, and/or to identify candidate reference lines, as discussed herein.

In some examples, and generally, the perception component 922 may include functionality to perform object detection, segmentation, and/or classification. In some examples, the perception component 922 may provide processed sensor data indicative of the presence of an entity in proximity to the vehicle 902 and/or classifying the entity as an entity type (e.g., automobile, pedestrian, bicycle, animal, building, tree, road surface, curb, sidewalk, stop light, stop sign, unknown, etc.). In additional or alternative examples, the perception component 922 may provide processed sensor data indicative of one or more features associated with the detected entity (e.g., tracked object) and/or the environment in which the entity is located. In some examples, the features associated with the entity may include, but are not limited to, an x-position (global and/or local position), a y-position (global and/or local position), a z-position (global and/or local position), an orientation (e.g., roll, pitch, yaw), an entity type (e.g., classification), an entity velocity, an entity acceleration, an entity range (size), etc. Features associated with an environment may include, but are not limited to, the presence of another entity in the environment, the state of another entity in the environment, time of day, day of the week, season, weather, darkness/light indication, etc.

The memory 918 may further include one or more maps 924 that may be used by the vehicle 902 to navigate within the environment. For purposes of this discussion, a map may be any number of data structures modeled in 2-, 3-, or N-dimensions that may provide information about the environment, such as, but not limited to, topology (e.g., intersections), streets, mountain ranges, roads, terrain, and the environment in general. In some examples, the map may include, but is not limited to, texture information (e.g., color information (e.g., RGB color information, Lab color information, HSV/HSL color information), etc.), intensity information (e.g., LIDAR information, RADAR information, etc.), spatial information (e.g., image data projected onto a mesh, individual "surfels" (e.g., polygons associated with individual colors and/or intensities), reflectance information (e.g., specular reflectance information, retroreflectance information, BRDF information, BSSRDF information, etc.), and the like. In one example, the map may include a 3-dimensional mesh of the environment. In some examples, the map can be stored in a tiled format, with each tile of the map representing a discrete portion of the environment, and can be loaded into the working memory as needed. In at least one example, the one or more maps 924 can include at least one map (e.g., an image and/or a mesh).

In some examples, the vehicle 902 can be controlled based at least in part on the map 924. That is, the map 924 can be used in conjunction with the localization component 920, the perception component 922, the prediction component 928, and/or the planning component 936 to determine a position of the vehicle 902, identify objects within the environment, and/or generate a route and/or trajectory for navigating the environment.

In some examples, one or more maps 924 can be stored on a remote computing device (e.g., computing device 940) accessible via network 938. In some examples, multiple maps 924 can be stored, for example, based on characteristics (e.g., type of entity, time of day, day of the week, season of the year, etc.). Storing multiple maps 924 can have similar memory requirements but can improve the speed at which data in the maps can be accessed.

In at least one example, the vehicle computing device 904 can include one or more system controllers 926 that can be configured to control the steering, propulsion, braking, safety, emitter, communication, and other systems of the vehicle 902. These system controllers 926 can communicate with and/or control corresponding systems of the drive system 914 and/or other components of the vehicle 902.

In general, the prediction component 928 can include functionality for generating predictive information associated with objects in the environment. In some examples, the prediction component 928 can be implemented to predict the location of a pedestrian proximate a crosswalk area (or area or location associated with pedestrians crossing a road) in the environment when the pedestrian crosses or prepares to cross the crosswalk area. In some examples, the techniques discussed herein can be implemented to predict the location of an object (e.g., a vehicle, a pedestrian, etc.) as the vehicle crosses the environment. In some examples, the prediction component 928 can generate one or more predicted trajectories of the target object based on attributes of the target object and/or other objects proximate the target object.

The attribute component 930 can include functionality for determining attribute information associated with objects in the environment. In some examples, the attribute component 930 can receive data from the perception component 922 and determine attribute information for objects over time.

In some examples, attributes of an object (e.g., a pedestrian) can be determined based on sensor data captured over time and can include one or more of the pedestrian's position at a point in time (e.g., the position can be expressed in the frame of reference described above), the pedestrian's velocity at a point in time (e.g., magnitude and/or angle with respect to a first axis (or other reference line)), and the pedestrian's acceleration at a point in time. The information may include, but is not limited to, the pedestrian's speed (e.g., magnitude and/or angle with respect to a first axis (or other reference line)), the pedestrian's acceleration at that time, an indication of whether the pedestrian is in a drivable area (e.g., whether the pedestrian is on a sidewalk or road), an indication of whether the pedestrian is in a crosswalk area, an indication of whether the pedestrian is running a red light, the state of an indicator controlling the area (e.g., whether the crosswalk is controlled by a signal and/or the state of the signal), vehicle context (e.g., the presence of vehicles in the environment and attributes associated with the vehicles), flux through the crosswalk area over a period of time (e.g., the number of objects (e.g., vehicles and/or pedestrians) passing through the crosswalk area over a period of time), object association (e.g., whether the pedestrian is moving among a group of pedestrians), distance to the crosswalk in a first direction (e.g., global x-direction), distance to the crosswalk in a second direction (e.g., global y-direction), distance to the road within the crosswalk area (e.g., the shortest distance to the road within the crosswalk area), etc.

In some examples, attributes can be determined for the target object (e.g., a vehicle) and/or other objects (e.g., other vehicles) in proximity to the target object. For example, the attributes can include, but are not limited to, the speed of the object at a time, the acceleration of the object at a time, the position of the object at a time (e.g., in global or local coordinates), a bounding box associated with the object at a time (e.g., representing the range, roll, pitch, and/or yaw of the object), the lighting conditions associated with the object at a time (e.g., headlights, brake lights, hazard lights, turn signals, backlights, etc.), the distance of the object to map elements at a time (e.g., distance to stop lines, traffic lines, speed bumps, yield lines, intersections, roadways, etc.), the distance of the object to other objects, the classification of the object (car, vehicle, animal, truck, etc.), features associated with the object (whether or not a lane is being changed, whether or not a vehicle is double-parked, etc.), etc.

In some examples, any combination of object attributes may be determined as discussed herein.

The attributes may be determined over time (e.g., at times T _-M , ..., T _-2 , T _-1 , T ₀ (where M is an integer), and the various times represent any time up to the most recent time) and input to a destination prediction component 932 and/or a location prediction component 934 to determine predictive information associated with such objects.

The destination prediction component 932 may include functionality for determining a destination of an object in the environment, as discussed herein. In a pedestrian context, the destination prediction component 932 may determine which crosswalk area applies to the pedestrian based on the pedestrian being within a threshold distance of the crosswalk area, as discussed herein. In at least some examples, such a destination prediction component 932 may determine a point on the opposite sidewalk regardless of the presence of a crosswalk. Additionally, attributes of objects associated with any time period may be input to the destination prediction component 932 to determine a score, probability, and/or margin that the pedestrian is heading toward or associated with a crosswalk area.

In some examples, the destination prediction component 932 is a machine learning model, such as a neural network, a fully connected neural network, a convolutional neural network, or a recurrent neural network.

In some examples, the destination prediction component 932 can be trained by reviewing data logs to determine events in which a pedestrian crosses a crosswalk. Such events can be identified, attributes can be determined for the object (e.g., pedestrian) and the environment, and data representing the event can be identified as training data. The training data can be input to a machine learning model, and known results (e.g., ground truth, such as known "future" attributes) can be used to adjust the weights and/or parameters of the machine learning model to minimize error.

The location prediction component 934 may include functionality for generating or otherwise determining a predicted location associated with an object in the environment. For example, as discussed herein, attribute information may be determined for one or more objects in the environment, including the target object and/or other objects in proximity to the target object. In some examples, attributes associated with the vehicle 902 may be used to determine a predicted location associated with an object in the environment.

The location prediction component 934 can further include functionality for expressing attribute information in various frames of reference, as discussed herein. In some examples, the location prediction component 934 can use the object's position at time _T0 as the origin of the frame of reference, which can be updated for each instance of time.

In some examples, the location prediction component 934 may include functionality to identify candidate reference lines within the environment (e.g., based on map data), select a reference line (e.g., based on a similarity score), and determine a predicted location for the reference line.

In some examples, the location prediction component 934 is a machine learning model, such as a neural network, a fully connected neural network, a convolutional neural network, a recurrent neural network, or any combination thereof.

For example, the location prediction component 934 can be trained by reviewing data logs and determining attribute information. Training data representing relevant events (e.g., vehicles traveling a threshold distance from a reference line, pedestrians crossing a crosswalk, pedestrians running red lights, etc.) can be input into a machine learning model where known outcomes (e.g., ground truth, such as known "future" attributes/locations) can be used to adjust the weights and/or parameters of the machine learning model to minimize error.

In general, the planning component 936 can determine a path for the vehicle 902 to follow to traverse an environment. For example, the planning component 936 can determine various routes and trajectories, and various levels of detail. For example, the planning component 936 can determine a route to travel from a first location (e.g., a current location) to a second location (e.g., a location of interest). For purposes of this discussion, the route can be a sequence of waypoints for traveling between the two locations. As non-limiting examples, the waypoints include streets, intersections, Global Positioning System (GPS) coordinates, and the like. Additionally, the planning component 936 can generate instructions for guiding the autonomous vehicle along at least a portion of the route from the first location to the second location. In at least one example, the planning component 936 can determine how to guide the autonomous vehicle from a first waypoint in the sequence of waypoints to a second waypoint in the sequence of waypoints. In some examples, the instructions can be a trajectory, or a portion of a trajectory. In some examples, the multiple trajectories can be generated substantially simultaneously (e.g., within technical tolerances) according to a receding horizon technique, and one of the multiple trajectories is selected for the vehicle 902 to navigate.

In some examples, the planning component 936 can generate one or more trajectories for the vehicle 902 based at least in part on predicted positions associated with objects in the environment. In some examples, the planning component 936 can use temporal logic, such as linear temporal logic and/or signal temporal logic, to evaluate one or more trajectories for the vehicle 902.

As can be appreciated, the components discussed herein (e.g., localization component 920, perception component 922, one or more maps 924, one or more system controllers 926, prediction component 928, attribute component 930, destination prediction component 932, location prediction component 934, and planning component 936) are described as separate for purposes of explanation. However, the operations performed by the various components may be combined or performed by any other component. Additionally, any component discussed as being implemented in software may be implemented in hardware, and vice versa. Additionally, any functionality implemented in the vehicle 902 may be implemented in the computing device 940, or another component, and vice versa.

In at least one example, the sensor system 906 can include time of flight sensors, LIDAR sensors, RADAR sensors, ultrasonic transducers, sonar sensors, position sensors (e.g., GPS, compass, etc.), inertial sensors (e.g., inertial measurement units (IMUs), accelerometers, magnetometers, gyroscopes, etc.), cameras (RGB, infrared, intensity, depth, etc.), microphones, wheel encoders, environmental sensors (temperature sensors, humidity sensors, light sensors, pressure sensors, etc.), and the like. The sensor system 906 can include multiple instances of each of these or other types of sensors. For example, the time of flight sensors can include individual time of flight sensors located at the corners, front, rear, sides, and/or top of the vehicle 902. As another example, the camera sensors can include multiple cameras located at various locations on the exterior and/or interior of the vehicle 902. The sensor system 906 can provide input to the vehicle computing device 904. Additionally or alternatively, the sensor system 906 may transmit sensor data over one or more networks 938 at a particular frequency, after a predetermined period of time, or in near real-time to one or more computing devices 940.

The vehicle 902 may also include one or more emitters 908 for emitting light and/or sound, as described above. Emitters 908 in this example include internal audio and video emitters for communicating with passengers of the vehicle 902. By way of example and not limitation, internal emitters may include speakers, lights, signs, display screens, touch screens, haptic emitters (e.g., vibration and/or force feedback), mechanical actuators (e.g., seat belt tensioners, seat positioners, head rest positioners, etc.), and the like. Emitters 908 in this example also include external emitters. Exterior emitters in this example include, but are not limited to, lighting (e.g., turn signals, signs, light arrays, etc.) for communicating direction of travel or other indicators of vehicle operation, and one or more audio emitters (e.g., speakers, speaker arrays, horns, etc.) for audibly communicating with pedestrians or other nearby vehicles, one or more of which may include acoustic beam steering technology.

The vehicle 902 may also include one or more communication connections 910 that enable communication between the vehicle 902 and one or more other local or remote computing devices. For example, the communication connections 910 may facilitate communication with other local computing devices on the vehicle 902 and/or the drive system 914. The communication connections 910 may also enable the vehicle to communicate with other nearby computing devices (e.g., other nearby vehicles, signals, etc.). The communication connections 910 may also enable the vehicle 902 to communicate with remotely operated computing devices or other remote services.

The communication connection 910 may include a physical and/or logical interface for connecting the vehicle computing device 904 to another computing device or network, such as the network 938. For example, the communication connection 910 may enable Wi-Fi based communications, such as over frequencies defined by the IEEE 802.11 standard, short-range radio frequencies such as Bluetooth, cellular communications (e.g., 2G, 3G, 4G, 4G LTE, 5G, etc.), or any suitable wired or wireless communication protocol that enables each computing device to interface with other computing devices.

In at least one example, the vehicle 902 can include one or more drive systems 914. In some examples, the vehicle 902 can have a single drive system 914. In at least one example, when the vehicle 902 has multiple drive systems 914, the individual drive systems 914 can be located at opposite ends of the vehicle 902 (e.g., the front and rear, etc.). In at least one example, the drive system 914 can include one or more sensor systems for detecting conditions surrounding the drive system 914 and/or the vehicle 902. By way of example and not limitation, the sensor systems can include one or more wheel encoders (e.g., rotary encoders) that sense the rotation of the wheels of the drive module, inertial sensors (e.g., inertial measurement units, accelerometers, gyroscopes, magnetometers, etc.) that measure the orientation and acceleration of the drive module, cameras or other imaging sensors, ultrasonic sensors that acoustically detect objects surrounding the drive system, LIDAR sensors, RADAR sensors, etc. Some sensors, such as wheel encoders, can be unique to the drive system 914. In some cases, the sensor systems on the drive system 914 may overlap or complement the corresponding systems on the vehicle 902 (e.g., sensor system 906).

The drive system 914 may include many vehicle systems, such as a high voltage battery, a motor for propelling the vehicle, a steering system (which may be electric) including an inverter that converts direct current from the battery to alternating current for use in other vehicle systems, a steering motor, and a steering rack, a brake system including hydraulic or electric actuators, a suspension system including hydraulic and/or pneumatic components, a stability control system for distributing braking force to mitigate loss of traction and maintain control, an HVAC system, lighting (e.g., lighting such as head/tail lights that illuminate the exterior surroundings of the vehicle), and one or more other systems (e.g., cooling systems, safety systems, on-board charging systems, electrical components such as DC/DC converters, high voltage junctions, high voltage cables, charging systems, charging ports, etc.). Additionally, the drive system 914 may include a drive system controller that may receive and pre-process data from the sensor systems and control the operation of various vehicle systems. In some examples, the drive system controller may include one or more processors and a memory communicatively coupled to the one or more processors. The memory may store one or more components for performing various functions of the drive system 914. Additionally, drive system 914 also includes one or more communication connections that enable each drive system to communicate with one or more other local or remote computing devices.

In at least one example, the direct connection 912 can provide a physical interface for coupling one or more drive systems 914 with the body of the vehicle 902. For example, the direct connection 912 can provide for the transfer of energy, fluid, air, data, etc. between the drive system 914 and the vehicle. In some examples, the direct connection 912 can further releasably secure the drive system 914 to the body of the vehicle 902.

In at least one example, the localization component 920, the perception component 922, the one or more maps 924, the one or more system controllers 926, the prediction component 928, the attribute component 930, the destination prediction component 932, the location prediction component 934, and the planning component 936 can process the sensor data as described above and transmit their respective outputs to one or more computing devices 940 via one or more networks 938. In at least one example, the localization component 920, the one or more maps 924, the one or more system controllers 926, the prediction component 928, the attribute component 930, the destination prediction component 932, the location prediction component 934, and the planning component 936 can transmit their respective outputs to one or more computing devices 940 at a particular frequency, after a predetermined period of time, in near real-time, etc.

In some examples, the vehicle 902 can transmit sensor data over the network 938 to one or more computing devices 940. In some examples, the vehicle 902 can transmit raw sensor data to the computing device 940. In other examples, the vehicle 902 can transmit processed sensor data and/or representations of the sensor data to the computing device 940. In some examples, the vehicle 902 can transmit sensor data at a particular frequency, after a predetermined period of time, in near real-time, etc. In some examples, the vehicle 902 can transmit sensor data (raw or processed) to the computing device 940 as one or more log files.

The computing device 940 may include a processor 942 and memory 944 that stores the training component 946.

In some examples, the training component 946 can include functionality for training one or more models to determine predictive information, as discussed herein. In some examples, the training component 946 can communicate information generated by the one or more models to the vehicle computing device 904 to modify how the vehicle 902 is controlled in response to different conditions.

For example, the training component 946 can train one or more machine learning models to generate the predictive components discussed herein. In some examples, the training component 946 can include functionality to search the data log and determine attribute and/or location (e.g., in any one or more reference frames) information associated with objects. Log data corresponding to particular scenarios (e.g., pedestrians approaching and crossing a crosswalk area, pedestrians running red lights, objects offset from the center line and rounding a curve, etc.) can represent training data. The training data can be input to the machine learning model, and known results (e.g., ground truth, such as known "future" attributes) can be used to adjust the weights and/or parameters of the machine learning model to minimize error.

For example, some or all aspects of the components discussed herein may include any model, algorithm, and/or machine learning algorithm. For example, in some examples, the components in memory 944 (and memory 918 described above) may be implemented as a neural network. In some examples, training component 946 may utilize a neural network to generate and/or execute one or more models for determining segmentation information from sensor data, as discussed herein.

As described herein, an exemplary neural network is a biologically inspired algorithm that passes input data through a series of connected layers to generate an output. Each layer of a neural network may comprise another neural network, or may comprise any number of layers (convolutional or not). As can be understood in the context of the present disclosure, a neural network may utilize machine learning, which may refer to a broad class of algorithms in which an output is generated based on trained parameters.

Although discussed in the context of neural networks, any type of machine learning consistent with this disclosure may be used. For example, machine learning algorithms may include regression algorithms (e.g., ordinary least squares regression (OLSR), linear regression, logistic regression, stepwise regression, multivariate adaptive regression splines (MARS), locally weighted scatterplot smoothing (LOESS)), instance-based algorithms (e.g., ridge regression, least absolute attrition selection operator (LASSO), elastic nets, least angle regression (LARS)), decision tree algorithms (e.g., classification and regression trees (CART), iterative binary tree 3 (ID3), chi-squared automated interaction detection (CH2), etc.), and may be used in conjunction with other algorithms. AID), decision stumps, conditional decision trees), Bayesian algorithms (e.g., Naïve Bayes, Gaussian Naïve Bayes, Multinomial Naïve Bayes, Average Ordinary Attribute Classifier (AODE), Bayesian Belief Networks (BNN), Bayesian Networks), clustering algorithms (e.g., k-means, k-medians, Expectation Maximization (EM), Hierarchical Clustering), Association Rule Learning Algorithms (e.g., Perceptron, Backpropagation, Hopfield Networks, Radial Basis Function Network (RBFN)), deep learning algorithms (Deep Boltzmann Machine (DBM), Deep Belief Networks (DBN), Convolutional Neural Networks (CNN), Stacked Auto-Encoders), dimensionality reduction algorithms (e.g., Principal Component Analysis (PCA), Principal Component Regression (PCR), Partial Least Squares Regression (PLSR), Sammon Mapping, Multidimensional Scaling (MDS), Projection Pursuit, Linear Discriminant Analysis (LDA), Mixed Discriminant Analysis (MDA), Quadratic Discriminant Analysis (QDA), Flexible Discriminant Analysis (FDA)), ensemble algorithms (e.g., Boosting, Bootstrapped These may include, but are not limited to, Aggregation (Bagging), AdaBoost, Stacked Generalization (Blending), Gradient Boosting Machines (GBM), Gradient Boosted Regression Trees (GBRT), Random Forest), SVM (Support Vector Machine), supervised learning, unsupervised learning, semi-supervised learning, etc.

Additional example architectures include neural networks such as ResNet50, ResNet101, VGG, DenseNet, and PointNet.

The processor 916 of the vehicle 902 and the processor 942 of the computing device 940 may be any suitable processor capable of executing instructions to perform operations and process data as described herein. By way of example and not limitation, the processors 916 and 942 may comprise one or more central processing units (CPUs), graphics processing units (GPUs), or other devices or portions of devices that process electronic data and convert it to other electronic data that may be stored in registers and/or memory. In some examples, integrated circuits (such as ASICs), gate arrays (such as FPGAs), and other hardware devices may also be considered processors so long as they are configured to implement coded instructions.

Memories 918 and 944 are exemplary non-transitory computer-readable media. Memories 918 and 944 may store operating systems and one or more software applications, instructions, programs, and/or data to implement the methods and functions attributed to the various systems described herein. In various implementations, the memory may be implemented using any suitable memory technology, such as static random access memory (SRAM), synchronous dynamic RAM (SDRAM), non-volatile/flash memory, or other types of memory capable of storing information. The architectures, systems, and individual elements described herein may include many other logical, programmatic, and physical components, but those shown in the accompanying drawings are merely examples for the discussion herein.

9 is illustrated as a distributed system, it should be noted that in alternative examples, components of the vehicle 902 may be associated with the computing device 940 and/or components of the computing device 940 may be associated with the vehicle 902. That is, the vehicle 902 may perform one or more of the functions associated with the computing device 940, and vice versa. Additionally, aspects of the prediction component 928 (and subcomponents) may be performed on any of the devices discussed herein.

10 illustrates an example process 1000 of capturing sensor data, determining attributes associated with an object, determining a predicted location based on the attributes, and controlling a vehicle based on the predicted location. For example, some or all of the process 1000 may be performed by one or more components of FIG. 9 as described herein. For example, some or all of the process 1000 may be performed by a vehicle computing device 904. Additionally, any operations described in the example process 1000 may be performed in parallel, in a different order than shown in the process 1000, with any of the operations of the process 1000 shown omitted, and/or in combination with any of the operations discussed herein.

In operation 1002, the process may include receiving sensor data of the environment. In some examples, operation 1002 may include receiving and/or capturing time of flight data, LIDAR data, image data, RADAR data, etc. of the environment. In some examples, operation 1002 may be performed by a vehicle (e.g., an autonomous vehicle) as the vehicle traverses the environment.

At operation 1004, the process may include determining that an object is in the environment based at least in part on the sensor data. For example, operation 1004 may include classifying the object as a pedestrian in the environment. In some examples, operation 1004 may include determining whether the object (e.g., a pedestrian) is on a sidewalk, on a road, running a red light, etc.

In operation 1006, the process may include determining whether the object is associated with a destination in the environment. For example, operation 1006 may include accessing map data of the environment and determining whether a crosswalk area is within a threshold distance of the object. If there is one crosswalk area and the object is on the sidewalk, operation 1006 may include identifying a location across the drivable area as the destination. If the object is on a road and is proximate to one crosswalk, operation 1006 may include disambiguating between the two destinations. In some examples, operation 1006 may include determining a likelihood that the object will approach and/or cross a particular crosswalk area based at least in part on attributes associated with the object. In some examples, operation 1006 provides such destinations to pedestrians regardless of the presence of a crosswalk area proximate to the pedestrian.

In some examples, operation 1006 may include inputting attributes into a destination prediction component (e.g., destination prediction component 320) to determine a destination associated with the object in the environment. In some examples, the attributes input into the destination prediction component 320 may be the same as or similar to the attributes determined below in operations 1008 and 1010. In some examples, attributes may be determined for the object prior to determining the destination in the environment. Also, in some examples, attributes may be determined in parallel using a reference frame based on different destinations in the environment to determine likely destinations in the environment.

If the object is not associated with the destination (e.g., "NO" at operation 1006), operation 1006 can continue with operation 1002 to capture additional data within the environment.

If the object is then associated with the destination (e.g., "YES" at operation 1006), operation can continue to operation 1008.

At operation 1008, the process may include determining a first attribute associated with the object, the first attribute associated with a first time. In some examples, the attribute may include a position of the object (e.g., a pedestrian) at a time (e.g., the position may be expressed in a frame of reference as discussed herein), a size of the object or a bounding box associated with the object (e.g., length, width, and/or height), a speed of the pedestrian at a time (e.g., a size and/or angle relative to a first axis (or other reference line)), an acceleration of the pedestrian at a time, an indication of whether the pedestrian is in a drivable area (e.g., whether the pedestrian is on a sidewalk or a road), an indication of whether the pedestrian is in a crosswalk area, an indication of whether the pedestrian is running a red light, a state of an indicator controlling the area (e.g., whether the crosswalk is controlled by a light, These may include, but are not limited to, one or more of: vehicle context (such as the presence of vehicles in the environment and attributes associated with the vehicles), flux through the crosswalk area in a period of time (such as the number of objects (such as vehicles and/or additional pedestrians) passing through the crosswalk area in a period of time), object association (e.g., whether a pedestrian is moving among a group of pedestrians), distance to the crosswalk in a first direction (e.g., global x-direction), distance to the crosswalk in a second direction (e.g., global y-direction), distance to a road in the crosswalk area (e.g., shortest distance to a road in the crosswalk area), distance to other objects, etc.

In operation 1010, the process may include determining a second attribute associated with the object, the second attribute being associated with a second time after the first time. In some examples, operation 1010 may be omitted (such that only attributes associated with the first time may be determined and/or used), while in some examples attributes associated with additional or different time instances may be determined.

At operation 1012, the process may include determining a predicted location of the object at a third time after the second time based at least in part on the first attribute, the second attribute, and the destination. In some examples, operation 1012 may include inputting the attribute information into a location prediction component (e.g., location prediction component 404) and receiving as an output a predicted location associated with the object in the environment. As described herein, in some examples, the attribute and/or the predicted location may be expressed in one or more frames of reference based at least in part on the location of the object at the first time and/or the second time and the location of the destination in the environment.

In operation 1014, the process may include controlling the vehicle based at least in part on the predicted position. In some examples, operation 1014 may include generating a trajectory to stop the vehicle or otherwise control it to safely traverse the environment.

11 illustrates an exemplary process of capturing sensor data, determining that a first object and a second object are in an environment, determining attributes associated with the second object, determining a predicted position based on the attributes and the reference line, and controlling the vehicle based on the predicted position. For example, some or all of the process 1100 may be performed by one or more components of FIG. 9 as described herein. For example, some or all of the process 1100 may be performed by the vehicle computing device 904. Additionally, any operations described in the illustrated process 1100 may be performed in parallel, in a different order than shown in the process 1100, with any of the operations of the illustrated process 1100 omitted, and/or in combination with any of the operations discussed herein.

In operation 1102, the process may include receiving sensor data of the environment. In some examples, operation 1102 may include receiving and/or capturing time of flight data, LIDAR data, image data, RADAR data, etc. of the environment. In some examples, operation 1102 may be performed by a vehicle (e.g., an autonomous vehicle) as the vehicle traverses the environment.

At operation 1104, the process may include determining that a first object is in the environment based at least in part on the sensor data. For example, operation 1104 may include determining a target object that is subject to the predictive action as discussed herein. By way of example, determining the target object may include selecting an object as the target object from a plurality of objects in the environment. In some examples, the target object may be selected based on a likelihood of intersection between the target object's path and the vehicle capturing the sensor data (e.g., vehicle 902), a distance between the target object and the vehicle capturing the sensor data (e.g., vehicle 902), etc.

In operation 1106, the process may include determining whether a second object is proximate to the first object in the environment. In some examples, operation 1106 may include determining whether the second object is within a threshold distance of the first object. In some examples (e.g., in a crowded environment), operation 1106 may include determining the N objects closest to the first object (where N is an integer). Without limitation, in at least some examples, such determination excludes objects with certain characteristics, such as objects of different classifications, objects with opposite directions of motion, etc.

If the second object is not proximate to the first object (e.g., “NO” at operation 1106), the process may return to operation 1102. However, in some examples, the process may continue to operation 1112 where a predicted position of the first object is determined without attributes associated with the second object (e.g., the predicted position of the first object may be determined based, at least in part, on attributes associated with the first object). That is, the predicted position of the first object may be determined in some examples regardless of whether the second object is proximate to the first object and/or regardless of whether attributes have been determined for any second object.

If the second object is proximate to the first object (e.g., "YES" at operation 1106), processing continues at operation 1108.

At operation 1108, the process may include determining a first attribute associated with the second object, the second attribute associated with the first time. In some examples, attributes may be determined for the first object, the second object, and/or other objects in the environment. For example, the attributes may include, but are not limited to, one or more of: the speed of the object at a time; the acceleration of the object at a time; the position of the object at a time (e.g., in global or local coordinates); a bounding box associated with the object at a time (e.g., representing the range, roll, pitch, and/or yaw of the object); the state of lighting associated with the object at a first time (e.g., headlights, brake lights, hazard lights, turn signals, backlights, etc.); a wheel direction indication for the object; the distance of the object to a map element at a time (e.g., distance to a stop line, turn line, speed bump, yield line, intersection, roadway, etc.); the distance relative to other objects in one or more frames of reference; a classification of the object (car, vehicle, animal, truck, bicycle, etc.); a feature associated with the object (whether the object is changing lanes, whether it is a double-parked vehicle, etc.); lane features, etc.

In operation 1110, the process may include determining a second attribute associated with the second object, the second attribute being associated with a second time after the first time. In some examples, operation 1110 may be omitted (such that only the attributes associated with the first time may be used), while in some examples, attributes associated with additional or different time instances may be determined.

At operation 1112, the process may include determining a predicted location of the first object at a third time after the second time based at least in part on the first attribute and the second attribute, the predicted location being relative to a reference line in the environment. In some examples, operation 1112 may include inputting attribute information associated with the first object and/or the second object into a location prediction component (e.g., location prediction component 802) to determine a predicted location associated with the first object.

In some examples, operation 1112 may include receiving or otherwise determining a reference line most closely associated with the predicted position and representing the predicted position relative to the reference line. For example, operation 1112 may include determining a similarity score between the predicted position and a candidate reference line, selecting the reference line based on the similarity score, or any other mechanism.

In operation 1114, the process may include controlling the vehicle based at least in part on the predicted position. In some examples, operation 1114 may include generating a trajectory to stop the vehicle or otherwise control the vehicle to safely traverse the environment.

(Example item)
A: A system comprising: one or more processors; and one or more computer-readable media storing instructions executable by the one or more processors that, when executed, cause the system to perform operations comprising: capturing sensor data of an environment using sensors of an autonomous vehicle; determining that an object is within the environment based at least in part on the sensor data; determining that the object is associated with a destination within the environment based at least in part on the map data and the sensor data; determining a first attribute associated with the object, the first attribute associated with a first time; determining a second attribute associated with the object, the second attribute associated with a second time after the first time; inputting the first attribute, the second attribute, and the destination into a machine learning model, the first attribute and the second attribute represented by a frame of reference based at least in part on the destination; receiving from the machine learning model a predicted position of the object at a third time after the second time; and controlling the autonomous vehicle based at least in part on the predicted position of the object in the environment at the third time.

B: The system of paragraph A, wherein the object is a pedestrian and the destination is associated with the perimeter of a crosswalk area in the environment and faces a drivable surface associated with the pedestrian.

C: The system of paragraph A or B, further comprising an operation of determining that the object is associated with a destination based at least in part on inputting the first attribute and the second attribute into a destination prediction component, and receiving the destination from the destination prediction component, the destination prediction component comprising another machine learning model.

D: Any of the systems of paragraphs A to C, wherein the operation further comprises: the predicted position associated with the object at the third time comprises a lateral offset based at least in part on the frame of reference, and a distance along an axis of the frame of reference representing a difference between the position of the object at the second time and the predicted position.

E: Any of the systems of paragraphs A to D, further comprising: an operation of establishing a frame of reference, the first position of the object at the second time being associated with coordinates of the frame of reference, the first axis being based at least in part on the destination and the surface of material, the second axis being perpendicular to the first axis, and the predicted position being based at least in part on the frame of reference.

F: A method comprising: receiving sensor data representative of an environment; determining, based at least in part on the sensor data, that an object is in the environment; determining a location in the environment, the location associated with a pedestrian crossing area; determining a first attribute associated with the object, the first attribute associated with a first time; determining a second attribute associated with the object, the second attribute associated with a second time after the first time; inputting the first attribute, the second attribute, and the location into a machine learning model; and receiving from the machine learning model a predicted location associated with the object at a third time after the second time.

G: The method of paragraph F, further comprising capturing sensor data using a sensor on the vehicle and controlling the vehicle based at least in part on a predicted position of the object in the environment at a third time.

H: The method of paragraph F or G, wherein the location is a first location, further comprising: determining the first location based at least in part on at least one of map data or sensor data representative of the environment; determining a threshold area associated with the first location; determining a second location of the object within the environment; determining that the second location of the object is within a threshold area; and selecting the location as a destination associated with the object based at least in part on the second location being within the threshold area and at least in part on the first attribute or at least one of the second attributes.

I: The method of any of paragraphs F-H, wherein the location is a first location, and further comprising establishing a frame of reference, wherein a second location of the object at a second time is associated with coordinates of the frame of reference, the first axis being based at least in part on the coordinates and the first location, the second axis being perpendicular to the first axis, and the first attribute being based at least in part on the frame of reference.

J: The method of paragraph I, further comprising determining a velocity of the object at a second time and determining an angle between a velocity vector representing the velocity and the first axis, the second attribute comprising the angle.

K: The method of paragraph I or J, wherein the location is a first location and the predicted location associated with the object at a third time comprises a lateral offset relative to the second axis and a distance along the first axis representing a difference between the second location and the predicted location of the object at the second time.

L: The method of any of paragraphs F-K, further comprising determining a number of objects entering the crosswalk area during a period of time, the second attribute comprising the number of objects.

M: The method of any of paragraphs F to L, wherein the object is a first object, further comprising: determining that a second object is in the environment based at least in part on the sensor data; determining at least one of a position, a velocity, or an acceleration associated with the second object as an object context; and determining a predicted position associated with the object further based at least in part on the object context.

N: Any of the methods of paragraphs F to M, further comprising binning at least a portion of the predicted locations to determine binned predicted locations.

O: The method of any of paragraphs F to N, wherein the first attribute comprises at least one of: a position of the object at a first time; a velocity of the object at a first time; a path of the object at a first time; a first distance between the object and a first portion of the pedestrian crossing area at a first time; a second distance between the object and a second portion of the pedestrian crossing area at a first time; an acceleration of the object at a first time; an indication of whether the object is in a drivable area; a state of a control area indicator; a vehicle context; or an association of the object.

P: A non-transitory computer-readable medium having stored thereon instructions that, when executed, cause one or more processors to perform operations including receiving sensor data representative of an environment; determining, based at least in part on the sensor data, that an object is within the environment; determining a location within the environment, the location associated with at least one of a crosswalk area or a non-driveable area of the environment; determining a first attribute associated with the object, the first attribute associated with a first time; determining a second attribute associated with the object, the second attribute associated with a second time after the first time; inputting the first attribute, the second attribute, and the location into a machine learning model; and receiving from the machine learning model a predicted location associated with the object at a third time after the second time.

Q: The non-transitory computer-readable medium of paragraph P, wherein the location is a first location, and the operations further include determining the first location based at least in part on at least one of map data representing the environment or sensor data representing the environment, determining a threshold area associated with the first location, determining a second location of an object within the environment, determining that the second location of the object is within the threshold area, and selecting the first location as a destination associated with the object based at least in part on that the second location of the object is within the threshold area and at least one of the first attribute or the second attribute.

R: The non-transitory computer readable medium of paragraph P or Q, wherein the location is a first location, and the operation is establishing a frame of reference, wherein a second location of the object at a second time is associated with coordinates of the frame of reference, a first axis is based at least in part on the coordinates and the first location, a second axis is perpendicular to the first axis, and the first attribute is based at least in part on the frame of reference.

S: The non-transitory computer readable medium of paragraph R, wherein the location is a first location and a predicted location associated with the object at a third time comprises a lateral offset along a second axis and a distance along the first axis representing a difference between the second location of the object at the second time and the predicted location.

T: The non-transitory computer-readable medium of any of paragraphs P to S, further comprising determining that the object is not associated with a pedestrian crossing area and determining that the location is associated with a non-driveable area of the environment.

U: A system comprising one or more processors and one or more computer-readable media having instructions executable by the one or more processors stored thereon, the instructions, when executed, causing the system to perform operations comprising: capturing sensor data of an environment using sensors on the autonomous vehicle; determining that an object is in the environment based at least in part on the sensor data; receiving a reference line associated with the object in the environment; determining a first attribute associated with the object, the first attribute being associated with a first time; determining a second attribute associated with the object, the second attribute being associated with a second time after the first time; inputting the first attribute, the second attribute, and the reference line into a machine learning model; receiving from the machine learning model a predicted position of the object at a third time after the second time, the predicted position being relative to the reference line in the environment; and controlling the autonomous vehicle based at least in part on the predicted position of the object in the environment at the third time.

V: The system of paragraph U, further comprising: the object is a first object; the action is determining a third attribute associated with a second object proximate to the first object, the third attribute being associated with a first time; and a fourth attribute associated with the second object, the fourth attribute being associated with a second time; and inputting the third attribute and the fourth attribute into a machine learning model to determine a predicted location of the first object at the third time.

W: The system of paragraph V, wherein at least one of the first attribute, the second attribute, the third attribute, or the fourth attribute is
the velocity of the second object at the first time;
the acceleration of the second object at the first time;
a position of a second object at a first time;
a bounding box associated with the second object at the first time;
a lighting condition associated with a second object at a first time;
a first distance between the second object and the map element at a first time;
a second distance between the first object and the second object;
Classifying the second object;
or a feature associated with a second object;
A system comprising at least one of the following:

X: Any system of paragraphs U to W, where the predicted position is a distance along the reference line and a lateral offset from the reference line.

Y: Any of the systems of paragraphs U to X, wherein the machine learning model is a first machine learning model and the reference line is received from a second machine learning model trained to output the reference line.

Z: A method comprising: receiving sensor data representing an environment; determining that an object is in the environment; receiving a reference line associated with the object; determining a first attribute associated with the object, the first attribute being associated with a first time; determining a second attribute associated with the object, the second attribute being associated with a second time after the first time; inputting the first attribute, the second attribute, and the reference line into a machine learning model; and receiving from the machine learning model a predicted position of the object at a third time after the second time, the predicted position relative to the reference line in the environment.

AA: The method of paragraph Z, further comprising: capturing sensor data using a sensor of the vehicle; and controlling the vehicle based at least in part on a predicted position of the object in the environment at a third time.

AB: The method of paragraph AA, wherein the object is one of a number of objects in the environment, and further comprising selecting the object as a target object based at least in part on a distance between the object and the vehicle in the environment.

AC: The method of any of paragraphs Z to AB, wherein the object is one of a number of objects in an environment, the object being a target object, further comprising: selecting a number of the number of objects based at least in part on a number of objects proximate to the target object; determining attributes associated with the object; and inputting the attributes into a machine learning model to determine a predicted location.

AD: The method of paragraph AC, further comprising selecting an object based at least in part on a classification associated with the object.

AE: Any of the methods of paragraphs Z to AD, in which the reference line corresponds to a center line of the driveable area and the predicted position includes a distance along the reference line and a lateral offset from the reference line.

AF: Any of the methods of paragraphs Z to AE, wherein the first attribute and the second attribute are expressed with respect to a frame of reference, and the coordinates of the frame of reference are based at least in part on the position of the object at the second time.

AG: Any of the methods of paragraphs Z to AF, wherein the first attribute comprises at least one of: a velocity of the object at the first time, an acceleration of the object at the first time, a position of the object at the first time, a bounding box associated with the object at the first time, a lighting condition associated with the object at the first time, a distance between the object and a map element at the first time, a classification of the object, and a feature associated with the object.

AH: The method of paragraph AG, wherein the object is a first object, the distance is a first distance, and determining that a second object is in proximity to the first object in the environment, the first attribute further comprising a second distance between the first object and the second object at a first time.

AI: A non-transitory computer-readable medium having stored thereon instructions that, when executed, cause one or more processors to perform operations including receiving sensor data representative of an environment; determining, based at least in part on the sensor data, that an object is in the environment; receiving a reference line associated with the object; determining a first attribute associated with the object, the first attribute being associated with a first time; a second attribute associated with the object, the second attribute being associated with a second time after the first time; inputting the first attribute, the second attribute, and the reference line into a machine learning model; and receiving from the machine learning model a predicted position of the object at a third time after the second time, the predicted position being relative to the reference line in the environment.

AJ: The non-transitory computer readable medium of paragraph AI, further comprising: determining that a second object is in proximity to the first object in the environment; determining a third attribute associated with the second object, the third attribute being associated with a first time; determining a fourth attribute associated with the second object, the fourth attribute being associated with a second time; and inputting the third attribute and the fourth attribute into a machine learning model to determine a predicted location associated with the first object.

AK: The non-transitory computer-readable medium of paragraph AI or AJ, wherein the first attribute and the second attribute are expressed with respect to a frame of reference, and the coordinates of the frame of reference are based at least in part on the position of the object at the second time.

AL: A non-transitory computer-readable medium of paragraph AK, in which the predicted position is represented as a distance along a reference line and a lateral offset from the reference line.

AM: The non-transitory computer readable medium of any of paragraphs AI to AL, wherein the first attribute comprises at least one of a velocity of the object at the first time, an acceleration of the object at the first time, a position of the object at the first time, a bounding box associated with the object at the first time, a lighting condition associated with the object at the first time, a distance between the object and a map element at the first time, a classification of the object, or a feature associated with the object.

AN: The non-transitory computer-readable medium of paragraph AM, wherein the object is a first object, the distance is a first distance, and the first attribute further comprises a second distance between the first object and a second object at a first time.

Although the above example terms are described with respect to certain implementations, it should be understood that in the context of this document, the subject matter of the example terms may also be implemented via methods, apparatus, systems, computer-readable media, and/or other implementations.

(Conclusion)
One or more examples of the technology described herein have been described; however, various modifications, additions, permutations, and equivalents thereof fall within the scope of the technology described herein.

In the description of the examples, reference is made to the accompanying drawings, which form a part of this specification, illustrating specific examples of the claimed subject matter. It is understood that other examples may be used and that modifications or substitutions, such as structural changes, may be made. Such examples, modifications or substitutions do not necessarily depart from the intended scope of the claimed subject matter. Although the procedures herein may be presented in a certain order, in some cases the order may be changed such that certain inputs are provided at different times or in a different order without changing the functionality of the described systems and methods. It is also possible to perform the disclosed procedures in a different order. Furthermore, the various calculations described herein need not be performed in the order disclosed, and other examples using alternative orders of calculations may be readily implemented. In addition to changing the order, the calculations may also be decomposed into sub-calculations with the same results.

Claims (14)

1. A method comprising:
receiving sensor data representative of an environment;
determining, based at least in part on the sensor data, that an object is within the environment; and
determining at least one of a pedestrian crossing area or a non-driveable area within the environment;
determining a first attribute associated with the object, the first attribute being associated with a first time;
determining a second attribute associated with the object, the second attribute being associated with a second time after the first time; and
inputting the first attribute, the second attribute, and the crosswalk area or the non-driveable area into a machine learning model , the first attribute and the second attribute being represented by a frame of reference defined at least in part by a predicted destination associated with the object ;
receiving from the machine learning model a predicted position associated with the object at a third time after the second time; and
A method for providing the above. capturing said sensor data using sensors on a vehicle;
controlling the vehicle based at least in part on the predicted position associated with the object at the third time; and
The method of claim 1 further comprising: determining the pedestrian crossing area or the non-traveling area based at least in part on at least one of map data representative of the environment or the sensor data;
determining a threshold area associated with the pedestrian crossing area or the non-travelable area ;
determining a location of the object within the threshold area;
predicting a destination associated with the object based at least in part on the location of the object within the threshold region and at least one of the first attribute or the second attribute;
The method of claim 1 or 2, further comprising: Establishing said frame of reference,
the position of the object at the second time is associated with coordinates in the frame of reference;
a first axis is based at least in part on the coordinate and the pedestrian crossing area or the non-travelable area ;
the second axis being perpendicular to the first axis;
The method of claim 1 , wherein the first attribute is based at least in part on the frame of reference. determining a velocity of the object at the second time; and
determining an angle between a velocity vector representing the velocity and the first axis;
Further equipped with
The method of claim 4 , wherein the second attribute comprises the angle. The predicted location associated with the object at the third time is:
a lateral offset relative to the second axis; and
a distance along the first axis representing the difference between the position of the object at the second time and the predicted position;
The method of claim 4 comprising: The method according to claim 1 , wherein the first attribute and the second attribute represent the number of objects passing through the pedestrian crossing area or the non-travelable area within a given period of time . determining, based at least in part on the sensor data, that other objects are within the environment; and
determining an object context indicative of whether the object is associated with the other object, the object context including at least one of a position, a velocity, or an acceleration associated with the other object ;
determining the predicted position associated with the object at the third time based at least in part further on the object context;
The method of claim 1 , further comprising: The method of any one of claims 1 to 8, further comprising binning at least a portion of the predicted positions to determine binned predicted positions. The first attribute is
a position of the object at the first time;
the velocity of the object at the first time;
a path of the object at the first time;
a first distance between the object and a first portion of the pedestrian crossing area at the first time;
a second distance between the object and a second portion of the pedestrian crossing area at the first time;
the acceleration of the object at the first time;
an indication of whether the object is in a drivable area;
Status of indicators controlling the area
Or object association,
The method according to claim 1 , further comprising at least one of: A computer program product comprising coded instructions which, when executed on a computer, implements the method of any one of claims 1 to 10. 1. A system comprising:
one or more processors;
One or more non-transitory computer-readable media having instructions stored thereon that, when executed, cause the system to perform operations, including:
receiving sensor data representative of an environment;
determining, based at least in part on the sensor data, that an object is within the environment; and
determining at least one of a pedestrian crossing area or a non-driveable area in the environment;
determining a first attribute associated with the object, the first attribute being associated with a first time;
determining a second attribute associated with the object, the second attribute being associated with a second time after the first time; and
inputting the first attribute, the second attribute, and the crosswalk area or the non-driveable area into a machine learning model , the first attribute and the second attribute being represented by a frame of reference defined at least in part by a predicted destination associated with the object ;
receiving from the machine learning model a predicted position associated with the object at a third time after the second time; and
A system equipped with The operation ,
determining the pedestrian crossing area or the non-traveling area based at least in part on at least one of map data representative of the environment or the sensor data representative of the environment;
determining a threshold area associated with the pedestrian crossing area or the non-travelable area ;
determining a position of the object within the environment;
determining that the position of the object is within the threshold region;
determining the pedestrian crossing area or the non-travelable area as a destination associated with the object based at least in part on the location within the threshold area and at least one of the first attribute or the second attribute;
The system of claim 12 further comprising: The operation ,
Establishing said frame of reference,
the position of the object at the second time is associated with coordinates in the frame of reference;
a first axis is based at least in part on the coordinate and the pedestrian crossing area or the non-travelable area ;
the second axis being perpendicular to the first axis;
14. The system of claim 12 or 13, wherein the first attribute is based at least in part on the frame of reference.

Images