Shane Soh

Anticipating the future actions of a human is a widely studied problem in robotics that requires spatio-temporal reasoning. In this work we propose a deep learning approach for anticipation in sensory-rich robotics applications. We introduce a sensory-fusion architecture which jointly learns to anticipate and fuse information from multiple sensory streams. Our architecture consists of Recurrent Neural Networks (RNNs) that use Long Short-Term Memory (LSTM) units to capture long temporal… Show more

Anticipating the future actions of a human is a widely studied problem in robotics that requires spatio-temporal reasoning. In this work we propose a deep learning approach for anticipation in sensory-rich robotics applications. We introduce a sensory-fusion architecture which jointly learns to anticipate and fuse information from multiple sensory streams. Our architecture consists of Recurrent Neural Networks (RNNs) that use Long Short-Term Memory (LSTM) units to capture long temporal dependencies. We train our architecture in a sequence-to-sequence prediction manner, and it explicitly learns to predict the future given only a partial temporal context. We further introduce a novel loss layer for anticipation which prevents over-fitting and encourages early anticipation. We use our architecture to anticipate driving maneuvers several seconds before they happen on a natural driving data set of 1180 miles. The context for maneuver anticipation comes from multiple sensors installed on the vehicle. Our approach shows significant improvement over the state-of-the-art in maneuver anticipation by increasing the precision from 77.4% to 90.5% and recall from 71.2% to 87.4%. Show less

In this work we anticipate driving maneuvers several seconds before they occur. For this purpose we built a vehicular-sensory platform with multiple cameras, tactile sensors, GPS, and a computing device. Our sensors capture the driving context from both inside and outside of the car. We introduce a sensory-fusion Recurrent Neural Network (RNN) architecture which jointly learns to anticipate and fuse information from multiple sensory streams. Our architecture use Long-Short… Show more

In this work we anticipate driving maneuvers several seconds before they occur. For this purpose we built a vehicular-sensory platform with multiple cameras, tactile sensors, GPS, and a computing device. Our sensors capture the driving context from both inside and outside of the car. We introduce a sensory-fusion Recurrent Neural Network (RNN) architecture which jointly learns to anticipate and fuse information from multiple sensory streams. Our architecture use Long-Short Term Memory (LSTM) units to capture long temporal dependencies.

We evaluate our approach on a diverse data set with 1180 miles of natural free-way and city driving and show that we can anticipate maneuvers 3.5 seconds before they occur with over 80% F1-score in real-time. Show less

Advanced Driver Assistance Systems (ADAS) have made driving safer over the last decade. They prepare vehicles for unsafe road conditions and alert drivers if they perform a dangerous maneuver. In this work we anticipate driving maneuvers a few seconds before they occur. For this purpose we equip a car with cameras and a computing device to capture the driving context from both inside and outside of the car. We propose an Autoregressive Input-Output HMM to model the contextual information along… Show more

Advanced Driver Assistance Systems (ADAS) have made driving safer over the last decade. They prepare vehicles for unsafe road conditions and alert drivers if they perform a dangerous maneuver. In this work we anticipate driving maneuvers a few seconds before they occur. For this purpose we equip a car with cameras and a computing device to capture the driving context from both inside and outside of the car. We propose an Autoregressive Input-Output HMM to model the contextual information along with the maneuvers. We evaluate our approach on a diverse data set with 1180 miles of natural freeway and city driving and show that we can anticipate maneuvers 3.5 seconds before they occur with over 80% F1-score in real-time. Show less