Azadeh Dinparastdjadid, a senior research scientist on the Safety Research and Best Practices team at Waymo, discusses how to measure surprising road user behavior, ahead of her presentation on the same theme at the ADAS & Autonomous Vehicle Technology Expo California Conference, which takes place September 20 & 21, 2023, in Santa Clara.
As a member of Waymo’s Safety Research and Best Practices team, Azadeh is currently exploring road user behavior modeling to establish behavioral benchmarks. She earned her PhD and MSc in industrial engineering from the University of Wisconsin-Madison in 2020, and received her MSc in mechanical engineering from the University at Buffalo in 2016.
What is your presentation about?
The significance of expectations and their violations (commonly referred to as ‘surprise’) in the context of road traffic has long been acknowledged. However, despite the important conceptual role surprise plays in traffic safety research, there is no precise quantitative definition or computational model of surprise in this domain. We will demonstrate, for the first time, how computational models of surprise rooted in cognitive science and neuroscience combined with state-of-the-art ML generative models can be used to detect surprising human behavior in complex, dynamic environments like road traffic. In traffic safety, such models can support the identification of traffic conflicts, modeling of road user response time, and driving behavior evaluation for human and autonomous drivers. We also present novel approaches to quantify surprise and use naturalistic driving scenarios to demonstrate a number of advantages over existing surprise measures from the literature.
This sounds fascinating – but doesn’t the very nature of ‘surprise’ make it impossible to model?
The premise of surprise is based on expectation violations. Therefore, to effectively model surprise, you need a generative model capable of producing beliefs that reflect an agent’s expectations. Thanks to modern AI and machine learning, it is now possible to develop behavior prediction models that can generate road user expectations. As discussed in the paper, for surprise in the context of road traffic, we use a generative model called Multipath that was developed by our research team at Waymo, but our framework is generalizable to other domains where a generative model can be trained to generate predictions.
Can you share how your research has influenced Waymo’s automotive driving policy?
As described in our paper, surprise has a variety of applications in traffic safety, including defining traffic conflicts, road user response timing modeling and driving behavior evaluation. For example, surprise is at the heart of Waymo’s paper, Modeling road user response timing in naturalistic settings: a surprise-based framework, which presents a framework Waymo uses for analyzing and modeling response timing in a crash-imminent situation on the road. It helps establish how well an attentive and non-impaired human driver avoids collisions to provide a reference point on the spectrum of human driving on the basis of which the Waymo drivers’ performance can be assessed. The presented response process is modeled as a belief update process driven by perceived violations of this prior belief; that is, by surprising stimuli.
Our proposed models of surprise can be used more broadly to evaluate the quality of driving behavior, for both human and autonomous drivers. These models can be used offline during AV development and as part of the onboard automated driving system itself.
What are some of the biggest surprises and unexpected scenarios you have uncovered so far from Waymo’s own real-world test data?
Nearly every accident on the road involves an element of surprise. Through our data we have observed various surprising behaviors (e.g. running red lights) from other road users including pedestrians, cyclists and other drivers.
Some surprising behaviors on the road can be attributed to factors such as distraction, drowsiness or intoxication. Autonomous vehicles can improve road safety by entirely eliminating these risk factors – they are designed to follow applicable speed limits, they don’t get tired, distracted, drunk or fatigued. The data to date indicates that the Waymo Driver is already reducing traffic injuries and fatalities in cities where we operate.
What is the key message you would like to convey to the audience in Santa Clara?
That surprise is measurable. Our paper introduces a novel generalizable framework on how to measure surprise in real-world complex environments. In the specific context of traffic safety, we believe that surprise has many applications, including but not limited to understanding crash causation and human behavior in critical situations, and evaluating on-road behavior.
In general, to enable wide-scale deployment and societal acceptance of autonomous vehicles, it is of key importance to ensure that they exhibit good behavior and act as a good citizen on the road. At Waymo, we have a conceptual framework called Drivership for how to think about good on-road behavior in the context of AVs. One of its principles is that AVs shouldn’t surprise other users on the road. The metrics we have developed will help pave the way to measure and then further reduce surprise from our driving behavior.
Don’t miss Azadeh Dinparastdjadid’s presentation, which is part of the ‘Strategies, innovations and requirements for the safe deployment of ADAS and autonomous technologies’ session, taking place on Day 1 (Sept 20) of the conference (rates apply), alongside the free-to-attend exhibition at ADAS & Autonomous Vehicle Technology Expo California.