ADAS & Autonomous Vehicle International
  • News
    • A-L
      • ADAS
      • AI & Sensor Fusion
      • Business
      • Connectivity
      • Cybersecurity
      • Expo
      • HMI
      • Last-mile delivery
      • Legislation & Standards
      • Localization/GNSS
    • M-Z
      • Mapping
      • Off-Highway
      • Robo-Taxis
      • Sensors
      • Shared Mobility
      • Safety
      • Simulation
      • Testing
      • Trucks
      • V2X
  • Features
  • Online Magazines
    • January 2025
    • September 2024
    • April 2024
    • January 2024
    • Subscribe
  • Opinion
  • Videos
  • Supplier Spotlight
  • Events
LinkedIn Facebook Twitter
  • Automotive Interiors
  • Automotive Testing
  • Automotive Powertrain
  • Professional Motorsport
  • Tire Technology
  • Media Pack
    • 2026 Media Pack
    • 2025 Media Pack
LinkedIn Facebook
Subscribe
ADAS & Autonomous Vehicle International
  • News
      • ADAS
      • AI & Sensor Fusion
      • Business
      • Connectivity
      • Cybersecurity
      • Expo
      • HMI
      • Last-mile delivery
      • Legislation & Standards
      • Localization/GNSS
      • Mapping
      • Off-Highway
      • Robo-Taxis
      • Sensors
      • Shared Mobility
      • Safety
      • Simulation
      • Testing
      • Trucks
      • V2X
  • Features
  • Online Magazines
    1. April 2025
    2. January 2025
    3. September 2024
    4. April 2024
    5. January 2024
    6. Subscribe
    Featured
    April 15, 2025

    In this Issue – April 2025

    Online Magazines By Web Team
    Recent

    In this Issue – April 2025

    April 15, 2025

    In this Issue – January 2025

    November 29, 2024

    In this Issue – September 2024

    July 23, 2024
  • Opinion
  • Videos
  • Supplier Spotlight
  • Events
  • Awards
    • About
    • 2025 winners
    • Judges
  • Webinars
LinkedIn Facebook
Subscribe
ADAS & Autonomous Vehicle International
AI & Sensor Fusion

Cornell algorithms help self-driving cars learn from own memories

Anthony JamesBy Anthony JamesJune 24, 20225 Mins Read
Carlos Diaz-Ruiz, a doctoral student, drives the data collection car and demonstrates some of the data collection techniques the autonomous vehicle researchers use to create their algorithms (Credit: Ryan Young/Cornell University)
Share
LinkedIn Twitter Facebook Email

Autonomous vehicles that rely on artificial neural networks to navigate the world around them have no memory of the past, unlike humans, and are therefore in a constant state of seeing the world for the first time – no matter how many times they’ve driven down a particular road before. 

This is particularly problematic in adverse weather conditions, when the car cannot safely rely on its sensors, say researchers at the Cornell Ann S. Bowers College of Computing and Information Science and the College of Engineering, who are currently researching how best to overcome this limitation by providing self-driving cars with the ability to create ‘memories’ of previous experiences and use them in future navigation.

Doctoral student Yurong You is lead author of ‘HINDSIGHT is 20/20: Leveraging Past Traversals to Aid 3D Perception,’ which You presented virtually in April at ICLR 2022, the International Conference on Learning Representations [‘Learning representations’ includes deep learning, a kind of machine learning].

“The fundamental question is, can we learn from repeated traversals?” said senior author Kilian Weinberger, professor of computer science at Cornell Bowers CIS. “For example, a car may mistake a weirdly shaped tree for a pedestrian the first time its laser scanner perceives it from a distance, but once it is close enough, the object category will become clear. So the second time you drive past the very same tree, even in fog or snow, you would hope that the car has now learned to recognize it correctly.”

“In reality, you rarely drive a route for the very first time,” added co-author Katie Luo, a doctoral student in the research group. “Either you yourself or someone else has driven it before recently, so it seems only natural to collect that experience and utilize it.”

Spearheaded by doctoral student Carlos Diaz-Ruiz, the group compiled a data set by driving a car equipped with lidar sensors repeatedly along a 15km loop in and around Ithaca, 40 times over an 18-month period. The traversals capture varying environments (highway, urban, campus), weather conditions (sunny, rainy, snowy) and times of day.

This resulting data set – which the group refers to as Ithaca365, and which is the subject of one of the other two papers – has more than 600,000 scenes.

“It deliberately exposes one of the key challenges in self-driving cars: poor weather conditions,” said Diaz-Ruiz, a co-author of the Ithaca365 paper. “If the street is covered by snow, humans can rely on memories, but without memories a neural network is heavily disadvantaged.”

HINDSIGHT is an approach that uses neural networks to compute descriptors of objects as the car passes them. It then compresses these descriptions, which the group has dubbed SQuaSH (Spatial-Quantized Sparse History) features, and stores them on a virtual map, similar to a “memory” stored in a human brain.

The next time the self-driving car traverses the same location, it can query the local SQuaSH database of every lidar point along the route and “remember” what it learned last time. The database is continuously updated and shared across vehicles, thus enriching the information available to perform recognition.

“This information can be added as features to any lidar-based 3D object detector;” You said. “Both the detector and the SQuaSH representation can be trained jointly without any additional supervision, or human annotation, which is time- and labor-intensive.”

While HINDSIGHT still assumes that the artificial neural network is already trained to detect objects and augments it with the capability to create memories, MODEST (Mobile Object Detection with Ephemerality and Self-Training) – the subject of the third publication – goes even further.

Here, the Cornell research paper authors let the car learn the entire perception pipeline from scratch. Initially the artificial neural network in the vehicle has never been exposed to any objects or streets at all. Through multiple traversals of the same route, it can learn what parts of the environment are stationary and which are moving objects. Slowly it teaches itself what constitutes other traffic participants and what is safe to ignore.

The algorithm can then detect these objects reliably – even on roads that were not part of the initial repeated traversals.

The researchers hope that both approaches could drastically reduce the development cost of autonomous vehicles (which currently still relies heavily on costly human annotated data) and make such vehicles more efficient by learning to navigate the locations in which they are used the most.

Both Ithaca365 and MODEST will be presented at the Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2022), to be held June 19-24 in New Orleans.

Other contributors include Mark Campbell, the John A. Mellowes ’60 Professor in Mechanical Engineering in the Sibley School of Mechanical and Aerospace Engineering; assistant professors Bharath Hariharan and Wen Sun, from computer science at Bowers CIS; former postdoctoral researcher Wei-Lun Chao, now an assistant professor of computer science and engineering at Ohio State; and doctoral students Cheng Perng Phoo, Xiangyu Chen and Junan Chen.

The research for all three papers was supported by grants from the National Science Foundation; the Office of Naval Research; and the Semiconductor Research Corporation.

Share. Twitter LinkedIn Facebook Email
Previous ArticleHyundai Mobis develops comprehensive system to monitor drivers’ vital signs
Next Article MIT researchers release open-source photorealistic simulator for autonomous driving

Related Posts

ADAS

Nvidia Drive full-stack autonomous vehicle software rolls out

June 13, 20253 Mins Read
Testing

Tier IV launches autonomous test vehicle development kit

June 13, 20252 Mins Read
Connectivity

NXP and Rimac collaborate on vehicle control architecture

June 13, 20253 Mins Read
Latest News

WeRide collaborates with RTA and Uber to launch pilot operations

June 16, 2025

Aurrigo founder David Keene receives MBE for the decarbonization of airports

June 13, 2025

WATCH NOW: Driving performance, efficiency and reliability – material solutions for vehicle domain controllers

June 13, 2025
FREE WEEKLY E-NEWSLETTER

Receive breaking stories and features in your inbox each week, for free


Enter your email address:


Our Social Channels
  • Facebook
  • LinkedIn
Getting in Touch
  • Free Weekly E-Newsletters
  • Meet the Editors
  • Contact Us
  • Media Pack
    • 2026 Media Pack
    • 2025 Media Pack
RELATED UKI TOPICS
  • Automotive Interiors
  • Automotive Testing
  • Automotive Powertrain
  • Professional Motorsport
  • Tire Technology
  • Media Pack
    • 2026 Media Pack
    • 2025 Media Pack
© 2025 UKi Media & Events a division of UKIP Media & Events Ltd
  • Terms and Conditions
  • Privacy Policy
  • Cookie Policy
  • Notice & Takedown Policy
  • Site FAQs

Type above and press Enter to search. Press Esc to cancel.

We use cookies on our website to give you the most relevant experience by remembering your preferences and repeat visits. By clicking “Accept”, you consent to the use of ALL the cookies.
Cookie settingsACCEPT
Manage consent

Privacy Overview

This website uses cookies to improve your experience while you navigate through the website. Out of these, the cookies that are categorized as necessary are stored on your browser as they are essential for the working of basic functionalities of the website. We also use third-party cookies that help us analyze and understand how you use this website. These cookies will be stored in your browser only with your consent. You also have the option to opt-out of these cookies. But opting out of some of these cookies may affect your browsing experience.
Necessary
Always Enabled

Necessary cookies are absolutely essential for the website to function properly. These cookies ensure basic functionalities and security features of the website, anonymously.

CookieDurationDescription
cookielawinfo-checbox-analytics11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Analytics".
cookielawinfo-checbox-functional11 monthsThe cookie is set by GDPR cookie consent to record the user consent for the cookies in the category "Functional".
cookielawinfo-checbox-others11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Other.
cookielawinfo-checkbox-necessary11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookies is used to store the user consent for the cookies in the category "Necessary".
cookielawinfo-checkbox-performance11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Performance".
viewed_cookie_policy11 monthsThe cookie is set by the GDPR Cookie Consent plugin and is used to store whether or not user has consented to the use of cookies. It does not store any personal data.

Functional

Functional cookies help to perform certain functionalities like sharing the content of the website on social media platforms, collect feedbacks, and other third-party features.

Performance

Performance cookies are used to understand and analyze the key performance indexes of the website which helps in delivering a better user experience for the visitors.

Analytics

Analytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics the number of visitors, bounce rate, traffic source, etc.

Advertisement

Advertisement cookies are used to provide visitors with relevant ads and marketing campaigns. These cookies track visitors across websites and collect information to provide customized ads.

Others

Other uncategorized cookies are those that are being analyzed and have not been classified into a category as yet.

SAVE & ACCEPT