Through a collaboration with Oxford and Cambridge Universities, and as part of the NeurIPS conference on machine learning (ML), Yandex is to launch the global ‘Shifts Challenge’. The three-pronged competition is designed to tackle the problem of distributional shift in ML and features the largest autonomous vehicle (AV) data set in the industry to date.
Containing 600,000 scenes, equivalent to over 1,600 hours of driving, the data set was collected through self-driving technology tests that were conducted in the USA, Israel and Russia in a range of weather conditions.
Yandex states that overcoming distributional shift is a crucial aspect of training ML models, and that it is essential to build robust models that can operate in all circumstances. This is a prerequisite for models that want to operate in ‘real-life situations’, such as autonomous vehicles roaming city streets. Because of this, competitions such as this one by Yandex are a key tool in accelerating research in this sector.
The ‘Shifts Challenge’ consists of three competition tracks that focus on AV trajectory predication, machine translation and weather forecasting. Alongside the 600,000 scene AV data set, participants on the other two tracks will have access to other high-quality data sets from the Yandex.
Participants on the AV track will be invited to train their motion prediction models on certain types of scenes and subsequently test them in different conditions in different locations for further improvements to be made. Models will then be evaluated by the challenge committee.
Final ranking is to be based on the model’s prediction accuracy alongside its ability to estimate the uncertainty of its predictions in any given case. Uncertainty estimation shows how sure the model is about its decisions. This is as important as the accuracy of predictions made by the models and is crucial for AV technology to be robust and reliable.
“As deep-learning approaches become more powerful, they are being applied in ever more interesting and diverse areas,” commented Mark Gales, who heads up Cambridge University’s collaboration in the Shifts Challenge. “It is increasingly important for these systems to ‘know when they don’t know’, to prevent bad decisions. Through participation in the global Shifts Challenge, researchers have an unprecedented opportunity to evaluate on large-scale, real-world data their models’ ability to measure confidence in their own predictions.”
“The main obstacle to the development of robust models which yield accurate uncertainty estimates is the availability of large, diverse data sets which contain examples of distributional shift from real, industrial tasks,” explained Andrey Malinin, Yandex senior research scientist and Shifts Challenge lead.
“Most research in the area has been done on small image classification data sets with synthetic distributional shift. Unfortunately, promising results on these data sets often don’t generalize to large-scale industrial applications, such as autonomous vehicles.
“We aim to address this issue by releasing a large data set with examples of real distributional shift on tasks which are different from image classification. We hope that this will set the new standard in uncertainty estimation and robustness research.”