For autonomous vehicle simulation, it is increasingly important for developers to design systems that can generate highly detailed, simulated worlds that the artificial intelligence (AI) driver perceives as real. It is also important to create large simulations that cover all the aspects on which the AI driver needs to be trained and tested.
To ensure AI drivers and systems meet the requirements for on-road use, Nvidia has created the Neural Reconstruction Engine, a new AI toolset for the Nvidia Drive Sim simulation platform, which uses AI networks to turn recorded video data into simulation.
Nvidia’s solution utilizes AI to automatically extract the key components needed for simulation, which include the environment, 3D assets and scenarios. Once extracted, the individual pieces are reconstructed into simulations that have an extremely high level of realism but are now reactive and can be changed as engineers see fit. The solution streamlines the process of creating simulated worlds, making it more cost-effective when using the toolset.
Real-world 2D video data is converted by the AI pipeline into a 3D digital twin that can then be loaded into Nvidia’s Drive Sim. Other 3D assets can also be reconstructed and placed into a library of assets for use in other simulations.
Just like in the real world, a multitude of different scenarios take place during a simulation in an environment combined with assets. Nvidia’s new Neural Reconstruction Engine works to assign AI-based behaviors to actors within the scene, so when these events or scenarios occur, the actors behave just like they did in the real drive. Additionally, because of the AI behavior model, figures in the simulation can respond and react to scene, element or vehicle changes. Furthermore, because of the scenarios occurring in a simulated world, developers can change the time and location of events, in addition to incorporating new elements.
Within Drive Sim, developers can also adjust dynamic and static objects, a vehicle’s path and the location, orientation and parameters of vehicle sensors.
To further improve and train perception systems, the same scenes in Drive Sim can be used to generate pre-labeled synthetic data, and by using real-world data to build scenes, the sim-to-real gap is reduced.
The Neural Reconstruction Engine will be integrated into future releases of Drive Sim, and enables both physics-based and neural-driven simulation on the same cloud-based platform.