In self-driving cars, reliable Convolutional Neural Networks (CNN) are essential, so says Arrk Engineering, a German company specializing in AI validation tools. Using CNNs, artificial intelligence (AI) is supposed to automatically recognize other traffic participants. However, the more autonomously a car drives, the greater the demands on the safety of the algorithms.
In order to protect the human life, a deep understanding of the inner-processes of these neural networks is necessary. However, a CNN operates as a black box per se, so the complex decision paths are difficult to understand and are making it hard to assess safety risks. These challenges can be solved by appropriate visualization. For this purpose, the company has developed a validation tool for analyzing the decision-making processes.
Using interactive visualization, the program enables a deeper insight into each layer of a CNN. The company notes that all weights of the neurons can be manually adjusted to see their impact on the final object recognition. Furthermore, the influences of different confounding factors as well as certain training methods can be easily detected and thus the CNN can be optimized in a later step.
The current methods for analyzing and validating neural networks originate primarily from scientific research. However, Arrk points out that these methods rarely take into account industry-standard functional safety requirements. For example, ISO 26262 and ISO/PAS 21448 require automotive manufacturers to have a much more comprehensive knowledge of the functioning and decision paths of neural networks than has been discussed in scientific discourse to date.
“Therefore, in order to better understand the issues in object recognition by CNNs, we have developed software that enables standardized validation,” explained Václav Diviš, senior engineer ADAS and autonomous driving at Arrk. “During developing this visualization tool, we laid the foundation of a new evaluation method: the so-called Neurons’ Criticality Analysis (NCA).”
Based on this principle, a reliable statement can be made about how important or harmful individual neurons are for correct object recognition.
The interaction of the individual neurons in the numerous layers of a CNN is extremely complex. Each layer and each neuron perform special tasks in the recognition of an object – for example, a rough sorting according to shapes or the filtering of certain colors. However, each step contributes in a different manner to the success of correct object recognition and can, in some cases, even worsen the result. This complexity leads to the fact that the importance of individual neurons for the decision has been inscrutable so far.
To address this, Arrk says it has developed an interactive graphical interface to visualize these paths. “In this way, the decision-making process of a CNN can be visually represented,” Diviš said. “In addition, the relevance of certain neurons to the final decision can be increased, decreased or even eliminated. In real time, the tool immediately determines the impact of these changed parameters after each adjustment has been made. Thus, the importance of certain neurons and their task can be more easily identified and understood.”
Arrk chose the cross-platform programming interface OpenGL to ensure the greatest possible flexibility. This means that the software can be used universally on any device, be it PC, cell phone or tablet. “We also placed particular emphasis on optimizing the calculation and the subsequent graphical display,” noted Diviš. “Therefore, frame per seconds (FPS) in particular were checked in our final benchmark tests. Within this framework, we were able to determine that even when processing a video and using a webcam, the frame rate was stable at around 5fps – even when visualizing 90% of all possible feature maps, which is roughly equivalent to 10,000 textures.”
For teaching the CNN within Arrk Engineering’s visualization tool, the deep learning APIs TensorFlow and Keras are used as a basis, serving as a flexible implementation of all classes and functions in Python. Other external libraries can also be easily connected. Once the neural network has been sufficiently trained, the analysis of critical and anti-critical neurons can begin – the Neurons’ Criticality Analysis.
“For this, we offer modifications such as addition of random noise, the removal of color filters, and the masking of certain user-defined areas,” Diviš said. “Changing these values directly shows how much individual neurons influence the decision in the end. It also reveals which parts of the neural network may be interfering with the overall recognition process.”
With the help of a sophisticated algorithm, the criticality of each individual neuron is automatically calculated. If the value of a neuron is above a certain level, it influences the correct image recognition. The critical threshold can be adjusted as desired. “The final definition of this threshold depends on numerous factors – for example, functional safety requirements, but ethical aspects also play a role here and should not be underestimated,” Diviš added. “Depending on what is desired, this value can be adjusted beyond that, ensuring the greatest possible flexibility of the tool.” As a result, the tool can be adapted to the new requirements at any time in the event of changes in safety regulations.
With its visualization tool, Arrk states that when used in conjunction with the NCA, safety risks in autonomous driving can be reduced through the use of additional mechanisms for plausibility checks. “Our goal is to minimize or better partition the number of critical neurons so that we can rely on robust image recognition of the AI,” Diviš summed up. “So we’re already looking forward to feedback from the field, as this will allow us to further optimize the tool.”