Why the black box is a major concern in deep learning algorithms?

Deep learning has evolved much in the past few years. From face recognition, self-driving cars to photo editors, election prediction, to fraud detection, its' applications have also diversified to a considerable margin. One of the most prominent use cases of deep learning is computer vision. Computer vision generally employs convolutional neural networks, to recognize and analyze visual inputs to execute follow up actions. However, how neural networks identify objects in images is a mystery: black box problem.

This is mainly because the inner workings of a neural network are shielded from human eyes, making it hard to diagnose errors or biases. For instance, researchers and software developers can download open-source deep neural networks (DNN) tools, like Tensorflow from Google and CNTK from Microsoft, and train them for applications with little to no knowledge of architecture involved. This may create problems in adopting neural networks as they offer less interpretability than traditional machine learning (e.g., decision trees) or artificial intelligence models.

If we have models with interpretable features, it will be much easier to understand the cause of the mistake or bias, or decision made by a neural network model. For instance, if a self-driving car behaves erratically by suddenly turning right every time a person drives it, or a radiologist finds a suspicious area in a medical image, in either case, an explanation is required of how the model arrived at that error. This will not only help figure the bottlenecks but also address them.

Hence the fact that most of these models are notoriously opaque, or act as a black box, raises many ethical questions and creates trust issues. In computer vision, tackling such issues is highly crucial to reduce AI bias and prevent errors. Though a full-fledged fix is still years away, several promising solutions are emerging. These include fine-tuning, unmasking AI, explainable AI (XAI), and more.

Recently, researchers from Duke University have come up with a way to address the black box conundrum. By modifying the reasoning process behind the predictions, it is possible that researchers can better troubleshoot the networks or understand whether they are trustworthy. Their method trains the neural network to show its work processes by demonstrating its understanding along the way, showing which concepts it's employing to make its decision. This approach is different from earlier attempts that focused on what the computer was "looking" at rather than its reasoning following the learning stage itself. For instance, suppose an image of a library is given. The approach makes it possible to determine whether the network layers relied on the representation of "books" to identify it.

Using this new method, the neural network can retain the same accuracy as the original model and show the reasoning processes behind how the results are determined, even with minute adjustments to the network. "It disentangles how different concepts are represented within the layers of the network," says computer science Professor Cynthia Rudin, at Duke University.

According to the Duke University blog, the method controls the way information flows through the network. It involves substituting one standard part of a neural network with a new part. The new part constrains only a single neuron in the network to fire in response to a particular concept that humans understand (like hot or cold, book or bike).  Zhi Chen, a Ph.D. student in Rudin's lab at Duke reveals that by having only one neuron control the information about one concept at a time, it is much easier to understand how the network "thinks."

The researchers stated that the module could be wired into any neural network trained for image recognition, one of the applications of computer vision. In fact, in one experiment, the team connected the solution to a network designed to recognize skin cancer, which had been trained with thousands of images labeled and marked by oncologists. To their surprise, the network had summoned a concept of "irregular borders" without any guidance from the training labels. The system was not annotated with that tag, but the system made its own judgment based on the information it had gathered from its training images.

"Our method revealed a shortcoming in the dataset," Rudin said. She assumes that if they had included this information in the data, it would have made it more evident whether the model was reasoning correctly. Rudin further cites this as a clear illustration of why one should not blindly trust 'black box' models in deep learning.

The team's research appeared on Nature Machine Intelligence, which can be read here.