Machine learning has achieved great success in many applications. However, one of the key problems with machine learning algorithms is that their output is not always comprehensible to humans. Especially neural networks, which are now so popular, output probability estimates that supposedly communicate the certainty in a prediction. These probability estimates however are highly misleading and may well be the key reason why neural networks are considered so noninterpretable. If for example a neural network outputs that an image contains a specific object with 90% probability this does not mean that it is right 90% of the time.

In this project, we have investigated how neural networks can be calibrated so that their outputs are "truthful" and really do match reality and human expectations. We developed and compared metrics to investigate how uncalibrated neural networks are and then investigated various approaches to recalibrate neural networks. Ultimately we found that this can be done with pretty high accuracy most of the time.

Another key question that arises in the context of interpretable neural networks is what kind of feature spaces deep neural networks learn. There is a general misconception that neural networks are very accurate, however that their intermediate representations are a "black box". We decided to investigate this thoroughly and conjectured that the intermediate features representations of convolutional neural networks do indeed learn a linearized version of the input image manifold. Proving this conjecture is very hard and has so far been elusive. However, we managed to come up with an experimental set up that provides insights into whether this claim is true or not. If neural networks really learn a linearized version of the manifold, then traversals along the manifold must become linear in feature space. For example, if you have a picture of a man without facial hair, adding facial hair would move that image towards a completely different location in the image manifold. Similarly, aging people moves images from areas of the manifold populated by images of young people towards areas populated by images of older people. These transformations are highly nonlinear if performed in pixel space, however we managed to show that if you perform these transformations in deep feature space they reduce to a simple linear translation. By reconstructing the manipulated image we can show that the final outcome is indeed on the image manifold.

As a key area of application, we have applied our interpretable learning techniques to the medical domain. We have collaborated with the School of Medicine at Washington University in St. Louis and worked on predicting hospital readmissions for hospitalized patients. For this task, we need not only accurate predictions, but also explanations on potential risks. We have developed such a model that can be used by physicians.

For broader impacts, we have disseminated our results in conferences, seminars, and workshops. Our research effort to understand the internal workings of neural networks has led us to designs of much more compact in your network architectures. These especially suitable for mobile applications. It is therefore highly likely that our research will have a significant impact through technology transfer. We have trained graduate and undergraduate students, and integrated research into the teaching of our graduate courses.

Last Modified: 04/25/2020
Modified by: Yixin Chen