Ontology Concept Extraction Algorithm for Deep Neural Networks

Abstract

An important drawback of deep neural networks limiting their application in critical tasks is the lack of explainability. Recently, several methods have been proposed to explain and interpret the results obtained by deep neural networks, however, the majority of these methods are targeted mostly at AI experts. Ontology-based explanation techniques seem promising, as they can be used to form explanations using domain terms (corresponding to ontology concepts) and logical statements, which is more understandable by domain experts. Recently, it has been shown, that inner representations (layer activations) of deep neural network can often be aligned with ontology concepts. However, not every concept can be matched with the output of every layer, and it can be computationally hard to identify the particular layer that can be easily aligned with the given concept, which is aggravated by the number of concepts in a typical ontology. The paper proposes an algorithm to address this problem. For each ontology concept it helps to identify neural network layer, which produces output that can be best aligned with the given concept. These connections can then be used to identify all the ontology concepts relevant to the sample and explain the network output in a user-friendly way.