Interpretability for Neural Networks from the Perspective of Probability Density

Currently, most of works about interpretation of neural networks are to visually explain the features learned by hidden layers. This paper explores the relationship between the input units and the output units of neural network from the perspective of probability density. For classification problems, it shows that the probability density function (PDF) of the output unit can be expressed as a mixture of three Gaussian density functions whose mean and variance are related to the information of the input units, under the assumption that the input units are independent of each other and obey a Gaussian distribution. The experimental results show that the theoretical distribution of the output unit is basically consistent with the actual distribution.