Measuring Information Transfer in Neural Networks
share
Estimation of the information content in a neural network model can be prohibitive, because of difficulty in finding an optimal codelength of the model. We propose to use a surrogate measure to bypass directly estimating model information. The proposed Information Transfer (L_IT) is a measure of model information based on prequential coding. L_IT is theoretically connected to model information, and is consistently correlated with model information in experiments. We show that L_IT can be used as a measure of generalizable knowledge in a model or a dataset. Therefore, L_IT can serve as an analytical tool in deep learning. We apply L_IT to compare and dissect information in datasets, evaluate representation models in transfer learning, and analyze catastrophic forgetting and continual learning algorithms. L_IT provides an informational perspective which helps us discover new insights into neural network learning.
READ FULL TEXT