Neural Activation Patterns (NAPs): Visual Explainability of Learned Concepts. (arXiv:2206.10611v1 [cs.LG])

Web: http://arxiv.org/abs/2206.10611

A key to deciphering the inner workings of neural networks is understanding
what a model has learned. Promising methods for discovering learned features
are based on analyzing activation values, whereby current techniques focus on
analyzing high activation values to reveal interesting features on a neuron
level. However, analyzing high activation values limits layer-level concept
discovery. We present a method that instead takes into account the entire
activation distribution. By extracting similar activation profiles within the
high-dimensional activation space of a …

arxiv explainability lg neural patterns