The Fundierung problem of explainable machine learning
To quote Gian-Carlo Rota (Indiscrete Thoughts):<p>When reading a text he notes<p>&quot;I have learned the content of the text by reading the text. But logical hygiene demands that we keep the terms &quot;text&quot; and &quot;content of the text&quot; separate and equal. The text may be an object. The content of the text is not an object in any ordinary sense. Nevertheless, the content is more &quot;important&quot; than the text.&quot;<p>&quot;The distinctness of text and content is undeniable. It may be confirmed by eidetic variations. For example, I may learn the same content by reading another text.&quot;<p>&quot;... the conclusion is inescapable: contents do not exist anywhere, yet it is contents and not brains or texts that matter.&quot;<p>In ML, the training set is &quot;the text&quot; whereas the weights represent &quot;the content&quot;. Having trained a neural net, there doesn&#x27;t appear to be any obvious way to &quot;explain the content&quot;.