Who is working on explaining the knowledge encoded into machine learning models? The thing about machine learning (ML) that worries me is that &quot;knowledge&quot; acquired in ML is hidden: we usually can't explain the criteria or methods used by the machine to provide an answer when we ask it a question.
It's as if we asked an expert financial analyst for advice and he/she replied, &quot;Invest in X&quot;; then when we asked &quot;Why?&quot;, the analyst answered, &quot;Because I have a feeling that's the right thing for you to do.&quot;  It makes us dependent on the analyst.
Surely there are some researchers trying to find ways for ML systems to encapsulate and refine their &quot;knowledge&quot; into a form that can then be taught to a human or encoded into a much simpler machine.  Who, if any, are working on that?
I have one concept, which allows to structure knowledge gathered by a ML system in both categorical and algorithms-like structure.

The key idea here is that we contain in our minds some network-like structure in our minds, which helping us not only to classify some data, but also to form an abstract meaning about text or message.

Other than that, in opposite to our minds, it looks like this solution can produce these structures in readable for us forms.