How does the brain know whether or not it comprehends a novel concept? There seem to be at least two kinds of confusion regarding novel concepts.

In one, the brain simply can't form an abstract model from whatever information is being presented. It's where you can't "wrap your head around" something.

In another, the brain perceives a cognitive dissonance, where something doesn't "add up", but further information provides a unifying model that can explain everything being presented.

They seem to be related.

The brain is somehow able to evaluate the fidelity of an abstract model that it's trying to construct. How does this happen?
Interesting question!  A related phenomenon called the illusion of explanatory depth (IOED) suggests that the human cognitive system has a systematic weakness in this kind of evaluation--I believe the classic example is asking people if they know how a helicopter works (most people say yes), and then asking them to explain how a helicopter works (very few people even get as far as mentioning airfoils or lift).

Although the mechanism behind the illusion of explanatory depth (and thus the mechanism behind evaluation of incomprehension in general) is understudied, a 2010 review examined six studies and drew a few conclusions: 


IOEDs have been found in natural, mechanical, and social-psychological domains.
People who adopt a concrete construal style (as opposed to an abstract style), either naturally or from prompting, diminish the scope of their IOED. 


Missing the trees for the forest: A construal level account of the illusion of explanatory depth. Alter, Adam L.; Oppenheimer, Daniel M.; Zemla, Jeffrey C. Journal of Personality and Social Psychology, Vol 99(3), Sep 2010, 436-451. http://dx.doi.org/10.1037/a0020218