Does breath-holding cause permanent brain damage?
In order to reply to your question I (*) need to dissect it into 3 parts.

1 - Can you voluntarily hold your breath so long that damage actually occurs?
There is evidence that apnoe-divers damage their brains while diving. In the above, the marker protein S100B, representing brain, damage was elevated. This marker had been found previously to be increased in stroke and other forms of brain damage. Therefore, it appears that brain damage occurs when trained freedivers hold their breath on average 335 s (range 281-403 s). I could not find published data on whether this results in cognitive changes. Another really interesting study I found was also on US Navy divers but with oxygen supply. Their cognition was tested and transient alterations in affect, visual focusing, and physical activity were temporarily seen for 10 days after diving. Pressure changes might have an impact on top of the lack of oxygen. 

2 - To summarize a study from a good previous post, researchers measured oxygen saturation in the blood while subject were holding their breath. Oxygen saturation dropped as low as 75% (see original paper). 

3 - Additional studies. A blood oxygen level dependent functional-MRI study in adults and children showed similar activated regions while holding their breath. Breath holding in childrens’ brains causes stronger alterations. Their brains seem to be more vulnerable than adult brains. The insular cortex was identified as a specific region in the brain, being responsible for breathing again after holding breath. These findings originate from a positron-emission study which was able to localize air hunger to the insular cortex. The insula is part of the limbic system which regulates essential tasks in humans including temperature, nausea, and pain. Feeling shortness of breath is important for survival and therefore as shown with PET also located in the limbic system.

(*Note that I'm submitting this well researched answer for Tim, an MD, who took the time to write the text but doesn't have the time to sign up or reach the required reputation limit to post. I only filtered out some medical lingo and streamlined the text. The kudos are his.)