Neural Networks unmatched on MNIST?
To quote the relevant Wikipedia article: "The original creators of the database keep a list of some of the methods tested on it.[5] In their original paper, they use a support vector machine to get an error rate of 0.8 percent"

Feel free to look up that original paper, but to me the quote strongly suggests that the first record holder was a support vector machine.

Edit: As liori points out the quote is misleading: In the original paper Yann LeCun et al. actually tried a slew of methods and one version of ConvNet scored best (0.7).  

But according to the MNIST-webpage, after that initial paper, DeCoste and Scholkopf, MLJ 2002, reached an error of 0.56 with a SVM. So if that webpage is complete, the claim is still false.