Getting Validation Accuracy of 99% with MNIST with less than 10000 parameters CNN
Policy distillation is an excellent way to efficiently reduce number of parameters and maintain the quality of the original (large) network. 

According to the paper, while having orders of magnitude less numbers, distilled models can actually improve score from 84% up to 124% and above.

The basic idea is that we train a new, smaller network from scratch, to predict what the original (large) network would produce.

An analogy would be: it's a lot easier to assign the desired mindset to a young child, than to update an old person's way of thinking.