Getting Validation Accuracy of 99% with MNIST with less than 10000 parameters CNN Given MNIST dataset in keras,the challenge is to develop a CNN neural net model with less than 10k parameters with 99% validation accuracy.  


I tried making the model for the same but am getting accuracy as 98.71.
Please find below the code for the same.

    def create_model():
        lay1=Conv2D(2,kernel_size=(1,1),activation='relu',padding='same')(inputs)
        lay1=Conv2D(2,kernel_size=(7,7),strides=(2,2),activation='relu',padding='same')(lay1)
        lay1=Conv2D(2,kernel_size=(1,1),activation='relu',padding='same')(lay1)
        lay1=MaxPooling2D(pool_size=(7,7),strides=(2,2),padding='same')(lay1)
        lay2=Conv2D(4,kernel_size=(1,1),activation='relu',padding='same')(inputs)
        lay2=Conv2D(4,kernel_size=(7,7),strides=(2,2),activation='relu',padding='same')(lay2)
        lay2=Conv2D(4,kernel_size=(1,1),activation='relu',padding='same')(lay2)
        lay2=MaxPooling2D(pool_size=(7,7),strides=(2,2),padding='same')(lay2)
        lay3=Conv2D(6,kernel_size=(1,1),activation='relu',padding='same')(inputs)
        lay3=Conv2D(6,kernel_size=(7,7),strides=(2,2),activation='relu',padding='same')(lay3)
        lay3=Conv2D(6,kernel_size=(1,1),activation='relu',padding='same')(lay3)
        lay3=MaxPooling2D(pool_size=(7,7),strides=(2,2),padding='same')(lay3)
        fc=concatenate([lay1,lay2,lay3])
        fc=Flatten()(fc)
        fc=Dense(10,activation='relu')(fc)
        outputs=Dense(10,activation='softmax')(fc)
        model=Model(input=inputs,output=outputs)
        model.compile(loss=keras.losses.categorical_crossentropy,
          optimizer=keras.optimizers.Adadelta(),
          metrics=['accuracy'])
        return model


The total parameters coming are 8,862 and the batch size used for the above is 32 and the number of epochs are 10.  

Can you please suggest ways to improve the model with the constraints on the number of parameters so that the validation accuracy is 99% or above?
Policy distillation is an excellent way to efficiently reduce number of parameters and maintain the quality of the original (large) network. 

According to the paper, while having orders of magnitude less numbers, distilled models can actually improve score from 84% up to 124% and above.

The basic idea is that we train a new, smaller network from scratch, to predict what the original (large) network would produce.

An analogy would be: it's a lot easier to assign the desired mindset to a young child, than to update an old person's way of thinking.