Keras cNN Transfer Model: Reduce Final Model Size?
There's actually some pretty interesting research on this topic. Key words would be model compression or CNN pruning (doing things like reducing the model size by removing filters with low activations). 

Another interesting method uses a technique called teacher-student networks. Basically, this involves training the big network (the teacher) and using the output of the teacher as the labels when you train the small student network. This usually serves to improve the student over training with the ground truth.

For example, if you were training on MNIST, you'd have 10 classes and a softmax output of the teacher network that might look like [0, 0, 0, 0, 0.33, 0, 0, 0, 0, 0.67], in the case of a 9 that looks like a 4, when the original ground truth was [0, 0, 0, 0, 0, 0, 0, 0, 0, 1].  

I'm not totally current on this research so give these a google first, there might be some better best practices out there. Hope this at least gets you going in the right direction.