Tips for keeping the distribution of weights normal I am working on a project where the Neural Network weights must be quantized on 8 or 16 bits for an embedded platform, thus I will lose some precision.

Since our platform does not have floating point arithmetic we need to quantize the weights. By quantizing i mean taking the max absolute value of the weights and divide it by the maximum signed number representable on 8 or 16 bits. this operation will give us a quantization factor $(qf)$.
the final quantized weights will be integer$(value * qf)$.

If my weights are very sparse and have a very bad distribution, I lose more precision.

For example, to the left here is the distribution of weights for one layer, and to the right is the distribution of weights after I added to the loss function the Kurtosis and skew measures of the weights, and it improved a bit the shape of the distribution while keeping the same accuracy, even a bit higher.

Does anybody have any other suggestions? Has anyone tackled this problem before?
Quantizing of weights is used very often in implementing neural networks with the tensorflow library on mobile devices. The reason is, the such microcontrollers doesn't have a floating point unit and the developer is trying to increase the overall performance. Quantizing means, to map float values to integer values:

Integer Float
0      -5.0
128    0.0
256    5.0


The problem isn't trivial, because the float point numbers can have a certain minimum, maximum and precision. Maintaining the original distribution is also an issue.

To avoid such kind of problems, the best idea is to implement a floating point library from scratch. In the stackbased Forth language this can be realized in under 400 bytes. It was described in the paper “Willi Stricker: Forth Floating Point Word-Set without Floating Point Stack, 2012”