Tips for keeping the distribution of weights normal
Quantizing of weights is used very often in implementing neural networks with the tensorflow library on mobile devices. The reason is, the such microcontrollers doesn't have a floating point unit and the developer is trying to increase the overall performance. Quantizing means, to map float values to integer values:

Integer Float
0      -5.0
128    0.0
256    5.0


The problem isn't trivial, because the float point numbers can have a certain minimum, maximum and precision. Maintaining the original distribution is also an issue.

To avoid such kind of problems, the best idea is to implement a floating point library from scratch. In the stackbased Forth language this can be realized in under 400 bytes. It was described in the paper “Willi Stricker: Forth Floating Point Word-Set without Floating Point Stack, 2012”