Neural Networks: Simulate the working of Dynamic Fixed Point representation of the weights on hardware I am looking to implement a neural network on hardware using Verilog. I have completed and tested with floating point representation and a 20 bit fixed point representation. I want to further reduce the memory utilisation by using dynamic fixed point. I want to identify the less important weights, and assign them a smaller bit-width, while keeping the more important weights with a higher bit-length.
How can I identify the more/less important weights?
What I can think of is, reducing bit-width for all the small weights [-0.001, 0.001]. But isn't that like pruning?
Thanks!
Start by reading the research literature on this topic.  There's a large number of published papers that discuss how to quantize the weights or intermediate values used in deep learning.  For instance, here are a few papers:


https://arxiv.org/abs/1412.7024, https://arxiv.org/abs/1605.06402, https://arxiv.org/abs/1612.01064, https://arxiv.org/abs/1701.08978


Start by reading them, and do a literature search to identify other relevant papers (e.g., read their related work section and find other earlier papers that are relevant; find papers that cite them using Google Scholar and read them; repeat).

You might also look at the Ristretto tool and this project.