Optimization of Filter Coefficients to Given Word Length
It is definitely a difficult mathematical problem (integer programming), but I don't see much research going on in the DSP community. I do not know about the maths community. There were some important papers in the early 1980's on the optimum design of FIR filters with quantized coefficients:


D. M. Kodek, "Design of Optimal Finite Wordlength FIR Digital Filters Using
Integer Programming Techniques," IEEE Trans. ASSP 28, 304-308 (1980).
D. M. Kodek and K. Steiglitz, "Comparison of Optimal and Local Search Methods
for Designing Finite Worldlength FIR Digital Filters," IEEE Trans. Circuits Systems 28, 28-32 (1981).


But as far as I know none of the published methods developed into a standard design method for filters with quantized coefficients, unlike for the design of filters with infinite precision coefficients (e.g., Parks McClellan).

I think there are two reasons for that: first, optimum methods use integer programming techniques, which are computationally inefficient, and which will only compute a local optimum that can be arbitrarily bad. These problems become aggravated for large filter lengths. Second, there is not so much need anymore for the optimum design of filters with quantized coefficients. This has to do with the fact that more and more systems are implemented in floating point, and - if fixed-point is needed - the word length is usually large enough such that suboptimal heuristic design methods are sufficient. Such methods iteratively design the filter assuming infinite precision, quantize the coefficients, check the filter properties and, if necessary, adapt the specs (including the filter length, error weighting, etc.), re-design the filter according to the new specifications, and so on, until the requirements are met.

For IIR filters, the optimum design for quantized filter coefficients is even more difficult than for FIR filters due to the rational transfer function. I haven't seen any useful optimization methods in the DSP literature, only heuristic methods such as the one described above. Note that different filter structures will result in different possible pole locations in the complex plane after coefficient quantization. The possible pole locations for the direct-form structure are relatively sparse at small pole angles, i.e., for low cut-off frequencies, whereas the pole locations of the coupled-form structure are evenly distributed on a rectangular grid.