How to determine Hilbert coefficients
The resulting analytic signal after using a the Hilbert transform filter is ideally a signal without any negative frequencies, hence, attenuating -1 (using normalized frequency) to 0 while transmitting 0 to 1 (note that this is a complex output and therefore one do not have the standard symmetry around 0). This is obtained by phase-shifting positive frequencies pi/2 and negative frequencies -pi/2 while keeping the magnitude to be one and adding the output with the original signal.

Since it is not possible to obtain this change in phase-shift without a transition band one will need to specify how wide the transition band should be.. Hence, in your case the 0.1 would indicate that your designed filter will perform well from 0.05 to 0.95 with a transition band from -0.05 to 0.05 and similarly from 0.95 to 1.05.

Increasing the order with constant transition band will produce a higher attenuation/smaller approximation error in the passband, while decreasing the transition band with the same order will have the opposite effect.

Also, if you will implement it on an FPGA I would suggest multiplying the coefficients with a power of two, e.g. 8192 to get a better range for the output and use of the bits.