What is the relation of the transition band's width and the filter order for the FIR windowing method
There are only heuristic formulas for estimating the filter order. For a Kaiser window (which is probably the most frequently used window for filter design) the required filter order can be estimated from [1]

$$M=\frac{A-8}{2.285\,\Delta\omega}\tag{1}$$

where $A=-20\log\delta$ ($\delta$ is the maximum deviation from the desired response), and $\Delta\omega$ is the (smallest) transition bandwidth. This formula is of course only valid for the approximation of ideal frequency selective filters (low pass, high pass, etc.). This formula is implemented in Matlab's kaiserord.m function.

[1] Oppenheim, Schafer, Buck, Discrete-Time Signal Processing, 2nd ed., p. 476