Compressibility factor of hydrogen at low pressure and constant temperature
In my textbook the opposite meaning is given to $a$ (an attractive parameter) and $b$ (a size parameter), but this may be a matter of differences in labelling, not in meaning. I will for the sake of consistency stick to your convention. For $\ce{He}$ the van der Waals parameters have the following values: 


$b = \pu{3.4598 J mol-1 M-1}$
$a = \pu{0.023733 M-1}$


As you can see from the following figure at $RT$ ($\pu{298K}$), while your first approximation is quite accurate for $\ce{He}$ already at $V_\mathrm m \approx \pu{1 L/mol}$, your second approximation fails until higher  $V_\mathrm m$. It is negative, but negliglibly so, and at that point is not particularly useful as an approximation as already $Z \approx 1$. 



The approximation fails (at RT) because the condition $b/RT \gt a$ is never observed:
$$\frac{b}{RT} = \pu{0.0014 M-1}$$
whereas
$$a = \pu{0.0237 M-1}$$

You can solve for the temperature at which these two become equal:

$$T_b = \frac{b}{aR}$$

For $\ce{He}$, $T_b = 17.53\ \mathrm K$. Below $T_b$ you may observe $Z&lt;1$.

Sure enough, below that $T$ your second approximation starts to hold, as shown here for $T = 10\ \mathrm K$:



or in terms of $P$:



Note: it is not clear that the virial equation and van der Waals parameters should capture the behavior of the gases over a wide range of $P$ or $V_m$ at such low $T$.