Why is having low variance important in offline policy evaluation of reinforcement learning?
Having low variance is important in general as it reduces the number of samples needed to obtain accurate estimates. This is the case for all statistical machine learning, not just reinforcement learning.

In general, if you are estimating a mean or expected quantity by taking many samples, the variation in the error is proportional to $\frac{\sigma}{\sqrt{N}}$ for a direct arithemtic mean of all samples, and behaves similarly for other averaging approaches (such as recency-weighted means using a learning rate). The bounds on accuracy can be made better by either increasing $N$ i.e. taking more samples, or by decreasing the variance $\sigma^2$.

So anything you can do to reduce variance in your measurements has a direct consequence of reducing the number of samples required to achieve the same degree of accuracy.

In the case of off-policy reinforcement learning, there is added variance - compared to on-policy learning - due to different probabilities of taking an action in behaviour and target policies. This is due to the need to adjust reward signals using importance sampling - multiplying by the importance sampling ratio will make the reward signal vary more (in fact it can become unbounded). This is not really any more of a challenge than any other source of variance, but as it interferes with the goal of speedy learning, a lot of research effort has been put into methods that reduce the variance.