The Bias-Variance Trade-Off
One way to look at this is through the idea of under-/overfitting

First off, here is a sketch of the generally observed relationship between bias and variance, in the context of model size/comlpexity:



Say you have a model which is learning quite well, but your test accuracy seems to be pretty low: 80%. The model is essentially not doing a great job of mapping input features to outputs. We have a high bias. But for a wide variety of input (assuming a good test set), we consistently obtain this 20% error; we have a low variance. We are underfitting

Now we decide to use a bigger model (e.g. a deep neural network), which is able to capture more details of the feature space and so maps inputs to outputs more accurately. We now have an improved test accuracy: 95%. At the same time, we notice that several runs of the model produce different results; sometime we have 4% error, and sometimes 6%. We have introduced a higher amount of variance. We are perhaps somewhere around the optimum model complexity shown on the graph above.

You say ok... let's create a monolithic neural network. It totally nails training and ends with a perfect accuracy: 100%. However, the test accuracy now drops to 90%! So we have zero bias, but a large variance.
We are overfitting. The model is almost as good as a look-up table for training data, but doesn't generalise at all when it sees new samples. Intuitively, that 10% error corresponds to a difference in distribution between the training and test sets used $\rightarrow$ the model knows the training distribution in extreme detail, some of which do not apply to the test set (i.e. the reality). 

In summary: The bias tends to decrease faster than the variance increases, because you can likely still make a more competitive model for your dataset; the model is underfitting. It is like the low-hanging fruit that you can easily get - so an incremental improvement on the red curve above gives a big decrease in bias (increase in performance). Obviously that pattern cannot go on indefinitely, with each increment in model complexity, you get a lower increase in performance; i.e. you have diminishing returns. Furthermore, as you begin to overfit, the model becomes less able to generalise and so exhibits larger errors on unseen data; variance is creeping in.



For some more intuition between bias/variance in machine learning, I'd recommend this talk by Andrew Ng. There is also a text summary of the talk, for a quicker overview.

For a brief but more mathematical explaination, head over to this post of Cross-Validated. The second answer there is very recent and is perhaps better than the (old) accepted answer.