How much could Earth's temperature increase by the time we run out of oil reserves?
That estimation is based on current extraction rates and based in 2013, so we would have oil reserves until 2066.
Assuming that's correct, it would resemble the "business as usual" future emissions scenario, a.k.a. RCP8.5 (more about RPCs).

According to the last IPCC report, the range of mean global temperature predictions for RCP8.5 looks like this:



We are now 0.5°C above the figure's baseline, and in the next 80 years, which is about year 2100 the temperature would be about 3.5°C above current temperature. But the range spanned by different predictions goes from 3°C to 5.5°C.

Note that the warming effect of putting additional $\text{CO}_2$ in the atmosphere last many centuries, so even if emissions stop suddenly bu year 2066, the planet will keep getting hotter and hotter for a long time.

Also note that the warming is not homogeneous, some areas will warm much more than others, in RCP8.5 scenario many areas would warm by more than 6°C, this animation by NOAA gives you a nice idea of the rate and pattern of warming.

And regarding your question:  Is that a problem to really be worried about? the answer is definitely YES! The RCP8.5 scenario is the most pessimistic one, if that one realizes we will be in a lot of trouble, dealing with climate change, weather extremes, sea level rise and massive migrations.