How long would it take for all the Bitcoin Key pairs to be calculated? Is this even something one should be worried about?
No, you shouldn't worry about it.

3Blue1Brown has made a great video explaining how large 256 bit is, check it out