How can I challenge a professor who is ignoring me in an online course?
I would be polite but persistent. 

It may be the case that the online test system auto-grades answers. The instructor may have to go into the system to regrade your answers, and this may be causing the delay in the instructor's response. So, make sure your email contains all the needed information to understand what you answered and why, as well as the extent to which this affected your grade (X answers were marked as wrong for the the same reason). And try to be a bit patient before emailing again (a couple of days).

Is there a discussion of significant digits in your text? Often intro texts have something like this in an early chapter or an appendix. If so, you can point out that part of the book and say that is why you thought rounding to three significant digits was correct. 

In your communication, I would avoid saying you think your answer is better than that of your professor. I would instead try to emphasize that the book led you to believe that your answer was also correct.