It seems like the math of arduino is off
If your program is basically working but gives you rounding errors, it can be that you are facing some problems related to conversion from floating-point to binary or reverse. It is a very common problem in computer science. For example, the decimal number 0.1 has no exact binary  representation, it is infinite. For more information, you can read these pages:

http://www.exploringbinary.com/why-0-point-1-does-not-exist-in-floating-point/
https://www3.ntu.edu.sg/home/ehchua/programming/java/datarepresentation.html

In order to solve your problem, you can try to apply appropriate rounding to your results and if possible avoid mixing different data types and use double only if absolutely needed (which I guess is the case in your application).