Why the ray tracing result looks weird?
Those are moire patterns. They are  an aliasing artifact that usually occurs when sampling on a regular grid. Did you jitter the positions of your samples? If you just sampled an evenly spaced 10x10 grid within each pixel, that could explain it. Also, numerical errors or inaccuracy could it.