How to avoid round problem in tax? I have to deal with order with high quantities by product. 

Tax (VAT) is calculated by rule for unit price, rounded, then multiplied by quantity. With high quantity this kind of rounding cause problem.

How to deal with that?
You can specify the round mode you want in the tax type configuration page.

/admin/commerce/config/taxes/types/vat/edit

(It should be configure to no round by default)