How to compare security between S-Box and Addition in the block ciphers? I'm studying a block cipher based on ARX(Addition, Rotation, XOR). 

In a block cipher, nonlinearity comes from S-box or modular addition($Z=X\boxplus Y$ mod $2^n$). 

As far as I know, in the case of S-Box, its security can be explained in the algebraic aspect. 

How to explain the modular addition($\boxplus$ mod $2^n$) in the algebraic aspect?
So you have addition with carry bitwise if the sum is smaller than $2^n-1$ 
as well as the subtraction of $2^n$ if the sum is greater than $2^n-1.$

Helger Lipmaa has analysis that is relevant in 
"Efficient Algorithms for Computing Differential Properties of Addition" available at

http://kodu.ut.ee/~lipmaa/papers/lm01/

You can use that work as a starting point.