How do I find the number of inversions using a red black tree?
What you are trying to do is correct. However only number of rotations will not be able to tell you anything about the inversions. The whole procedure is as follows. 

You start with an empty red-black tree. Then you insert the elements in the given sequence one by one, in the order that they occur in the sequence. 

Whenever an element $x$ in the sequence is inserted, the elements larger than $x$ already existing in the red-black tree are inversions. You need to calculate this number.

You total up all the inversions for all elements and you get the total number of inversions.

You need to augment the red-black tree by keeping the size of sub-trees in the interior nodes. Then you can find the number of elements larger than $x$ either during insertion or after insertions by doing a search of element $x$.

Since you are confused about the strategy, I suggest you should use the second method. Don't count the larger elements during the insertion stage. Count them immediately after the insertion is done and before another insertion is to be done (in a separate function). This will take additional  $O(\log n)$ on the augmented tree.

If all your numbers are distinct then this algorithm suffices. However, if some numbers are same, then you have to modify your insertion to insert only at the rightmost possible position and/or counting only the strictly larger elements depending on your algorithm. You will need to further augment the data structure to hold the min-value and max-value of the sub-tree in the node.