Quite often I would come across conversations similar to the one below:

"Is it a difficult test?"

"Not for you it isn't."

So it's commonplace to hear constructions such as "Not for you it isn't" and "Not to me it doesn't" etc. But from my standpoint it seems like a case of double negative, as there are more than one negative element/form involved. Should I rephrase it to "For you it isn't" ? Or is it that it has become idiomatic and removing one negative element would sound clumsy in comparison?

Many thanks in advance.