Texture glitch when using simple pixel shader
So I followed Alan Wolfe suggestion (in comment to my question) And turned out he was right. I was using SamplerState.LinearWrap and that was the issue. When I changed this to AnisotropicWrap it looked much better. Below are some examples of different sampling types and how they affect texture:

graphicsDevice.SamplerStates[0] = SamplerState.PointWrap;




graphicsDevice.SamplerStates[0] = SamplerState.LinearWrap;




graphicsDevice.SamplerStates[0] = SamplerState.AnisotropicWrap;