Systemic sensory confusion?
When our brain works out where a part of our body is in relation to another part of our body, or in relation to anything for that matter, we combine:


visual information
information about how much our muscles are contracted
how much our joints are contracted
different forces being applied on our arms/legs or other body parts
where it remembers we used to be
how much force we applied to move where we wanted to move
where we would therefore expect to be 
and many other things


That's quite a few things that our body relies on! The one that's most fine tuned is our vision. Let's face it, whatever we are actively doing with our hands requires the most skill and knowledge of where our hands etc. are. It is also the system that we rely on the most. But other systems are helpful to make sure we don't fall, grabbing on to things and making sure we reflexively contract the opposite muscles to prevent falling, stop us dropping things and lot's more. It doesn't matter if they're not accurate, they just need to be fast and in doing so they can't be extremely exact. 

When we don't use vision, we rely on these other things. They're made to quickly adapt (e.g. hold a shopping bag for a while and when you let go your arm feels light or when you get out of a swimming pool your body feels heavy) and it is important that they do this to allow their functions as listed in the last paragraph.