Removing Doubles Destroy Topology
From what I have seen, you should try adjusting the merge distance. This can be found in the bottom of the tools panel after you remove doubles.



This process will adjust the limit for considering something a "double".