Phantom Faces and Normals
Removing doubles will merge vertices that are close together, in your case you still have duplicate vertices that are further apart than remove duplicates is configured to merge.

You can increase the merge distance used by pressing F6 or looking at the bottom of the toolbar region.



You may also want to switch to vertex select mode to see where the extra vertices are and either delete them manually or move them closer so that remove doubles works, snapping can be used to move them to the same location for merging.