Do goldfish really have a 5 second memory span?
My answer might sound unscientific, since I haven't had tanks filled with goldfish. However, I have had fish and other pets over the years, including cats, turtles and a very sick lizard. I have a scientific background, and on this strength, I will report based on personal observation, albeit with a small sample size.

My fish always "knew" me. When I approached the tank, they knew it was their "feeding person", and even when they were not getting fed, left what they were doing aside and came to the surface to "greet" me, or more likely, impending food. I even experimented by asking others to walk up to the tank and saw that most of them did not respond in a similar fashion, although sometimes they did. With me, I enticed the same reaction every time, even  after a new tank was populated or new additions were made. 

This makes me conclude fish, and goldfish, which I did keep, have "better than expected" memory. 

When they are put in a new tank - well, I suggest you move to a well crowded city that is new to you, and see how long it takes for you to get your orientation right. 

Hope it helps.