The hippocampus plays a crucial role within the neural systems for long-term memory, but little if any role in the short-term retention of some types of stimuli. Nonetheless, the hippocampus may be specialized for allocentric topographical processing, which impacts on short-term memory or even perception. To investigate this we developed performance-matched tests of perception (match-to-sample) and short-term memory (2 s delayed-match-to-sample) for the topography and for the nonspatial aspects of visual scenes. Four patients with focal hippocampal damage and one with more extensive damage, including right parahippocampal gyrus, were tested. All five patients showed impaired topographical memory and spared nonspatial processing in both memory and perception. Topographical perception was profoundly impaired in the patient with parahippocampal damage, mildly impaired in two of the hippocampal cases, and clearly preserved in the other two hippocampal cases (including one with dense amnesia). Our results suggest that the hippocampus supports allocentric topographical processing that is indispensable when appropriately tested after even very short delays, while the presence of the sample scene can allow successful topographical perception without it, possibly via a less flexible parahippocampal representation.