What is the mechanism behind cats' geolocating homing behavior?
It is explained with path integration in behavioral neuroscience. Not only cats, but other mammals, birds and even insects use path integration to return to a starting point.

Here is a relevant excerpt from the book Beyond the Cognitive Map: From Place Cells to Episodic Memory (by A. David Redish):


  Path integration is the ability to return directly to a starting
  point (sometimes called a home base or reference point) from
  any location in an environment, even in the dark or after a
  long circuitous route (Barlow, 1964; Gallistel, 1990; Maurer and
  Seguinot, 1995). Sometimes called dead reckoning, this ability
  has been shown in gerbils (Mittelstaedt and Mittelstaedt, 1980;
  Mittelstaedt and Glasauer, 1991), hamsters (Etienne, 1987, 1992;
  Chapuis and Scardigli, 1993), house mice (Alyan and Jander,
  1994), rats (Tolman, 1948; Alyan et al., 1997; Whishaw and
  Maaswinkel, 1997), birds (Mittelstaedt and Mittelstaedt, 1982;
  von Saint Paul, 1982), and even insects (Wehner and Srinivasan,
  1981) and arthropods (Mittelstaedt, 1983), as well as dogs, cats,
  and humans (Beritashvili, 1965).
  
  Path integration in animals has been the subject of argument
  for more than a century, including a notable debate in 1873 between
  Alfred Wallace and Charles Darwin in which Wallace suggested
  that animals find their way back via sequences of smells
  and Darwin argued that animals must be using dead reckoning
  (see Wallace, 1873a, 1873b; Darwin, 1873a, 1873b; Nature, 1873;
  Forde, 1873; Murphy, 1873). The carefully controlled experiments
  of Mittelstaedt and Mittelstaedt (1980) and Etienne (1987) have
  demonstrated conclusively that this ability is a consequence of integrating
  internal cues from vestibular signals and motor efferent
  copy.


emphasis mine

Cats, dogs and rodents can use taxon navigation as well along with path integration. Here is a relevant excerpt from Neural compass or epiphenomenon?
Experimental and theoretical investigations into the
rodent head direction system (by Matthijs van der Meer):


  “Internal allocentric” navigation. This type of navigation relies on a mapping of praxic commands to an allocentric spatial representation, allowing the animal to make a direct return to a home base after a complex outward path in the absence of cues, an ability referred to as path integration, discussed in detail in the next section. Returning by following an odour trail back, or navigating to a cue indicating the home base would be taxon navigation, but behavioural experiments have provided convincing evidence that rodents are able
  to do this without using external cues. As mentioned above, this ability requires some mechanism of continuously updating at least one’s directional heading (a “homing vector”) relative to the home base. This is an allocentric representation which does not need to be related to any external cues, unlike the next class of strategies; the reference point or direction can in principle be set to any location the animal desires.


emphasis mine

Further reading:


A connectionist model of path integration with and without a representation
of distance to the starting point (ROLAND MAURER, University of Geneva, Geneva, Switzerland) - Copyright 1998 Psychonomic Society, Inc.