How do animals recognize other animals of their own species? My motivation for this question is dog-based, but I suppose it would apply equally well to humans. How do animals recognize their own kind, particularly where there is large variation in appearance?

The background is that I am reading "Discrimination of human and dog faces and inversion responses in domestic dogs (Canis familiaris)" by Racca et al. (2010; Animal Cognition 13: 525-533). These authors show that when dogs are presented upright objects, human faces, and dog faces, they spend more time gazing on novel objects and human faces than they do on familiar ones. However, they spend significantly more time gazing on familiar dog faces than they do on novel ones.

This got me thinking how a dog "knows" another face belongs to a dog. Acedotally (which I realize is worth approximately nil in science), my dog reacts very differently to seeing another dog in an adjacent car compared to a person. Given that there is so much phenotypic variation in dogs, how does it know that other animal is a dog, and not a cat or very large rodent? I could understand a lion recognizing another lion, because all lions look more or less the same.
The obvious: animals use a variety of senses

There's a wikipedia article on intra-species recognition. The article states:


  Different species may employ different methods, but all of them are
  based on one or more senses (after all, this is how the organism
  gather information about the environment). The recognition may happen
  by chemical signature (smell), by having a distinctive shape (sight),
  by emitting certain sounds (hearing), or even by behaviour patterns.
  Often a combination of these is used....
  Intra-species recognition systems are often subtle...
  Sometimes, too, intra-species recognition is fallible


Of course, this is all fairly obvious, true, and not that helpful. 

A few articles of possible relevance


Mateo (2004) has a review of recognition systems in animals if you want a more general theoretical framework.
Pfennig (1998) discusses intra-species recognition and the relationship with mate selection.


Musings on your particular question

In your particular example of two cars driving along with windows closed, we are probably excluding senses of smell and sound and requiring the dogs to rely exclusively on sight. 
I wonder whether prototype theory or some other human visual categorisation theory would apply to dogs also?

References


Mateo, J. (2004). Recognition systems and biological organization: the
perception component of social recognition. In Annales Zoologici
Fennici, volume 41, pages 729-745.
PDF
Pfenning (1988) PDF


I'm way outside my expertise here. I tossed up whether to post this at all. I figured I'd post in the absence of another answer a week after posting. The above represents what I found from a little research.