My motivation for this question is dog-based, but I suppose it would apply equally well to humans. How do animals recognize their own kind, particularly where there is large variation in appearance?
The background is that I am reading "Discrimination of human and dog faces and inversion responses in domestic dogs (Canis familiaris)" by Racca et al. (2010; Animal Cognition 13: 525-533). These authors show that when dogs are presented upright objects, human faces, and dog faces, they spend more time gazing on novel objects and human faces than they do on familiar ones. However, they spend significantly more time gazing on familiar dog faces than they do on novel ones.
This got me thinking how a dog "knows" another face belongs to a dog. Acedotally (which I realize is worth approximately nil in science), my dog reacts very differently to seeing another dog in an adjacent car compared to a person. Given that there is so much phenotypic variation in dogs, how does it know that other animal is a dog, and not a cat or very large rodent? I could understand a lion recognizing another lion, because all lions look more or less the same.