Is there any research that assesses whether males or females judge faces of the opposite sex with a greater internal consistency? That is, is the standard deviation of attraction ratings for a specific opposite sex face on average smaller when looking at a specific gender?
There is a very large literature on this, and it features many subtle points, but I will try to summarize some general themes.
In general, subjects are very consistent at ranking pictures of others for attractiveness (thus, eliminating the popular notion of "beauty is in the eye of the beholder"). For instance Cunnigham et al. (1995) found a correlation of .9 between individual rating and the average rating of pictures of women. This correlation remains high even if the rater and photo are from different cultures. Further, men and women tend to find the same features good-looking (i.e. heteresexual men and women, deem similar women as good-looking). Johnston & Franklin (1993) let 20 male and 20 female subjects use a genetic algorithm to generate female faces that they deemed good-looking. With the exception of the lower lip (females preferred a larger lower lip) there was no statistical differences between the qualities of the faces produced by male and female subjects.
However, women have a higher variance in ratings of sex-objects (opposite sex for heterosexual, same-sex for homosexual females) than men (Jankowiak et al. 1992; Townsend & Wasserman 1997). But this should be taken with a grain of salt because attractiveness rankings have much higher variation when ranking males as opposed to females. Schulman & Hoskins (1986) found that ratings of female photos had statistically significant lower variance than male photos for both male and female raters. Thus, the effect could partially be in that both sexes are worse at judging attractiveness of males.
Of course, the hardest is judging attractiveness in oneself. Rand & Hall (1983) found that females has a .5 correlation between their self-perception of attractiveness and the rating of male judges. Men are much worse, with only a .1 (almost chance) correlation between self-attractiveness ratings and the ratings of female judges.
The story gets even more complicated, because women respond differently to how they load features when producing ratings depending on if they are looking for a short-term or long-term partner (Widerman & Dubois 1998) and even where they are in their menstrual cycle (Penton-Voak & Perrett 2001).
Due to the complicated interaction of all these factors (and more!) I don't think there is (yet) one final and definitive answer to your question.
- Cunningham, M., Roberts, A.R., Berbee, Anita, P., Druen, P.B., & Wu, C.  ""Their ideas of beauty are, on the whole, the same as ours": Consistency and variability in the cross-cultural perception of female physical attractiveness." Journal of Personality and Social Psychology, 68: 261-279.
- Jankowiak, W.R., Hill, E.M., & Donovan, J.M.  "The effects of sex and sexual orientation on attractiveness judgments: An evolutionary interpretation." Ethology and Sociobiology 13(2): 73-85.
- Johnston, V.S., & Franklin, M.  "Is beauty in the eye of the beholder?" Ethology and Sociobiology 14(3): 183-199.
- Penton-Voak, I.S., & Perrett, D.I.  "Male facial attractiveness: Perceived personality and shifting female preferences for male traits across the menstrual cycle." Advances in the Study of Behavior 30: 219-259.
- Rand, C., & Hall, J.  "Sex differences in the accuracy of self-perceived attractivness." Social Psychology Quarterly, 46: 359-363.
- Townsend, J.M., & Wasserman, T.  "The perception of sexual attractiveness: sex differences in variability." Archives of Sexual Behavior 26(3): 243-268.
- Wiederman, M.W., & Dubois, S.L.  "Evolution and Sex Differences in Preferences for Short-Term Mates: Results from a Policy Capturing Study" Evolution and Human Behavior 19: 153-170.