The question of whether "nice guys finish last", also known as the nice guy stereotype, is often studied in an economic or resource-allocation context as a more general case. According to the Competitive Altruism Hypothesis (e.g., Hardy and Van Vugt, 2006) altruistic or prosocial behavior helps the actor to accumulate social status, which in turn confers material benefits such as increased access to resources. In their words,
Individuals may behave altruistically for reputation reasons because selective benefits (associated with status) accrue to the generous.
Prosocial behavior and attraction
For the general case of resource allocation, reputation and status are plausible mechanisms for deriving benefits from prosocial and altruistic behavior, and covering this literature at length is probably beyond the scope of the question. Luckily, the question has also been tested specifically with respect to attraction, which is related but not identical to sexual behavior. In a 1995 study, Jensen-Campbell, Graziano and West reported the following results:
In Study 1, prosocial men were rated as more physically and sexually attractive, socially desirable, and desirable as dates than were nonprosocial men. Dominant men were no more attractive than low-dominance men, and male dominance did not interact with male prosocial orientation in eliciting attraction from women. In Study 2, prosocial orientation was manipulated to avoid "personalism," but still affected attraction. Across all measures attraction was an interactive function of dominance and prosocial tendencies. Dominance alone did not increase any form of attraction measured. In Study 3, male prosocial tendencies and dominance interacted to affect women's attraction to men.
Their study has a number of limitations which are worth noting before accepting this at face value, notably that it used attraction rating rather than some form of measure of actual sexual behavior. It therefore remains conceivable that nice guys finish last in real settings because more dominant men can, well, dominate less dominant men out of the 'competition' (attraction vs. sexual behavior problem). The study was also relatively statistically underpowered, which increases the chance of false positives. It is also worth considering that human sexual behavior is almost certainly more complex than can be captured by a single continuum.
A later study found that perceived niceness was a significant factor in women's expressed desire for long-term relationships, whereas physical attractiveness was a significant factor in expressed desire for more casual sex (Urbaniak and Kilmann, 2003). While these findings broadly concur with the Jensen-Campbell study and with the more general Competitive Altruism Hypothesis (nice guys finish first given enough time to accrue status), the study ultimately suffers from the same limitations.
Concluding remarks
It does not seem reasonable to make the unqualified claim that "nice guys finish last," but nor does it seem sound to claim that "nice guys finish first." What we can say is that nice guys do not always finish last, and not-nice guys do not always finish first, but rather, male niceness has different effects over time and contexts. Perceived niceness and prosocial behavior does appear to make a man more attractive on average, but it remains an empirical question whether this increased attractiveness really translates into successfully attracting women.
References
Hardy, C. L., & Van Vugt, M. (2006). Nice guys finish first: The competitive altruism hypothesis. Personality and Social Psychology Bulletin, 32(10), 1402-1413.
Jensen-Campbell, L. A., Graziano, W. G., & West, S. G. (1995). Dominance, prosocial orientation, and female preferences: Do nice guys really finish last?. Journal of Personality and Social Psychology, 68(3), 427.
Urbaniak, G. C., & Kilmann, P. R. (2003). Physical attractiveness and the “nice guy paradox”: Do nice guys really finish last?. Sex Roles, 49(9-10), 413-426.
