When confronted with the Prisoner's Dilemma, people often refuse to cooperate. Instead, they choose to betray other people, even when cooperation yields a more favorable outcome.
Why do people behave in that way? Is it a genealogical feature which is embedded during evolution?
'Short' answer
The way people respond to the prisoner's dilemma strongly depends on how the experiment is set up. In a repeated game, where the same 'prisoners' meet each other multiple times, the tit-for-tat strategy seems to be favored. This, because contenders learn the other's behavior and start anticipating the choice from their opponent to manipulate the outcome. It is not so that people by default betray their opponent. The betrayal (confession) is the selfish option and is favorable when you have no idea what the other will do, as it will always work in favor of yourself. The cooperation (keeping silent) is the most risky one and you only do that when you like to take big risks (unlikely, most people don't), or when you have a fairly good idea what your opponent will do.
Background
The prisoners’ dilemma is about cooperation and competition and applies to social settings, business and politics. In the traditional version of the game, the police have arrested two suspects A and B. Both were involved in he same crime and they are interrogated separately. The police offers A and B to confess and by doing so giving away the accomplice, or to keep silent (source: Library of Economics & Liberty).
When A will confess and gives away B, A will be granted a shorter sentence as A has confessed and has aided the cops in sentencing B, and vice versa when B confesses. If A so confesses, but B holds silent (not knowing he has been given away), B will face an exceptionally harsh sentence because of refusing to admit his crimes (say 5 years in jail).
When both confess they will both face reduced sentences because they are helping the police as a witness (say 2 year in jail).
When both keep silent they can only be sentenced on minor charges because of a lack of proof (say a sentence of just 1 year).
The “dilemma” faced by the prisoners is that, whatever the other does, each is better off confessing than remaining silent. But the outcome obtained when both confess is worse for each than the outcome they would have obtained had both remained silent.
The prisoner's dilemma can be viewed at as a puzzle, illustrating the conflict between individual and group rationality. A group whose members pursue rational self-interest may all end up worse off than a group whose members act contrary to rational self-interest. More generally, if the payoffs are not assumed to represent self-interest, a group whose members rationally pursue any goals may all meet less success than if they had not rationally pursued their goals individually (source: Stanford University).
A slightly different interpretation takes the game to represent a choice between selfish behavior and socially desirable altruism (behavior of an individual that benefits another at its own expense). The move corresponding to confession benefits the actor, no matter what the other does (selfish), while the move corresponding to silence benefits the other player no matter what that other player does (altruism). Benefiting oneself is not always wrong, of course, and benefiting others at the expense of oneself is not always morally required, but in the prisoner's dilemma game both players prefer the outcome with the altruistic moves to that with the selfish moves. This observation has led David Gauthier and others to take the Prisoner's Dilemma to say something important about the nature of morality (source: Stanford University).
Now - to come to your question - you talk about 'betrayal' (which is probably referring to 'confessing') and cooperation (probably referring to 'holding silent'). You have to realize that keeping silent means you have to know that the other person will (likely) do that too. In a one-shot situation you cannot know. However, it is only when the game is repeatedly performed, and when the subjects can learn the behavior of their 'accomplice' that subjects start to make educated guesses. A particularly fruitful strategy, surprisingly, in a repeated game is the 'tit-for-tat' approach, an almost childish approach to the dilemma, where people cooperate with their partner on the first round, then adjust their behavior to match your partner’s, as in, you do to them what they just did to you. If your partner cooperated (stayed silent) the last time, you also choose to cooperate (stay silent too) this time; if they defected (confessed) last time, you respond in kind by retaliating (confessing) against them (source: Seltzer in Psychology Today, 2016).
The point about PD is that co-operation is NOT the more favourable outcome - for the INDIVIDUAL. Thus, the best strategy on single shot or fixed session PD games is always to defect.
What if the experiment is open-ended? Then, as Robert Axelrod and numerous other mathematical models have demonstrated, the optimal strategy is to co-operate with Tit for Tat.
The ‘cause’ of defection can thus be seen as ‘self interest’. However, cooperation always decays over time which leads us to consider that Reputation is also a factor, worth a look:
Duca and Nax 2018 Groups and Scores: The Decline of Cooperation
Simply defecting deteriorates ones own reputation, and this in turn becomes a new cause of defection in future rounds
Actually we need to differentiate between two situations: when a perfectly rational subject (think a robot) plays the game and when a human plays the game.
When a robot plays the game, he knows he can't control other subject's actions, so he only considers his own choices and considers the two choices of the opponent equally possible. So put yourself in the shoes of that robot - he has two possibilities:
Betray:
- he gets 0 years or 2 years of prison (both equally possible)
Don't betray:
- he gets 1 year or 3 years of prison (both equally possible)
We see here that both cases are worse if he doesn't betray. The first part shows us that with betraying he gets 0 years, and with not betraying he gets 1 years, which is worse. The other part shows us that with betraying he gets 2 years and with not betraying he gets 3 years, which is again worse.
So he concludes that it's better to always betray (the important thing here is that he can't control what the other subject will do, so he must consider that both of the opponents choices are equally probable). 50% probability to get 0 years is better than 50% probability to get 1 year. On the other hand, 50% probability to get 2 years is better than 50% probability to get 3 years.
But now we come to humans. What's interesting is that, as the Wikipedia article says,
In reality, humans display a systemic bias towards cooperative behavior in this and similar games despite what is predicted by simple models of "rational" self-interested action.
Humans are actually not playing it as a robot would! Humans show a tendency to be more cooperative than robots, which contrasts with what you imply in your question.
The reason for humans being more cooperative could be that they care about their reputation, as the other user said.
