What causes people to betray each other when facing the prisoner's dilemma? When confronted with the Prisoner's Dilemma, people often refuse to cooperate. Instead, they choose to betray other people, even when cooperation yields a more favorable outcome. 

Why do people behave in that way? Is it a genealogical feature which is embedded during evolution?
'Short' answer
The way people respond to the prisoner's dilemma strongly depends on how the experiment is set up. In a repeated game, where the same 'prisoners' meet each other multiple times, the tit-for-tat strategy seems to be favored. This, because contenders learn the other's behavior and start anticipating the choice from their opponent to manipulate the outcome. It is not so that people by default betray their opponent. The betrayal (confession) is the selfish option and is favorable when you have no idea what the other will do, as it will always work in favor of yourself. The cooperation (keeping silent) is the most risky one and you only do that when you like to take big risks (unlikely, most people don't), or when you have a fairly good idea what your opponent will do.  

Background
The prisoners’ dilemma is about cooperation and competition and applies to social settings, business and politics. In the traditional version of the game, the police have arrested two suspects A and B. Both were involved in he same crime and they are interrogated separately. The police offers A and B to confess and by doing so giving away the accomplice, or to keep silent (source: Library of Economics &amp; Liberty). 

When A will confess and gives away B, A will be granted a shorter sentence as A has confessed and has aided the cops in sentencing B, and vice versa when B confesses. If A so confesses, but B holds silent (not knowing he has been given away), B will face an exceptionally harsh sentence because of refusing to admit his crimes (say 5 years in jail).    

When both confess they will both face reduced sentences because they are helping the police as a witness (say 2 year in jail).

When both keep silent they can only be sentenced on minor charges because of a lack of proof (say a sentence of just 1 year). 

The “dilemma” faced by the prisoners is that, whatever the other does, each is better off confessing than remaining silent. But the outcome obtained when both confess is worse for each than the outcome they would have obtained had both remained silent. 

The prisoner's dilemma can be viewed at as a puzzle, illustrating the conflict between individual and group rationality. A group whose members pursue rational self-interest may all end up worse off than a group whose members act contrary to rational self-interest. More generally, if the payoffs are not assumed to represent self-interest, a group whose members rationally pursue any goals may all meet less success than if they had not rationally pursued their goals individually (source: Stanford University). 

A slightly different interpretation takes the game to represent a choice between selfish behavior and socially desirable altruism (behavior of an individual that benefits another at its own expense). The move corresponding to confession benefits the actor, no matter what the other does (selfish), while the move corresponding to silence benefits the other player no matter what that other player does (altruism). Benefiting oneself is not always wrong, of course, and benefiting others at the expense of oneself is not always morally required, but in the prisoner's dilemma game both players prefer the outcome with the altruistic moves to that with the selfish moves. This observation has led David Gauthier and others to take the Prisoner's Dilemma to say something important about the nature of morality (source: Stanford University).

Now - to come to your question - you talk about 'betrayal' (which is probably referring to 'confessing') and cooperation (probably referring to 'holding silent'). You have to realize that keeping silent means you have to know that the other person will (likely) do that too. In a one-shot situation you cannot know. However, it is only when the game is repeatedly performed, and when the subjects can learn the behavior of their 'accomplice' that subjects start to make educated guesses. A particularly fruitful strategy, surprisingly, in a repeated game is the 'tit-for-tat' approach, an almost childish approach to the dilemma, where people cooperate with their partner on the first round, then adjust their behavior to match your partner’s, as in, you do to them what they just did to you. If your partner cooperated (stayed silent) the last time, you also choose to cooperate (stay silent too) this time; if they defected (confessed) last time, you respond in kind by retaliating (confessing) against them (source: Seltzer in Psychology Today, 2016).