Is the natural avoidance of incest something that is learned or is the human brain programmed by instinct to have a negative response to incest? This of course would have an evolutionary advantage and would hence make perfect sense.
This is not my field, but I gave it a quick search. This article seems to speak directly to this question, summarizing and comparing multiple theories to each other. In light of these theories, the article also seems to propose a synthesized theory of the avoidance. Formal citation below. Hope this helps!
Paul, R. A. (2011). Incest Avoidance: Oedipal and Preoedipal, Natural and Cultural. Journal of the American Psychoanalytic Association, 58(6), 1087–1112. https://doi.org/10.1177/0003065110395759
The answer is that we don't really know. It is understandably difficult to conduct experiments on humans in this field, so many theories remain speculative.
The two competing forces (nature and nurture), known as the inbreeding avoidance hypothesis, and the incest taboo, both have mixed support with respect to humans. However, as is typical of human behaviour, there is probably a mix of both at play.
The Westermarck Effect
The negative effect of inbreeding on genetic fitness is well supported in animal studies, though there is less confidence about humans. This suggests that a cognitive mechanism for inbreeding avoidance might be adaptive, but it remains to be determined how kin detection actually works.
The Westermarck Effect was proposed as early as 1891 as the instinctual mechanism in humans that might implement kin detection. The effect is typically assumed to involve a genetic predisposition to learn to identify kin through early childhood experience. Supportive studies demonstrate cases where genetically unrelated individuals who are raised together in their early childhood are less likely to find each other sexually attractive, less likely to find similar individuals attractive, and less likely to marry such individuals. There are also counterpart studies on genetic siblings who are not raised together, and hence do not experience this effect.
This effect is found in animals, and is often implemented through olfactory detection - animals tend to avoid mating with others whose smell resembles those they have been raised with - and recently, the sense of smell has been proposed to account for the Westermarck effect in humans as well. Conclusive evidence is elusive however, as classical conditioning would spread avoidance to other senses. Furthermore, much of the evidence for this effect in humans has been called into question over the years.
While the apparent universality of the incest taboo suggests a genetic predisposition, the exact way in which it is implemented varies significantly across cultures (for example, how different cultures treat marriages between first-cousins). As such, a cultural origin has been surmised.
Exogamy was proposed in 1949 as the mechanism behind the anthropological Alliance Theory. This theory suggests that the incest taboo is socially adaptive, as it encourages marriages between families, increasing social stability. This was a common practice for maintaining peace among the ruling class of many monarchic societies, for example. Accordingly, when intra-family relations (endogamy) were more socially adaptive (because of isolation or protection from enemies), then the incest taboo was not present, as was the case in a few ancient societies such as Egypt, Inca, Hawaii, and China.
While a cultural origin may explain biologically counter-productive exceptions, it does not eliminate a biological impetus - a selective preference for societies that adopt the taboo over ones that don't.
Before I can answer this question, I would like to draw your attention to an iTunesU talk hosted by Professor Ralph Richard (Rick) Banks at Stanford Law about marriage restriction, which covers incestual relationships and the arguements for legal restriction in incestual marriage and relationships.
In this talk, Rick is not talking about incest between an adult and child as this is obviously wrong and covered by child sexual abuse laws. What he is referring to is incestual relationships between 2 consenting adults.
One argument he has is that society and the law generally prohibits relationships between people related by marriage - step relatives such as step-brother and step-sister or step-mother/father and step-son/daughter such as Nick Cameron and Danielle Heaney, and these cases have no bearing on genetics which others have put forward. And on that subject, Rick argues that the risk of genetic problems occurring does not legally stop anyone having a sexual relationship with someone with genetic abnormalities, so why should the law stop consensual incest between adults?
This question would come into the nature vs nurture debate (Are behaviours inherently biologically or environmentally determined?) And cases such as Nick and Danielle, above, highlight some of the reasons why this sort of question pops up on occasions.
There are papers and articles which argue that incest causes genetic abnormalities which can weaken the gene pool within populations, and there are arguments which counter-act those arguments because there is the phenomenon of Genetic Purging.
In fact, Rick Banks puts forward a valid argument which is that the main reason for restricting incest is that it can be socially destructive for individuals in the family context. However, is that because of social barriers to incest, or some other psychological reason? — I am digressing as that is another question.
I would say that the answer to your question on learnt or instinctive avoidance of incest is a matter of opinion at the moment. Some people say that the taboo comes from some sort of anti-incest mechanism within the biological framework of the person. However, from looking at things on balance through articles and papers I have read, including some of those linked in the genetic purging link above, I, along with some anthropologists, say that incest taboos are learnt social conventions. Look at how it was generally believed that homosexuality for example is down to genetic faults or mental illness.
It's selected against due to negative recessive traits/phenotypes/diseases being more pronounced and more prevalent in the gene pool of the species or population.
Individuals who do are less fit and don't fare as well on average and become less numerous in the population.
There is a social component too, as some comments say. To what degree is social but it is taboo in most species, certainly for mammals.
Example: We breed K9s to different phenotypes. We inbred to get the desired traits such as body size, color, temperament, etc. The further one diverges from the ancestral populations' phenotypes, the more health issues and generic disease are expressed. For instance, hip dysplasia. A wild individual would probably die and not get to pass any genes along if they were suffering from an ailment due to close inbreeding. In this case the selection is artificial, but the point is the same.
A good source to reference is Richard Dawkins' book The greatest show on earth