I have watched a number of times this excellent video where Dan Pink discusses the science of motivation. The video states that the higher incentives, the lower performance for non-rudimentary (not easy and quick) tasks.
A theory which I believe explains this is the ground-breaking work that Carol Dweck has done on mindsets and how they relate to performance.
To recap, People can have either a fixed mindset where they view abilities as fixed, are more driven by performance goals and use helpless strategies when confronted with tasks beyond their capabilities. If given a choice between task beyond their capability and one beneath their capability they will choose latter. If forced to do a task beyond their capability, they will not persist too long and give up early- and not put too much effort as putting efforts shows that they don't have the talent.
The other mindset is a growth mindset which sees abilities as malleable, are more driven by learning goals, and use mastery or hit-and trial and persistent strategies when confronted with a task beyond one's capability.
It has been shown that one can be put into a growth or a fixed mindset by very easy manipulations like praising for effort rather than talent.
Given that fixed mindset is also performance driven, there is a high probability that rising the stakes high focuses one on performance goals at the cost of learning goals and primes one with a fixed mindset. Ask any person preparing for a high stakes exam whether he wants to learn the subject matter or get good score and you'll see what mindset a high stakes competition entails.
Given that performance motivation or fixed mindset is detrimental for hard task (task just beyond ones abilities); its no wonder that performance deteriorates as rewards loom large for non-rudimentary tasks.
I think you have to be careful with the proposition that "rewards do not increase performance on non-rudimentary tasks". The experiments that Dan Pink cites involve experiments where participants are in a room and are supervised by an experimenter while they complete a task. This social pressure by the experimenter may well be enough for participants to be focused and diligent in completing the task.
But in the real world:
people often have time to prepare for situations that involve high stakes rewards. For example, performance on exams often have major rewards tied to their outcome. A major goal of exams is to motivate students to study hard for the subject matter.
effort and time on task is often not fixed. Employees can do overtime if the task is hard but the rewards are good (or if the pay implies that the extra effort is expected). They can engage in less off-task behaviour.
people choose the tasks they do at all. When people choose a career, it is assumed that among many other things, that many will consider the probability of getting a job and the likely salary and conditions (i.e., rewards). When a person chooses to work for a particular employer, salary relative to other potential employers is often an important factor. Thus, rewards can influence whether an individual engages in a particular activity at all.
Thus, I think one needs to be careful when generalising the meaning of these experiments to work and other contexts.
In terms of theories that might explain the results:
There is a fundamental concept in motivation illustrating this effect- a bunch of studies have been done which I don't have the time of digging up citations for now, but the central findings are as follows:
The different perceived locus of causality in intrinsic motivation leads to a different set of cognitive processes being activated generating differences in performance.
A good summary article you should read is (Ryan & Deci, 2000) in Contemporary. Edu. Psyc.
One thing to think about is what's meant by "performance." In basic cognitive tasks (e.g., memory experiments), changing the payoff matrix (and thus motivational factors) often influences bias but not discrimination -- in other words, upping the ante, so to speak, makes individuals more conservative (or careful) but doesn't much change their overall performance. One example: When incentives are low, an individual might attempt to answer 10 questions and get 5 correct; when incentives are high, an individual might attempt to answer 20 questions and get 10 correct. In a sense the latter is performing better -- after all, he or she is getting 10 more questions correct -- but is also making more mistakes.
Just another way of thinking about things.
High incentives (especially related to time needed to finish a task) makes one very concentrated but at the same time leaves little opportunity to look outsid the box (i.e. to develop a creative approach, sometimes needed to complete a non-rudimentary task).
For a popular science/economics talk, see also: http://www.ted.com/talks/dan_pink_on_motivation.html
