How is it that taking a break from a problem sometimes allows you to figure out the answer?

Question

As a computer programmer, I have noticed an interesting phenomenon: If I am stuck on a particular problem in my work, often if I stop thinking about the problem and do something else, the answer will suddenly come to me.

Is there a name for this phenomenon? How does this work? Has any research been done on this? How is it that taking a break from a problem sometimes allows you to figure out the answer?

Edit: I remember now where I heard about this phenomenon: on the Charlie Rose Brain Series, Eric Kandel of Colombia University says (at 43:20 in)

[The unconscious] can do many processes at the same time. You can either focus on one thing or another, you can't focus on two or three things at the same time. Because consciousness is very limited in what it can do; unconsciousness is much broader. And although we know very little about the true nature of creativity, one emerging theme that comes out of this is that, if you're trying to solve a mathematical problem... if you're trying to solve any intellectual problem... you keep on focusing at it, you may get stuck. Taking a break, taking a shower, going for a walk, playing golf, you come back refreshed. And often doing the other activity, boom, the idea will come to you.

But then they change the subject! What is the concept he's talking about, and where can I read more about how it works?

Answer 1

It sounds like you're talking about a classic example of Incubation.

Incubation is defined as a process of unconscious recombination of thought elements that were stimulated through conscious work at one point in time, resulting in novel ideas at some later point in time.

Here's a great article by John F. Kihlstrom: Intuition, Incubation, and Insight: Implicit Cognition in Problem Solving. Basically it is believed that Incubation or stopping conscious thought on a problem allows one to find more creative solutions to a problem:

In these cases, Wallas argued, thinkers enter an incubation stage in which they no longer consciously think about the problem. Wallas (1926) actually distinguished this phenomenon between two forms of incubation: "the period of abstention may be spent either in conscious mental work on other problems, or in a relaxation from all conscious mental work" (p. 86).

Wallas believed that there might be certain economies of thought achieved by leaving certain problems unfinished while working on others, but he also believed that solutions achieved by this approach suffered in depth and richness. In many cases of difficult and complex creative thought, he believed, deeper and richer solutions could be achieved by a suspension of conscious thought altogether, permitting "the free working of the unconscious or partially conscious processes of the mind" (p. 87).1 In either case, Wallas noted that the incubation period was often followed by the illumination stage, the "flash" (p. 93) in which the answer appears in the consciousness of the thinker.

Kihlstrom's references contain many good experiments backing up the claims made.

A reason incubation may work is because it releases "fixation"; that case of being "stuck" which is a sort of mental rut which prevents one from thinking of new answers or methods of solving a problem.

We become stuck on an idea that we believe should work but doesn't, which may hold us back from thinking of a different solution which actually does work, one we may have previously not considered or disregarded.

A great dissertation by Bo T. Christensen covers both ideas of Fixation and Incubation in depth: Creative Cognition: Analogy and Incubation.

Answer 2

Emotions are a very powerful problem solving blockers. Here is why:

Strong emotions (when you are upset, stressed, etc) disconnect partially your frontal lobe from the rest of the brain. Your frontal lobe is responsible for fonctions such as problem solving.

Moving away from the source of the emotion reduce it, and you get reconnected.

There is an interesting paper about problem solving and emotions.

Performance and data from some cognitive models suggested that emotions, experienced during problem solving, should be taken into account. Moreover, it is proposed that the cognitive science approach using both theoretical and experimental data may lead to a better understanding of the phenomena. A closer investigation of AC T-R cognitive architecture (Anderson 1993) revealed some properties analogous to phenomena known from the activation theory of emotion. A model of the classical Yerkes-Dodson experiment was built to test the predictions. The study explained such psychological phenomena as arousal, motivation and confidence within the mathematical notation. The influence of changes in these motivational states, controlled by emotion, on information processing has been investigated and it is shown that the dynamics corresponds to the well-known optimisation methods, such as best-first search and simulated annealing.

Other potential problems that prevent you from solving a problem:

• Fixation: Previous experience or familiarity can make problem solving less easier
• Confirmation bias: Tendency of people to favor information that confirms their beliefs or hypotheses.
• Availability heuristic
• Representativness heuristic

Answer 3

Check out the book The Psychology of Computer Programming by Gerald Weinberg. Originally published in 1971 it was recently produced in a Silver Anniversary version. I own an original copy and read it again about every five years. Weinberg and Glenford Myers both wrote amazing texts which will be useful forever.

In Weinberg, the situation you are describing is specifically addressed, and his explanation says it is the result of knocking down preset pathways cognitively, then when you have to reestablish them you have cleared some cognitive assumptions placed there in the initial thinking of the problem. He also identifies the simple event where you begin to explain the problem to another only to suddenly discover the solution whereas it eluded you for hours before. This, again, results from employing different neuronal circuitry when we speak out loud versus when we are doing slow thinking (a la Daniel Kahneman, Thinking Fast and Slow). Weinberg's work is full of fun explanations and examples of how weird software engineering psychology can be.

Answer 4

The phenomenon's called the incubation effect. Wikipedia operationally defines the incubation effect as any benefit of a break during problem solving. In Wallas’ (1926) four-stage model of innovative problem solving or creativity, the incubation stage is the stage in which one takes some time away from the problem (the stages are: preparation, incubation, illumination, verification).

I too oftentimes experience this phenomenon while programming.

To start answering your question about how the incubation effect works, I'll point to Segal's attention withdrawal and returning act hypotheses presented in the (2004) paper entitled Incubation in Insight Problem Solving.

To solve an insight puzzle [or a programming problem that one's stuck on], the solver must escape from the mental activity governed by the false assumption inside the deficient problem space. That could be achieved by a delicate mental condition that allows, on one hand, a withdrawal of the attention governed by the false assumption, and, on the other hand, some level of activation of the problem’s elements in the solver’s mind. The presence of the problem’s elements in the mind—not anymore constrained by the relations forced by the false assumption—provides the solver an opportunity to apply another structure to these elements, this time governed by a different assumption that may lead to the right solution. (p. 143)

I'll then refer you to this essay of mine for an exposition and critical analysis of the above-mentioned ideas, and for much much more. Feedback on the essay is welcome.

To address the quote from Kandell in the OP, here are some (paraphrased) point-form notes from PSY370 at University of Toronto on November 17, 2011. (Parts of the following might be difficult for you to understand; read some research, and take the class! :p).

• Is creativity just insight?

• If yes and we can explain insight, then it would make creativity scientifically tractable.

o Weisberg wants to make that move.

• Bowden proposes that there’s nothing to creativity other than insight prob solving. Computational model of creativity.

• Bowden: a person is creative when they solve a problem that they couldn’t solve before (because they misframed it).

• Creativity is whatever reconstrual allows you to do something or solve some problem you couldn’t solve before.

• Individually/personally creative.

• Historically creative if individual comes up with a way of solving the problem that humans couldn’t do before.

o Solution that feeds into distributed cognition.

o E.g. Einstein.

o The more an act is historically creative, the more creative we tend to find it.

• Both historical and personal creativity are just insight.

• (One of) Vervaeke's insight(s) about problem solving:

• Maybe creativity’s more about problem finding. We value both (problem finding and problem solving) for different reasons. There's a difference in how we’re motivated and valuing. Maybe to get at what’s different about creativity, we have to get at motivation and valuation. Insight: cognitive work. Creativity: cognitive play. The difference between work and play isn’t so much what we’re doing, but the context of motivation (what goals we’re pursuing). This is a fundamental difference between insight and creativity.

...

• In one sense, insight and creativity are fundamentally the same, but in another sense they’re fundamentally different.

• In insight we do learning-to-learn for the sake of improving creativity. In creativity, we put ourselves in a learning situation for the sake of improving learning-to-learn.

• Capacity for creativity is specific improvement for our insight.

• Ability to think projectively.

• No extra machinery to creativity other than insight.

• But different purpose.

o Instead of learning-to-learn for the sake of learning, we put ourselves in a learning context for the sake of learning-to-learn.

• Therefore, we can explain creativity with the theoretical machinery of insight but also explain the purpose of creativity.

• Explain mutual reinforcement of creativity and insight, like wake-sleep [re: Hinton] reinforcement.

As for an explanation of what insight is and how it works, I begin by endorsing Stephen and Dixon’s (2009, p. 94; italics mine) account.

We propose that the theory of open, nonlinear systems (see Ebeling & Sokolov, 2005; Hilborn, 1994; Klimontovich, 1991 for further discussion of this theory) is of utmost relevance to problem solving. It provides a compelling account of the unfolding of cognition in action and the phenomenon of insight. The exchange between an open, nonlinear system and its environment leads to complex interactions and energy flows. These interactions and flows amount to perturbations with which the system must cope. The solution to such perturbations is the self-organization of new steady states. Action is the complex interface at which the cognitive system and environment meet. The self-organizing steady states are cognitive structures that emerge as a result of exchanges between the cognitive system and environment. Insight is thus emergent structure forged amidst the nonlinear interactions of cognition, action, and the environment.

The dynamical systems phenomenon of self-organizing criticality (discovered by Bak, Tang, and Wisenfeld, 1987, 1988) can ground the theory of self-organized steady states in neural dynamics (Vervaeke, PSY370, Nov. 10, 2011; cf. Irving, Vervaeke, and Ferraro, 2010). This grounding greatly increases the plausibility of Stephen and Dixon’s (2009) account. The grounding of self-organized steady states, combined with acceptance of Stephen and Dixon’s (2009) account and acceptance of Schilling’s “small-world network model of cognitive insight” (2005, p. 1), leads to the claim that “insight is a process of self-organizing criticality that affords re-construal” (Vervaeke, PSY370, Nov. 10, 2011), since the process of self-organizing criticality helps to make small-world networks that cause re-construal by shifting back and forth between synchronous and asynchronous firing.

Insight is not computation. It’s a dynamical self-organizing system that makes small-world networks that cause reconstrual. We're now past the search-inference framework and the gestalt framework. Insight is developmentally running between perception and action: termed enaction. It's a dynamical system running on perception and action through recursive internal mutual modelling. The cerebellum is running on parts of brain that are acting and perceiving. (Vervaeke, PSY370, Nov. 10, '11).

I'm going to hand-wave and just say that good arguments for the cerebellum being integral to both insight and creativity were presented in that class.

See my essay for references.

Answer 5

What you are describing is a mental bottleneck (or that is the term coined by David Rock Anyway - I actually wrote about it (in the context of Dual N Back training in this blog post - to quote myself

A bottleneck happens when you can't solve a problem because you can't remove a bad thought from your brain. A bad thought is something you know doesn't work, but you keep thinking about anyway, simple because your brain lacks the energy to get rid of it and replace it with something new.

Essentially by taking a break from the problem you are allowing the bad thoughts to leave your brain, quite similar to rebooting a computer. Once the bad thoughts are gone you can load everything fresh and your brain is no longer burdened by inhibiting the bad thoughts, so you can devote more brainpower to solutions.

Answer 6

I believe that this excellent answer can shed some light on why you might be unable to figure out the problem. The answerer described it as mental exhaustion, mental fatigue or mental stress. This, though does not explain why you will suddenly see the solution, but I would guess that the tired part of the brain will continue working on the problem in some way.