The phenomenon's called the incubation effect. Wikipedia operationally defines the incubation effect as any benefit of a break during problem solving. In Wallas’ (1926) four-stage model of innovative problem solving or creativity, the incubation stage is the stage in which one takes some time away from the problem (the stages are: preparation, incubation, illumination, verification).
I too oftentimes experience this phenomenon while programming.
To start answering your question about how the incubation effect works, I'll point to Segal's attention withdrawal and returning act hypotheses presented in the (2004) paper entitled Incubation in Insight Problem Solving.
To solve an insight puzzle [or a programming problem that one's stuck on], the solver must escape
from the mental activity governed by the false assumption
inside the deficient problem space. That could be
achieved by a delicate mental condition that allows, on
one hand, a withdrawal of the attention governed by
the false assumption, and, on the other hand, some
level of activation of the problem’s elements in the
solver’s mind. The presence of the problem’s elements
in the mind—not anymore constrained by the relations
forced by the false assumption—provides the solver an
opportunity to apply another structure to these elements,
this time governed by a different assumption
that may lead to the right solution. (p. 143)
I'll then refer you to this essay of mine for an exposition and critical analysis of the above-mentioned ideas, and for much much more. Feedback on the essay is welcome.
To address the quote from Kandell in the OP, here are some (paraphrased) point-form notes from PSY370 at University of Toronto on November 17, 2011. (Parts of the following might be difficult for you to understand; read some research, and take the class! :p).
• Is creativity just insight?
• If yes and we can explain insight, then it would make creativity scientifically tractable.
o Weisberg wants to make that move.
• Bowden proposes that there’s nothing to creativity other than insight prob solving. Computational model of creativity.
• Bowden: a person is creative when they solve a problem that they couldn’t solve before (because they misframed it).
• Creativity is whatever reconstrual allows you to do something or solve some problem you couldn’t solve before.
• Individually/personally creative.
• Historically creative if individual comes up with a way of solving the problem that humans couldn’t do before.
o Solution that feeds into distributed cognition.
o E.g. Einstein.
o The more an act is historically creative, the more creative we tend to find it.
• Both historical and personal creativity are just insight.
• (One of) Vervaeke's insight(s) about problem solving:
• Maybe creativity’s more about problem finding. We value both (problem finding and problem solving) for different reasons. There's a difference in how we’re motivated and valuing. Maybe to get at what’s different about creativity, we have to get at motivation and valuation. Insight: cognitive work. Creativity: cognitive play. The difference between work and play isn’t so much what we’re doing, but the context of motivation (what goals we’re pursuing). This is a fundamental difference between insight and creativity.
• In one sense, insight and creativity are fundamentally the same, but in another sense they’re fundamentally different.
• In insight we do learning-to-learn for the sake of improving creativity. In creativity, we put ourselves in a learning situation for the sake of improving learning-to-learn.
• Capacity for creativity is specific improvement for our insight.
• Ability to think projectively.
• No extra machinery to creativity other than insight.
• But different purpose.
o Instead of learning-to-learn for the sake of learning, we put ourselves in a learning context for the sake of learning-to-learn.
• Therefore, we can explain creativity with the theoretical machinery of insight but also explain the purpose of creativity.
• Explain mutual reinforcement of creativity and insight, like wake-sleep [re: Hinton] reinforcement.
As for an explanation of what insight is and how it works, I begin by endorsing Stephen and Dixon’s (2009, p. 94; italics mine) account.
We propose that the theory of open, nonlinear systems (see Ebeling & Sokolov, 2005; Hilborn, 1994; Klimontovich, 1991 for further discussion of this theory) is of utmost relevance to problem solving. It provides a compelling account of the unfolding of cognition in action and the phenomenon of insight. The exchange between an open, nonlinear system and its environment leads to complex interactions and energy flows. These interactions and flows amount to perturbations with which the system must cope. The solution to such perturbations is the self-organization of new steady states. Action is the complex interface at which the cognitive system and environment meet. The self-organizing steady states are cognitive structures that emerge as a result of exchanges between the cognitive system and environment. Insight is thus emergent structure forged amidst the nonlinear interactions of cognition, action, and the environment.
The dynamical systems phenomenon of self-organizing criticality (discovered by Bak, Tang, and Wisenfeld, 1987, 1988) can ground the theory of self-organized steady states in neural dynamics (Vervaeke, PSY370, Nov. 10, 2011; cf. Irving, Vervaeke, and Ferraro, 2010). This grounding greatly increases the plausibility of Stephen and Dixon’s (2009) account. The grounding of self-organized steady states, combined with acceptance of Stephen and Dixon’s (2009) account and acceptance of Schilling’s “small-world network model of cognitive insight” (2005, p. 1), leads to the claim that “insight is a process of self-organizing criticality that affords re-construal” (Vervaeke, PSY370, Nov. 10, 2011), since the process of self-organizing criticality helps to make small-world networks that cause re-construal by shifting back and forth between synchronous and asynchronous firing.
Insight is not computation. It’s a dynamical self-organizing system that makes small-world networks that cause reconstrual. We're now past the search-inference framework and the gestalt framework. Insight is developmentally running between perception and action: termed enaction. It's a dynamical system running on perception and action through recursive internal mutual modelling. The cerebellum is running on parts of brain that are acting and perceiving. (Vervaeke, PSY370, Nov. 10, '11).
I'm going to hand-wave and just say that good arguments for the cerebellum being integral to both insight and creativity were presented in that class.
See my essay for references.