What research exists in the areas of formulating questions and "problem shaping"?

Question

After recalling Eric Steven Raymond and Rick Moen's How to Ask Questions The Smart Way and a discussion in a systems engineering course regarding the impact of the proper formulation of a problem in order to achieve the optimal solution(s), I began to look for research into methodologies that can be used to formulate a question prior to asking it.

In Raymond and Moen's post, they have a few short lines touching on what I'm referring to. They mention to prepare your question by thinking it through and to beware of asking questions based on false assumptions. However, I'm more interested in psychological research into how people go about these activities, preferably in some kind of methodology or framework that can be applied to assess how one is formulating questions or even how to go about forming a question that can then go through a methodology similar to Raymond and Moen's process.

Searching for information about asking questions has so far yielded things about creating surveys with effective questions and how questions can influence answers. Perhaps I'm not using the right terms or looking in the right fields.

I found the term "problem shaping" on Wikipedia. The article (which is a stub) describes the concept that I'm looking for. However, I checked into the related articles and they don't appear to describe anything that I'm looking for in more depth. The "see also" pages appear to focus on topics related to problem solving, not actually formulating and shaping the problem. It seems that in order to effectively solve the problem, you need an appropriately formulated problem statement, yet I can't find anything about the methodologies or processes to do so stated formally.

My specific questions:

• Is there any research (of any level of scientific rigor) in the frameworks or processes of developing a good question or problem statement?
• If so, what are some examples of work done and what information or products have been generated?
• If not, what topics might be used to provide insights to gain a deeper understanding of the formulation of questions or problem statements?

Answer 1

There are no known domain-general ways of formulating good questions and problems—what constitutes a good question or problem formulation depends on the field or domain. A good psychology question is different from a good literary question, which in turn is different from a good business question.

Because there are no known domain-independent ways of formulating good questions and problems, but I believe this is a valuable question, I will focus my answer on the limited literature covering the formulation of good behavioral and social science questions. In general, while formulating good questions is specific to the domain or related set of domains of interest, research on ideation or idea generation is covered by the field of creativity research (but this does not cover good questions).

Research heuristics and hypothesis generation

Theory

Many scientific methodology courses focus almost entirely on study design and hypothesis testing appropriate for the field, while neglecting the critical aspects of hypothesis generation and research heuristics. Research is somewhat scarce, but some frameworks have been proposed, of which I am most familiar with McGuire's 49 research heuristics (McGuire, 1997). These heuristics are grouped into five categories, which I will try to briefly summarize.

1. Sensitivity to provocative natural occurrences: Systematically paying attention, making systematic observations, looking for anomalies (unexpected, nonobvious relations) and attempting to explain these.
2. Simple conceptual analysis: Thought and experiments conceptual analysis such as counterexamples, imagining counterfactuals, reversing causal direction and so forth on a defined set of phenomena.
3. Complex conceptual analysis: Deductive reasoning and generalization, convergent-divergent ideation processes and metatheorical techniques.
4. Reinterpretations of past research: Expert knowledge, reevaluating the scope of evidence accounted for and meta-analysis or review.
5. Collecting new or reanalyzing old data: Quantitative and qualitative analysis of new evidence, or novel analysis of existing evidence.

McGuire's theoretical framework is not the only one. Lubart and Getz (1998), for example, proposed three different research heuristics within the domain of psychology based on a historical analysis of psychological research. They named these the trilogy-of-mind heuristic ("the mind comprises cognition, conation and affection"), the emotions-as-moods ("complex human emotion reduces to simple moods") heuristic, and the analysis-of-variance ("the map is the territory") heuristic. Both frameworks are ultimately based on analysis of heuristics researchers have used, rather than directly investigating these heuristics' efficacy, however.

Others have questioned whether the search for such heuristics is a good idea in the first place. For example, a critique of research heuristics in a related field leveled by Anderson, De Dreu and Nijstad (2004) is the dilemma that research heuristics may "routizine" research. This is a possible problem, and they also report evidence for "a heavy focus on replication–extension, cross-sectional designs, and a single level of analysis in innovation research." However, it is not clear that this routinization caused by research heuristics.

Evidence

Unfortunately, the neglect of hypothesis generation in scientific training is reflected in the empirical literature, if not the theoretical literature. There are no or very limited empirical research programmes or frameworks directly investigating the relative or absolute efficacy of different research heuristics or approaches, even in terms of relatively obtainable measures like impact factor or publication rate. Therefore, although I subjectively find them helpful, we ultimately cannot say with confidence whether or which of any of McGuire's (or others') heuristics are effective for formulating good questions and problems or not.

References

• Anderson, N., De Dreu, C. K. W. and Nijstad, B. A. (2004), The routinization of innovation research: a constructively critical review of the state-of-the-science. J. Organiz. Behav., 25: 147–173. doi: 10.1002/job.236
• Lubart, T. I. and Getz, I. (1998), The Influence of Heuristics on Psychological Science: A Case Study of Research on Creativity. Journal for the Theory of Social Behaviour, 28: 435–457. doi: 10.1111/1468-5914.00083
• McGuire, W. J. (1997). Creative hypothesis generating in psychology: Some useful heuristics. Annual review of psychology, 48(1), 1-30.