What research exists in the areas of formulating questions and "problem shaping"?
There are no known domain-general ways of formulating good questions and problems—what constitutes a good question or problem formulation depends on the field or domain. A good psychology question is different from a good literary question, which in turn is different from a good business question.
Because there are no known domain-independent ways of formulating good questions and problems, but I believe this is a valuable question, I will focus my answer on the limited literature covering the formulation of good behavioral and social science questions. In general, while formulating good questions is specific to the domain or related set of domains of interest, research on ideation or idea generation is covered by the field of creativity research (but this does not cover good questions).
Research heuristics and hypothesis generation
Theory
Many scientific methodology courses focus almost entirely on study design and hypothesis testing appropriate for the field, while neglecting the critical aspects of hypothesis generation and research heuristics. Research is somewhat scarce, but some frameworks have been proposed, of which I am most familiar with McGuire's 49 research heuristics (McGuire, 1997). These heuristics are grouped into five categories, which I will try to briefly summarize.

Sensitivity to provocative natural occurrences: Systematically paying attention, making systematic observations, looking for anomalies (unexpected, nonobvious relations) and attempting to explain these.
Simple conceptual analysis: Thought and experiments conceptual analysis such as counterexamples, imagining counterfactuals, reversing causal direction and so forth on a defined set of phenomena.
Complex conceptual analysis: Deductive reasoning and generalization, convergent-divergent ideation processes and metatheorical techniques.
Reinterpretations of past research: Expert knowledge, reevaluating the scope of evidence accounted for and meta-analysis or review.
Collecting new or reanalyzing old data: Quantitative and qualitative analysis of new evidence, or novel analysis of existing evidence.

McGuire's theoretical framework is not the only one. Lubart and Getz (1998), for example, proposed three different research heuristics within the domain of psychology based on a historical analysis of psychological research. They named these the trilogy-of-mind heuristic (&quot;the mind comprises cognition, conation and affection&quot;), the emotions-as-moods (&quot;complex human emotion reduces to simple moods&quot;) heuristic, and the analysis-of-variance (&quot;the map is the territory&quot;) heuristic. Both frameworks are ultimately based on analysis of heuristics researchers have used, rather than directly investigating these heuristics' efficacy, however.
Others have questioned whether the search for such heuristics is a good idea in the first place. For example, a critique of research heuristics in a related field leveled by Anderson, De Dreu and Nijstad (2004) is the dilemma that research heuristics may &quot;routizine&quot; research. This is a possible problem, and they also report evidence for &quot;a heavy focus on replication–extension, cross-sectional designs, and a single level of analysis in innovation research.&quot; However, it is not clear that this routinization caused by research heuristics.
Evidence
Unfortunately, the neglect of hypothesis generation in scientific training is reflected in the empirical literature, if not the theoretical literature. There are no or very limited empirical research programmes or frameworks directly investigating the relative or absolute efficacy of different research heuristics or approaches, even in terms of relatively obtainable measures like impact factor or publication rate. Therefore, although I subjectively find them helpful, we ultimately cannot say with confidence whether or which of any of McGuire's (or others') heuristics are effective for formulating good questions and problems or not.
References

Anderson, N., De Dreu, C. K. W. and Nijstad, B. A. (2004), The routinization of innovation research: a constructively critical review of the state-of-the-science. J. Organiz. Behav., 25: 147–173. doi: 10.1002/job.236
Lubart, T. I. and Getz, I. (1998), The Influence of Heuristics on Psychological Science: A Case Study of Research on Creativity. Journal for the Theory of Social Behaviour, 28: 435–457. doi: 10.1111/1468-5914.00083
McGuire, W. J. (1997). Creative hypothesis generating in psychology: Some useful heuristics. Annual review of psychology, 48(1), 1-30.