What is the name of the scientific study of hypothesis formation?
One umbrella term is causal reasoning, though this is a bit broader since there are theories of causal reasoning that are not about hypothesis testing.

A Google Scholar search for "causal reasoning psychology" generates several hundred thousand hits, and the first page is full of relevant papers.

Getting more specific, Klayman &amp; Ha (1987) wrote an influential article about the kinds of hypothesis testing you describe. Looking at the papers that cite that article might give you some additional leads into the literature.