Methodology & Meta-science
Is Hypothesis Testing Overused in Psychology?
A central goal of many of the open-science reforms proposed in reaction to the replication crisis is to reduce false-positive results in the literature. Very often, they assume that research consists of confirmatory hypothesis tests and that 'questionable research practices' are ways in which researchers cut corners to present evidence in favour of hypotheses that may in fact be false. Two increasingly popular methods to prevent this from happening are preregistration and Registered Reports: Both require that authors state their hypotheses and analysis plan before conducting their study, which is supposed to prevent twisting the data to fit the narrative (e.g. p-hacking) or twisting the narrative to fit the data (hypothesising after results are known). In theory, this practice safeguards the validity of inferences drawn from hypothesis tests by removing any 'wiggle room' authors could exploit to produce spurious positive results. In practice, many psychologists seem to struggle to fit their research into this revised, now much more narrow framework of confirmatory hypothesis testing: Preregistration has been accused of stifling creativity, is described as difficult even by its proponents, and analyses of published preregistrations show that most do not sufficiently restrict the above-mentioned wiggle room. I want to argue that by making the very strict requirements of a confirmatory hypothesis test so explicit, preregistration and Registered Reports reveal that hypothesis testing may be the wrong tool for a substantial number of research questions in psychology. The conflation of exploratory and confirmatory research that psychologists have been used to may have stifled the development of a framework for high-quality exploratory research, which is the necessary basis for developing hypotheses in the first place. As such, resistance against preregistration and some of the growing pains the format is experiencing may simply be a consequence of it laying bare the misfit between research goals and the excessive focus on hypothesis testing in psychology. If this is true, psychologists may be well advised to shift this focus and work towards better literacy in the exploratory ground work that precedes confirmatory hypothesis tests.