The cult of statistical significance. Hidden random uncertainties in science, their roots in human mind, and unexpected outcomes 1400-236KIS

This lecture explores the common practice of reducing statistical inference to tests of statistical significance, its possible causes, and its negative consequences for science. Referring to selected simple statistical methods, the lecture demonstrates the limited information provided by statistical testing and its overestimation. Simultaneously, various common manifestations of misconceptions about statistical significance or its absence are analyzed. The question of their causes leads to considerations of the functioning of the human mind, including in an evolutionary context. The question of their consequences leads to consideration of the undoubtedly significant role of the statistical testing paradigm in the increasingly perceived crisis of reproducibility in science. Selected statistical methods are considered not so much in the context of their mathematical foundations, but primarily in terms of the nature of the answers they provide about the studied fragments of reality. Parametric and nonparametric methods for one and two samples, analysis of variance in single and double classification, frequency analysis methods, correlation analysis, linear regression, and logistic regression, and examples of multivariate methods are included. Particular attention is devoted to the analysis of interactions, as one of the most neglected topics in textbooks. The discussion of these methods leads to consideration of more general issues and questions that occupy a significant portion of the lecture, such as:
(1) The most important arguments in the decades-long debate on the usefulness of significance testing – from acceptance, through recognition of its necessity, to the demand for its elimination from scientific practice. The power of statistical tests. Widespread but rarely respected recommendations regarding sufficient statistical power. The paradox of excessive power, resulting from misunderstandings of tests. The power of tests and the precision of estimates.
(2) The contemporary paradigm of statistical significance testing as a surprising product of cultural evolution – a hybrid of various concepts whose authors would likely disagree with the current result.
(3) What is a negative result? Inference from ignorance, the ubiquity of Type II errors. When is a negative result a result, and when is it a lack of a result? Knowing that there are no significant effects can be important knowledge; sometimes it is precisely what is needed. Important lessons from (bio)equivalence testing methods. The acceptance of null hypotheses, contrary to general principles, as a common element of statistical procedures, has a serious negative impact on the validity of inferences. The extreme consequences of John Ioannidis's views: when is advanced research equipment reduced to the role of the most expensive random number generator?
(4) The "hunt" for statistical significance, the so-called "torturing of data," and sometimes even the "torturing of reality." Interpretations of "almost significant" results: what might underlie them? The traditional distinction between statistical significance and substantive significance and its incompleteness from the perspective of controlling the risk of bias.
(5) Statistical significance perceived—absurdly—as a dimension of reality. Elements of analysis of the language used in describing the statistical analysis of research results. Confusion between statements about samples and statements about the population. Figures of speech that avoid not only quantitative questions but also meeting the universally required criterion of statistical significance. How seriously do researchers take statistical inference? Researchers' expectations of statisticians, the actual place of statistical inference in research processes. Statistical significance, empathy, and the ethics of data analysis.
(6) Diminishing effects in subsequent studies of the same phenomenon. The so-called significance filter and overestimation of effects, the "winner's curse." Corrections for multiple testing, differences in statisticians' views on the appropriateness of their use, the frequency of false discoveries, and selective presentation of results.
(7) How strong do we want certainty, and what should it refer to? The risk of Type I and Type III errors (directional, Type S) and confidence levels. Jerzy Neyman's mistake, considered the father of confidence intervals, in justifying the need for them – do researchers really want interval estimates? The earlier history of confidence intervals, the recurring, ineffective calls for the use of interval estimation, the so-called "statistical reform" and the "new statistics." The earlier history of confidence intervals, the recurring, ineffective calls for the use of interval estimation, the so-called "statistical reform" and the "new statistics".
(8) Two important obstacles to interval thinking: uncertainty aversion and excessive optimism. Psychological aspects and attempts at evolutionary explanations. The law of large numbers and belief in the "law of small numbers" (sensu Tversky and Kahnemann). Behavior in the face of uncertainty, L. Savage's Sure Thing Principle and its frequent violations. Depressive realism and the evolution of excessive optimism. Optimism as an individual's strategy or as an individual's implementation of an evolutionary strategy into which it happens to be embedded. From what perspective is the "obsession with averages" a mistake, and from what perspective is it not?
(9) The reproducibility and repeatability crisis in science (irreproducibility crisis), its publicity over the past two decades, and initiatives undertaken to understand the problem. Reproducibility and repeatability in metrology versus the less precise understanding of these terms in pure science. When is the result of a previous experiment considered to be replicable? The role of the statistical testing paradigm in the repeatability crisis. The cult of statistical significance as a phenomenon that distracts from important questions. Selected attempts to improve the situation and the related controversies. Some computational aspects of the discussed approaches to statistical inference are the subject of a separate course, "Revealing Statistical Uncertainties Hidden in Research Results."
Warning: Recognizing the scale of statistical uncertainty can discourage the mechanical and thoughtless application of many common procedures.