viewof showDs = {
// a button that toggles between showing and hiding the estimates, styled like Inputs.button.
// the value is stored on a wrapping div, because a button's own value property is always a string
const button = html`<button class="btn btn-quarto">Show effect sizes and 95% CIs</button>`
const view = html`<div>${button}</div>`
view.value = false
button.onclick = () => {
view.value = !view.value
button.textContent = `${view.value ? "Hide" : "Show"} effect sizes and 95% CIs`
view.dispatchEvent(new Event("input", {bubbles: true}))
}
return view
}7 Misinterpretation methods ✎ Draft
Many problems with published research are not errors in the numbers. The numbers are right, but the conclusions drawn from them do not follow. These are failures of claim–evidence alignment, and they are some of the easiest to detect, because the evidence needed to spot them is usually printed in the article itself.
This chapter starts with three misinterpretations of p-values that are common, consequential, and checkable:
- Absence of evidence is not evidence of absence. A nonsignificant result is read as showing that there is no effect.
- The difference between “significant” and “not significant” is not itself statistically significant. One effect is significant and another is not, and this is read as showing that the two effects differ.
- Interpreting non-significant results as significant. A nonsignificant result is described, concluded from, or later cited as if it had shown an effect.
The first two errors treat the significance threshold as a verdict about the world rather than a statement about the data, and both can usually be checked by calculating one additional number from what the authors report: a confidence interval for the effect in the first case, and a test of the difference in the second. The third ignores the threshold altogether while still borrowing its authority, and checking it usually requires going back to the original results, whether in the same article or in an article that is being cited.
7.1 Absence of evidence is not evidence of absence
Absence of evidence is not evidence of absence.
Altman & Bland (1995)
7.1.1 The error
A p-value above .05 is often reported as if it showed that there is no effect: the treatment “did not work”, the groups “did not differ”, the variable “had no influence”, the two conditions were “equivalent”, or the trial was “negative”. Altman & Bland (1995) point out that the label “negative” wrongly implies that the study has shown that there is no difference, “whereas usually all that has been shown is an absence of evidence of a difference”. These are very different statements.
Recall from Section 4.2.1.1 that the p-value measures how compatible the data are with the null model. A p-value above .05 tells us that the data are not incompatible enough with the null model for us to reject it. It does not tell us that the null model is correct, because the same data are usually also compatible with many other effect sizes, including some that would be large and important. Which effect sizes the data are compatible with is exactly what the confidence interval shows. A nonsignificant result with a narrow interval around zero is informative. A nonsignificant result with a wide interval, stretching from a small harmful effect to a large beneficial one, tells us very little.
The plot below shows two hypothetical studies with exactly the same p-value, laid out as in the dance of the p-values (Section 4.4.0.1): each study’s p-value is on the right and its effect size estimate is on the left. Both studies are nonsignificant, and both would typically be reported in the same way: “there was no significant effect, p = .505”. The plot starts by showing only the p-values, and on this information the two studies are indistinguishable. Now click the button to reveal their effect sizes and 95% confidence intervals. Study A estimated the effect precisely, and its interval shows that any effect is at most very small (d between −0.06 and 0.12). Study B estimated it very imprecisely, and its data are compatible with anything from a moderate negative effect to a large positive one. The p-value is the same, but what the two studies tell us about the size of the effect is completely different. Only Study A comes close to providing evidence that there is no meaningful effect; Study B provides almost no information either way.
7.1.2 Why it matters
Most studies are too small to detect effects of the size that would matter. Altman & Bland (1995) describe a review by Freiman and colleagues of 71 “negative” trials published in the New England Journal of Medicine: only about 30% were large enough to have a 90% chance of detecting even a 50% difference in effectiveness between the treatments being compared. They also describe a trial comparing two treatments for variceal bleeding that recruited 100 patients when its own power calculation had called for 1,800. It concluded that the treatments were equally effective, even though the 95% confidence interval included differences in cure rates of up to 20 percentage points. In the most dramatic of their examples, only 5 of 24 trials of fibrinolytic treatment (mostly streptokinase) after heart attack had found a statistically significant reduction in mortality. An overview of all the trials found a clinically worthwhile 22% reduction, but the lack of significance in most individual trials had led to a long delay before this was appreciated.
The consequences are not limited to medicine. In the 1970s, many US states began to allow drivers to turn right at red traffic lights. Before-and-after studies of individual intersections and jurisdictions typically found no statistically significant increase in crashes, and this was taken to mean that the change was safe (Hauer, 2004). The individual studies were small, however. When Preusser and colleagues (1982) examined larger samples of crash data from several jurisdictions, they found that crashes involving pedestrians and right-turning vehicles at signalised intersections increased by between 48% and 107%, and those involving cyclists by between 72% and 82%. Hauer (2004) uses this case as one of his examples of “the harm done by tests of significance”.
7.1.3 How common is it?
Aczel et al. (2018) examined every nonsignificant result mentioned in the abstracts of articles published in 2015 in three psychology journals (Psychonomic Bulletin & Review, Journal of Experimental Psychology: General, and Psychological Science). In 72% of the 137 cases, the authors interpreted the nonsignificant result as evidence that the effect was absent. When Aczel et al. reanalysed the results using Bayes factors, fewer than 5% provided strong evidence for the null hypothesis over the alternative.
76–85%
of psychology articles that discussed a nonsignificant result interpreted it as evidence that the effect was absent, with no significant change between 2009 and 2021 (Murphy et al., 2025).
Murphy et al. (2025) coded the discussion sections of 599 articles in ten psychology journals. Between 54% and 62% of articles explicitly claimed that a nonsignificant result meant there was no effect in the population, while only 4–8% explicitly discussed the possibility that an effect existed but had not been detected.
7.1.4 Examples
Consider the following statements, of the kind that appear in results sections every day:
- “A t test for independent samples on object-location memory showed no significant effect of priming, t(99) = .50, p = .616, d = .12.”
- “The mean proportions of perseverative responses to these tones were very similar between participant groups … with no significant group difference, t(30) = .66, p = .51, d = .23.”
The first is phrased carefully: it reports only that the effect was not significant. The second goes further, describing the groups as “very similar”. Whether that is justified depends on what the confidence interval looks like, which the authors do not report. We return to this example below and calculate it.
A higher-profile example comes from a large online study of the “Reading the Mind in the Eyes” test, a measure of theory of mind (Baron-Cohen et al., 2015). Among typical adults, women scored higher than men on average. Among adults with autism, the authors reported that a two-way ANOVA “showed … an absence of a sex difference between males and females with autism (p = 0.907)”, and the article’s title describes a “complete absence of typical sex difference”. The p-value alone cannot support this claim. The question to ask is: do these data show that men and women with autism have equivalent average scores, or only that a difference was not detected? The answer depends on the precision of the estimate of the sex difference within the autism group, and on what size of difference would be considered meaningful. Note also that the claim that the sex difference is absent in autism but present in typical adults is a comparison between a significant and a nonsignificant effect, which is the second error covered in this chapter (Section 7.2).
7.1.5 What would count as evidence of absence?
A nonsignificant difference test cannot show that an effect is absent, but other analyses can provide evidence that an effect is absent, or at least too small to matter:
- Confidence intervals. If the whole interval lies within a range of effects that everyone would agree are trivially small, the data are informative about the absence of a meaningful effect.
- Equivalence tests. The two one-sided tests (TOST) procedure formalises this. The researcher specifies a smallest effect size of interest, which defines equivalence bounds (for example, d = −0.3 to 0.3), and then tests whether the effect is significantly greater than the lower bound and significantly less than the upper bound (Lakens, 2017; Lakens et al., 2018). If both tests are significant, the effect can be declared practically equivalent to zero. This corresponds to checking whether the 90% confidence interval lies entirely inside the bounds.
- Bayes factors, which quantify the relative evidence for the null and alternative hypotheses (Section 4.3.4), as in the reanalysis by Aczel et al. (2018).
Crossing a difference test with an equivalence test gives four possible conclusions, not two:
| Equivalence test p < .05 | Equivalence test p ≥ .05 | |
|---|---|---|
| Difference test p < .05 | An effect exists, but it is too small to matter | An effect exists and it may be meaningful |
| Difference test p ≥ .05 | Evidence of absence of a meaningful effect | Inconclusive: absence of evidence |
The plot below shows four hypothetical studies, one in each cell of this table. Each study’s estimate is shown with its 95% confidence interval, along with the p-value of the traditional difference test (is the effect different from zero?) and of the equivalence test (is the effect smaller than the smallest effect size of interest?). Try changing the equivalence bounds. Notice that Study 3 is nonsignificant under the difference test, just like Study 4, but it is compatible with everything from a moderate negative effect to a large positive one. Its nonsignificance is absence of evidence. Only Study 4 provides evidence of absence, and only relative to bounds that someone has to justify.
7.1.6 Checking for this error in trustworthiness assessment
- Find the claims of absence. Look for phrases such as “no effect”, “did not differ”, “no difference”, “unaffected”, “unrelated”, “similar”, “equivalent”, “comparable”, “failed to replicate”, “no evidence that”, and “negative trial”, particularly in abstracts and discussions, where results are summarised most loosely.
- Find the evidence for each claim. Is the claim supported by an equivalence test, a Bayes factor, or a confidence interval that excludes meaningful effects? Or only by p > .05?
- If no interval is reported, reconstruct one. A confidence interval can usually be calculated from the reported test statistic, degrees of freedom, and group sizes (see Chapter 10).
- Ask what size of effect would matter, and whether the interval rules it out. If the authors reported a power analysis, the effect size they powered the study for is a natural benchmark: if the interval includes it, the study cannot claim to have ruled it out.
- Ask whether the argument depends on the absence claim. Claims of absence are often doing important work: that groups did not differ at baseline, that a treatment had no side effects, that a manipulation did not also change mood, or that an effect did not depend on participants’ awareness. They are also often convenient for the authors, which is a good reason to check them carefully.
7.1.6.1 Worked example
Take the second statement above: t(30) = .66, p = .51, d = .23, reported as showing that the two groups were “very similar”. Assuming equal group sizes of 16, we can reconstruct the 95% confidence interval for d from the t statistic using the noncentral t distribution:
# 95% CI for Cohen's d from an independent-samples t test, via the noncentral t distribution
ci_d_from_t <- function(t, n1, n2, conf = .95) {
alpha <- 1 - conf
df <- n1 + n2 - 2
ncp_lower <- uniroot(function(ncp) pt(t, df, ncp) - (1 - alpha/2), c(t - 10, t + 10))$root
ncp_upper <- uniroot(function(ncp) pt(t, df, ncp) - alpha/2, c(t - 10, t + 10))$root
scale <- sqrt(1/n1 + 1/n2)
round(c(d = t * scale, lower = ncp_lower * scale, upper = ncp_upper * scale), 2)
}
ci_d_from_t(t = 0.66, n1 = 16, n2 = 16) d lower upper
0.23 -0.46 0.93 The data are compatible with anything from a moderate effect in one direction (d = −0.46) to a large effect in the other (d = 0.93). With 16 participants per group, the study had only about 28% power to detect a difference of d = 0.5. The groups may be very similar, but this study cannot tell us whether they are. The appropriate description is that the difference was not statistically significant and was estimated imprecisely.
7.1.7 Test yourself
A randomised trial with 20 patients per group compared a new analgesic with placebo and found no significant difference in pain scores, t(38) = 1.10, p = .28, d = 0.35, 95% CI [−0.28, 0.97]. Mark each statement as true or false.
- The trial shows that the drug does not reduce pain.
- The data are compatible with the drug having no effect on pain.
- The data are compatible with the drug having a large effect on pain.
- The trial provides good evidence that the drug and placebo are equivalent.
- To claim that the drug’s effect is too small to matter, the authors would need to specify what size of effect would matter and show that the data rule out effects that large.
7.2 The difference between “significant” and “not significant” is not itself statistically significant
The difference between “significant” and “not significant” is not itself statistically significant.
Gelman & Stern (2006)
7.2.1 The error
The second error involves comparing two results by comparing their p-values. One effect is significant and another is not, and this is read as showing that the two effects are different: the treatment worked in women but not in men; the effect was found in the original study but not in the replication; the intervention group improved but the control group did not; the effect appeared with one analysis but disappeared with another.
This does not follow. A claim that two effects differ is a claim about the difference between them, and it has to be tested directly, with a test of the difference or of the interaction. Gelman & Stern (2006) emphasise that this is not simply the familiar point that the .05 threshold is arbitrary, so that p = .049 and p = .051 are practically the same result. Rather, “even large changes in significance levels can correspond to small, nonsignificant changes in the underlying quantities”.
7.2.2 A simple example
Gelman & Stern (2006) illustrate this with two independent studies that estimate the same effect with the same standard error: 25 ± 10 and 10 ± 10. Study A is significant (p = .01); Study B is not (p = .32). The tempting conclusion is that A found an effect, B did not, and therefore the two studies found different results.
But the difference between the two estimates is 15, and the difference has its own, larger, standard error: \(\sqrt{10^2 + 10^2} \approx 14\). The test of the difference gives p = .29, which is no more significant than Study B on its own. Study A crossing the threshold while Study B does not could easily happen even if the two true effects were identical.
The plot starts with Gelman and Stern’s numbers. The key feature is the third row, which shows the difference between the two studies and its confidence interval. Whether each study’s interval excludes zero tells us nothing about whether the studies differ from each other. Only the interval for the difference tells us that. Try moving Study B’s estimate to see how far apart the two estimates have to be before the difference becomes significant, and then try shrinking both standard errors.
Gelman & Stern (2006) add a third study with a much larger sample, which estimates the effect as 2.5 ± 1.0. This study is exactly as significant as Study A (p = .01). Judged by significance alone, it “replicates” Study A. But its estimate is ten times smaller, and the difference between A and this study is significant (22.5 ± 10.0, p = .03). Its estimate is much closer to that of Study B, the “failed” study. Set Study A to 25 ± 10 and Study B to 2.5 ± 1 to see this. Comparing significance levels can therefore mislead in both directions: it can make results that do not differ look different, and results that do differ look the same.
A related point concerns overlapping confidence intervals. Checking whether two intervals overlap is not a test of their difference either. Two independent estimates with the same standard error whose 95% intervals just touch differ with p ≈ .006, and intervals that overlap moderately can still differ significantly. Researchers are not good at judging this by eye (Belia et al., 2005). When in doubt, calculate the difference.
7.2.3 Where it appears
The error can occur wherever two results are compared, including:
- Subgroups. The effect is significant in one group (women, younger adults, patients) but not another, and the authors conclude that the effect is specific to that group, without testing the interaction.
- Conditions within a study. The intervention group improved significantly from baseline and the control group did not, so the intervention is declared effective, without comparing the change between groups. Nieuwenhuis et al. (2011) illustrate this and related forms of the error.
- Predictors. Gelman & Stern (2006) discuss an influential finding that men’s sexual orientation was significantly associated with the number of older brothers but not with the number of older sisters, which was widely reported as showing that only older brothers mattered. The two coefficients were never compared, and Gelman and Stern suspect, from the published figures, that the difference between them was not significant.
- Series of estimates. In another of their examples, a study of the effects of electromagnetic fields at many different frequencies grouped the frequencies by whether their effects were significant, and interpreted the groups as reflecting different underlying processes. Plotting the estimates with their standard errors showed a fairly consistent effect across frequencies, with the apparent structure attributable to sampling variability.
- Analytic choices. An effect is significant with one outlier exclusion rule or covariate set but not another, and this is taken as showing that the choice changes the result.
- Original studies and replications, covered below.
Moran et al. (2021) provides an example of the analytic choice type, from a registered replication report of a study of whether attitudes can be learned from stimulus pairings that participants are not aware of. The answer depended on how “unaware” participants were defined. Excluding participants using the original study’s awareness criterion, the pooled effect was g = 0.12, 95% CI [0.05, 0.20]; using an alternative criterion, it was g = 0.03, 95% CI [−0.06, 0.13]. Does the exclusion criterion change the result? The two intervals cannot answer this question on their own. Answering it requires an estimate of the difference between the two effects and its uncertainty, and because the two analyses share many of the same participants, that requires an analysis that accounts for the overlap rather than the simple formula used above.
7.2.4 How common is it?
50%
of the neuroscience articles that compared two effects in a way that invited this error made it: 79 of 157, among 513 articles in five top journals (Nieuwenhuis et al., 2011).
Nieuwenhuis et al. (2011) reviewed 513 behavioural, systems, and cognitive neuroscience articles published in Science, Nature, Nature Neuroscience, Neuron, and The Journal of Neuroscience. Of the 157 articles in which the authors compared two effects in a way that could lead to the error, 78 used the correct procedure (testing the difference or interaction) and 79 used the incorrect one. An additional analysis suggested that the error was even more common in cellular and molecular neuroscience. Their article is short and has a clear illustrative figure showing the common forms of the error, making it a good companion reading to Gelman & Stern (2006).
7.2.5 Checking for this error in trustworthiness assessment
- Find the comparative claims. Look for phrases such as “but not”, “only in”, “specific to”, “selectively”, “moderated by”, “depended on”, “disappeared when”, “was eliminated”, “was stronger in”, and “failed to replicate”.
- Check whether the difference was tested. Is there a test of the interaction, the moderation, or the difference between the estimates? If the claim rests only on one p-value being below .05 and another being above, the claim is not supported as stated.
- If the estimates are independent, test the difference yourself. For two independent estimates with standard errors, the difference is tested with \(z = (b_1 - b_2) / \sqrt{SE_1^2 + SE_2^2}\). If only a 95% confidence interval is reported, the standard error can be approximated as its width divided by 3.92.
- If the estimates are not independent, for example two outcomes measured in the same participants, two coefficients in the same regression model, or two analyses of overlapping samples, this formula is not valid, because it ignores the covariance between them. Note that the difference was not tested and that it cannot be reconstructed from the reported information.
# test of the difference between two independent estimates
diff_test <- function(est1, se1, est2, se2) {
diff <- est1 - est2
se_diff <- sqrt(se1^2 + se2^2)
z <- diff / se_diff
round(c(diff = diff, se = se_diff, z = z, p = 2 * pnorm(-abs(z))), 3)
}
# approximate standard error from a 95% confidence interval
se_from_ci <- function(lower, upper) {
(upper - lower) / (2 * qnorm(.975))
}
# Gelman and Stern's example
diff_test(est1 = 25, se1 = 10, est2 = 10, se2 = 10) diff se z p
15.000 14.142 1.061 0.289 7.2.6 Implications for replication
The same logic applies to replications. It is common to judge whether a replication “succeeded” by whether it was significant, and whether it “failed” by whether it was not. This compares significance levels, and has both of the problems described above. A nonsignificant replication does not show that the original and replication results differ, and a significant replication does not show that they agree.
There is a further problem. The precision of any test of the difference between an original study and its replication is limited by the less precise of the two, which is usually the original. Morey & Lakens (2016) argue that because original studies in psychology are often so small, many published results are effectively unfalsifiable by replication: even a replication with an enormous sample cannot show that its result differs from the original’s.
Consider an original study with 10 participants per group that found p = .03, corresponding to d = 1.05, 95% CI [0.12, 1.99]. Suppose a replication with 5,000 participants per group finds d = 0.20, 95% CI [0.16, 0.24]. The replication is highly significant, so by the criterion of significance it “succeeded”. Yet its estimate is one fifth of the original, and the test of the difference between them is not significant:
diff_test(est1 = 1.05, se1 = se_from_ci(0.12, 1.99),
est2 = 0.20, se2 = se_from_ci(0.16, 0.24)) diff se z p
0.850 0.477 1.780 0.075 The replication was not the problem. Because the original’s standard error is almost 0.48, the difference between the two studies would have to be about 0.94 standard deviations before it could reach significance, however large the replication. The original study was too imprecise to tell us much about the size of the effect, and so it is also too imprecise to be contradicted. In trustworthiness assessment, this is a reason to focus on effect size estimates and their precision, and to be cautious of any conclusion, from the original authors or from replicators, that rests on whether results fell on the same side of .05.
7.2.7 Test yourself
An article reports that a memory training programme improved recall in younger adults, p = .01, but not in older adults, p = .20. The abstract concludes that “the benefits of memory training are limited to younger adults”. Mark each statement as true or false.
- The results show that the effect of training differs between younger and older adults.
- To support the conclusion in the abstract, the authors would need to test the age group × training interaction.
- If the 95% confidence intervals for the two age groups overlap, the effects cannot be significantly different.
- If the two age groups are independent samples and the article reports the estimates and their standard errors, a reader can test the difference themselves.
- If a large replication finds a significant effect of training in younger adults, it has found the same result as the original study.
7.3 Interpreting non-significant results as significant
7.3.1 The error
The first error in this chapter reads a nonsignificant result as showing that an effect is absent. This one reads it as showing that the effect is present. A result that did not meet the significance threshold, usually the authors’ own, is described, interpreted, or cited as if it had.
The error can enter at three points, and each is harder to detect than the one before:
- In how the result is described. The result is reported as nonsignificant, but in language that suggests it nearly was significant.
- In what the authors conclude from it. The result is reported as significant or as supporting the hypothesis when, by the conventional test, it was not.
- In how other authors cite it. The original article reported its results accurately, but later articles describe it as having found something it did not.
7.3.2 Euphemisms for non-significance
The most visible form is the “marginally significant” result, covered in Section 4.4.2. Matthew Hankins’ list of over 500 phrases taken from the published literature, Still not significant, is the standard catalogue: “approaching significance”, “a trend towards significance”, “teetering on the brink of significance”, and so on. Such phrases are common: around 40% of p-values between .05 and .10 in APA journals were described as marginally significant (Olsson-Collentine et al., 2019).
From the point of view of trustworthiness assessment, the euphemisms are the least of the problems, because they are honest about the p-value. Their effect is to let the authors present a result as if it were support for their hypothesis while acknowledging that it is not. A reader who notices the phrase can see exactly what was found and judge it accordingly. The more serious cases are those where nothing in the wording signals that the result was nonsignificant.
7.3.3 Non-significant results reported as support
Sometimes a nonsignificant result is reported as significant, or as supporting a hypothesis, with no qualification. This can happen in several ways:
- Undisclosed one-tailed tests. A one-tailed test halves the p-value. It can be legitimate when the direction of an effect was specified in advance, but when it is not disclosed, readers assume the conventional two-tailed test and read the result as stronger evidence than it is. When it is chosen after seeing the data, it simply moves the threshold from .05 to .10.
- Incorrectly reported p-values. The reported p-value is below .05, but the p-value recalculated from the test statistic and degrees of freedom is above it. statcheck calls these decision errors, and they are more common for p-values reported as significant than for those reported as nonsignificant (Section 10.1.1).
- Conclusions that outrun the results. The results section reports the test correctly, but the discussion or abstract describes the hypothesis as supported, or the effect as “replicated”, without mentioning that the key test was nonsignificant.
7.3.3.1 Example: the facial feedback hypothesis
Strack et al. (1988) is one of the most cited studies in social psychology, and a staple of introductory textbooks. In their Study 1, 92 participants rated the funniness of four cartoons while holding a pen either between their teeth (which forces a smile-like expression), between their lips (which prevents one), or in their nondominant hand. As predicted by the facial feedback hypothesis, the cartoons were rated as funniest in the teeth condition (M = 5.14), least funny in the lips condition (M = 4.32), and in between in the control condition (M = 4.77). The authors wrote that the results “clearly support the predictions”, and that the predicted pattern was “confirmed by a significant linear contrast, t(89) = 1.85, p = .03”.
Recalculating the p-value from the reported test statistic gives a different answer:
# two-tailed and one-tailed p-values for t(89) = 1.85
c(two_tailed = 2 * pt(-1.85, df = 89),
one_tailed = pt(-1.85, df = 89)) |>
round(3)two_tailed one_tailed
0.068 0.034 The reported p = .03 is the one-tailed p-value. We could find no statement in the article that the test was one-tailed. By the conventional two-tailed test, the result is p = .068: not significant. (statcheck would flag this result as a decision error, unless the words “one-tailed” or similar appeared in the text.)
The article’s Study 2 adds a second problem. It measured the same funniness ratings, and they did not differ between conditions (“all ps > .20”). The authors explained this by arguing that facial feedback should affect participants’ feelings of amusement, a new measure introduced in Study 2, but not their judgements of how funny the cartoons were, which was the dependent variable on which Study 1’s conclusions were based. Amusement ratings did show the predicted effect. The abstract describes Study 2’s results as having “replicated Study 1’s findings”.
None of this was hidden: every number needed to see it was in the article. But the article’s conclusions, its abstract, and its subsequent reputation are all based on the reading that Study 1 found a significant effect. When Wagenmakers et al. (2016) coordinated 17 preregistered direct replications of Study 1, involving a combined 1,894 participants, the meta-analytic difference between the teeth and lips conditions was 0.03 units on the 10-point scale (95% CI [−0.11, 0.16]), compared with 0.82 in the original. The replication effort was a response to the study’s influence, but that influence had rested from the start on evidence that was weaker than it was presented as being.
7.3.4 Non-significant results cited as significant
The final form of the error is the hardest to detect, because the article that makes it may be accurately reporting what it believes to be the literature. The original study reported a nonsignificant result accurately, but later articles cite it as if it had found an effect. Greenberg (2009) traced the citation network of a set of biomedical claims and documented several ways in which citation can create “unfounded authority” for a claim, including what he termed citation diversion: citing an article’s content but changing its meaning. Once a misdescription appears in a few influential reviews, later authors can repeat it from the reviews without reading the original.
7.3.4.1 Example: newborns’ preference for face-like patterns
Johnson et al. (1991) is a foundational study in developmental psychology, with over 2,000 citations on Google Scholar. Its first experiment replicated an earlier finding that newborns less than an hour old follow a moving schematic face further with their eyes and head than they follow a scrambled face or a blank head shape. Its second experiment, in 43 newborns, aimed to find out what about the face attracted them. It compared the schematic face with three other stimuli: a “config” stimulus of three dark squares in the positions of the eyes and mouth, the same three squares upside down (“inverse”), and a vertically scrambled arrangement of facial features (“linear”).
The results for eye movements were as follows. The schematic face was followed further than the inverse and linear stimuli (both p < .01), but only “almost significantly” further than config (p = .07). The comparisons between config and the two control stimuli, which are the comparisons that would show whether config itself was preferred, were not reported. From the figure, newborns followed the face by about 45° on average and each of the other three stimuli, config included, by roughly 38–40° (Wilkinson et al., 2025). Head-turning did not differ significantly between any of the stimuli. The authors’ own conclusion was that their results suggested “a face-like configuration of elements alone is insufficient” to explain the preference: some characteristics of the facial features also seemed to matter.
That is not how the study came to be cited. In a theoretical article published the same year, two of the same authors (Morton & Johnson, 1991) proposed that newborns have an innate face-detection mechanism, CONSPEC, and needed to specify what information it contained. They reasoned that because their experiments had not been able to distinguish the face from the config stimulus, the simplest hypothesis was that CONSPEC’s representation of a face resembles config. In other words, the nonsignificant difference between face and config (p = .07) was treated as evidence that the two were equivalent, which is the first error in this chapter (Section 7.1), and this equivalence was then used to infer that config would be preferred over controls, even though that preference was not supported in the same experiment. Note that the reasoning also inverts the roles of the stimuli: config was a candidate face-like stimulus being tested, not a control against which the face was being compared.
Wilkinson et al. (2025) conducted a systematic review and meta-analysis of studies comparing newborns’ attention to face-like and non-face-like stimuli, and traced how Johnson et al. (1991) has been cited. Of the ten most prominent review articles from 2024–2025 that cited it, nine cited it as showing a general inborn face preference, and seven attributed to it a preference for three-blob config-like stimuli. Widely used textbooks present it as showing that three-blob stimuli are sufficient to produce the preference, and many articles explicitly cite it as showing a preference for config over its inversion: a comparison it did not report, between two stimuli whose means were almost identical. A 2025 retrospective by Valenza and colleagues, quoted by Wilkinson et al. (2025), describes the second experiment as having shown “for the first time” that the configuration of features alone is sufficient to trigger the preference within hours of birth, which is the opposite of what the original authors concluded. Because config was believed to be preferred at birth, it became the standard stimulus in later research on newborns, fetuses, adults, autistic people, and computational models, and later results using it in days-old infants were interpreted as continuous with an inborn preference.
Wilkinson et al. (2025) are careful to say that their results do not show that there is no inborn face preference, and that the aim is not to fault individual authors for a misattribution that has become widespread. The point for trustworthiness assessment is that a claim’s support in the literature can be much weaker than its citation count suggests. A citation is a claim about what another article found, and like any claim it can be checked.
7.3.5 Checking for this error in trustworthiness assessment
- Look for euphemisms. Phrases such as “marginally significant”, “approaching significance”, “a trend towards”, and “did not quite reach significance” signal that a result was nonsignificant. Check whether the conclusions treat it as if it were significant.
- Recalculate the key p-values. For the results that the article’s conclusions depend on, recalculate the p-value from the test statistic and degrees of freedom, for example using statcheck (Section 10.1.1). If a reported p-value is half the recalculated two-tailed value, the test was probably one-tailed. Check whether this was disclosed, and whether the direction of the effect was specified in advance, for example in a preregistration.
- Compare the abstract and discussion with the results. Words such as “supported”, “confirmed”, “demonstrated”, and “replicated” should correspond to tests that met the threshold the authors set. Check in particular whether a later study’s nonsignificant result on the original outcome has been replaced by a significant result on a different one.
- Check the comparison that matters. Make sure the test that was significant is the test that the claim requires. In Johnson et al. (1991), the claim that config is preferred requires comparing config with the control stimuli, not comparing the face with them.
- Read the cited original. When an important claim rests on a citation, particularly to a classic or foundational study, read the original results rather than relying on how the citing article, or a review, describes them. Check what was found, in which comparison, and what the original authors concluded.
7.3.6 Test yourself
An article’s abstract states that “consistent with our hypothesis, participants in the gratitude condition reported greater wellbeing than controls”. The results section reports “t(78) = 1.70, p = .047”. The article does not mention one-tailed tests or a preregistration. Mark each statement as true or false.
- The reported p-value is consistent with a two-tailed test of t(78) = 1.70.
- The reported p-value is consistent with a one-tailed test of t(78) = 1.70.
- If the test was one-tailed, the authors should have said so, and readers should ask whether the direction was specified before the data were seen.
- A result described as “approaching significance (p = .08)” provides good support for the hypothesis.
- If a widely cited review states that a classic study found an effect, it is safe to assume that the classic study reported a significant test of that effect.