# Interpretation methods <span class="badge badge-draft1">✎ Very rough draft</span>
```{r}
#| include: false
if(file.exists("../_setup.R")){source("../_setup.R")}
```
## Interpretation of *p*-values
- Intuitions for *p*-values
- Alpha, power, diagnosticity refreshers
- Distribution of unbiased *p*-values under null/alternative
- *p*-hacking and its impact on *p*-values and effect sizes
- Methods that aren't useful for trustworthiness assessment but are good for intuitions: distribution of biased *p*-values, *p*-curve, *z*-curve
- Too good to be true in the context of power vs significance rate
- Peeking and trends towards significance
## Misinterpretation of *p*-values
- Non-significant *p*-values as evidence for the null
- Difference between significant and non-significant is not itself significant
- Simply claiming differences with non-significant *p*-values
## Misinterpretation of specific tests
- Analysis-claim misalignment in general; most of the below are a specific class of this
- Any test that exists can and will be misaligned somewhere
- Nature and magnitude of misalignment, e.g., CLPM lends itself to causal interpretation but can't give it
- Goals confusion gives rise to misalignment: description, prediction, influence
- The demand for tests is understandable; "Hot Right Now" stats that are poorly understood as a general problem
### Efficacy claims
- Efficacy claims from anything other than a post-intervention comparison with a control group (pre-post, two interventions, etc.)
### MANOVA
- The "Protected F-test" logic: the belief that a significant omnibus MANOVA "protects" subsequent univariate ANOVAs from Type I error inflation
### Mediation and PROCESS macro
- Mediation
- PROCESS macro — "That's a lot to process"
### CLPM and variants
- Cross-lagged panel models and causal interpretation limitations
### Bifactor models in SEM
- Real-world data rarely perfectly follows proportionality constraints, so the less-restricted bifactor model will statistically outperform the higher-order model in nearly every CFA application, regardless of the "true" structure
- Bifactor model is more robust to minor misspecifications because its direct paths to items allow it to "soak up" variance that a more constrained higher-order model cannot
### ANOVA
- Hidden multiplicity in ANOVA
- Lack of power in higher order interactions (problems with both false positives and false negatives)
### Other commonly misused methods
- Stepwise regression
- Post hoc power analysis
- "Little Jiffy" Factor Analysis (PCA with Varimax & Kaiser Criterion of Eigenvalues > 1; default in SPSS)
- Choosing covariates based on significance or univariate associations
- Latent class growth models (note: very different from latent growth curve models)
- Magnitude-Based Inference
- ANCOVA applied to observational data — Lord's paradox
- Propensity Score Matching
- Partial Least Squares SEM
- Statistical confusion of assumption and inference / affirmation of the consequent in analyses
## Easter egg research
- Creating groups via clustering then testing for differences between them
- Assuming differences between groups and then reifying the groups
- K-means or Latent Class Analysis to create groups, followed by t-tests/ANOVA on the same variables to "validate" the groups
- Analyses that assume the number of clusters and then discover them (e.g., Latent Profile Analysis)
## Conditioning on post-treatment effects
- Common cause problems
## Causal claims, implications, and assumptions
- Causality tests don't exist, only causal assumptions
- Causal assumptions do exist and should be clarified
- Causal claims in non-experiments
- Causal claims in experiments that aren't experiments: pre-post change, follow-up comparisons against LOCF
- The litmus test of "is this only interesting if it's causal, yet it's described as non-causal evidence"
## Other interpretation issues
- Confusing IV and DV
- Table 2 fallacy