Interaction and effect size
size matters
work in progress / enrico toffalini @ psicostat
February 21st, 2024
Initial setup
rm(list=ls())
library(ggplot2)
library(effects)
library(effsize)
rnd = function(x,digits=2){ return(format(round(x,digits),nsmall=digits)) }
load("interaction_datasets.RData")
tsz=25Fully open code at: https://github.com/psicostat/workshops/tree/main/Interaction%20introduction%202024
WHAT’S IN A “Cohen’s f”?
Cohen’s f is often used for expressing effect size in interactions. General formula for Cohen’s f is like this:
\[Cohen's f = \sqrt{\frac{R^2}{1 - R^2}}\]
However, for interactions we should consider what is added by the interaction alone (above and beyond the main effects):
\[Cohen's f = \sqrt{\frac{R^2_1 - R^2_0}{1 - R^2_1}}\]
Cohen’s f = 0.25 (about Cohen’s f 2 = 0.06) is often taken as “medium” effect size. In fact, Cohen’s f = 0.40 (about Cohen’s f 2 = 0.15) is also taken as “medium” sometimes. Note that Cohen’s f = 0.25 requires R2 = 0.06, corresponding to about r = 0.24, while Cohen’s f = 0.40 requires R2 = 0.15, corresponding to about r = 0.39. So, it makes more sense to say that Cohen’s f = 0.25 is “medium”.
Anyways… what does that mean in actual interactions? The problem is that there are infinite cases that lead to the same effect size (e.g., Cohen’s f) in interactions.
All examples below will present about Cohen’s f ≈ 0.25 (Cohen’s f 2 ≈ 0.06).
This is also a useful resource: https://lakens.github.io/statistical_inferences/06-effectsize.html#effect-sizes-for-interactions
Example: Fully crossed interaction (2 x 2)
May happen in well-designed experimental studies, but otherwise infrequent in real life (?). Here, each main effect is small or even null, and the interaction accounts for most or all explained variance.
In model without interaction, R2 =
0.01. In model with interaction,
R2 = 0.07.
Overall, Cohen’s f = 0.25
(Cohen’s f 2 =
0.06).
But… a simpler way to understand the above effect may be via Standardized Mean Differences (SMDs, such as Cohen’s d):
the effect size (SMD) for X when W = 0 is Cohen’s d =
0.65(cell B - A).the effect size (SMD) for X when W = 1 is Cohen’s d =
-0.37(cell D - C).
Example: Treatment-like effect (2 x 2)
Imagine a situation like a treated-vs-control-group experiment with pre-test and post-test. In one condition (w = 0) x has not effect over y, while in the other (w = 1) x has a clearly visible effect over y. The interaction coefficients tells you (as usual) the difference-between-differences across conditions.
In model without interaction, R2 =
0.10. In model with interaction,
R2 = 0.15.
Overall, Cohen’s f = 0.25
(Cohen’s f 2 =
0.06).
But note how much the effects must vary across conditions to get there:
the effect size (SMD) for X when W = 0 is Cohen’s d =
0.09(cell B - A).the effect size (SMD) for X when W = 1 is Cohen’s d =
1.03(cell D - C).
…is this really a medium effect?! 😱
Example: Treatment-like effect (2 x 3)
Imagine a situation like a treated-vs-control-group experiment with pre-test, post-test, and follow-up. Not much different from before, and it’s a “familiar” situation. However, there are 2 interaction coefficients here; using the default contrasts, they tell you how the difference between levels of w (e.g., groups) varies from x = “00” to x = “01” (1st interaction coefficient), and from x = “00” to x = “10” (2nd interaction coefficient).
In model without interaction, R2 =
0.20. In model with interaction,
R2 = 0.25.
Overall, Cohen’s f = 0.25
(Cohen’s f 2 =
0.06).
But note how much the effects must vary across conditions to get there:
the effect size (SMD) at time x = “00” (cell D - A) is Cohen’s d =
0.01.the effect size (SMD) at time x = “01” (cell E - B) is Cohen’s d =
1.00.the effect size (SMD) at time x = “10” (cell F - C) is Cohen’s d =
1.04.
…is this really a medium effect?! 😱
Let’s try Superpower on this case, with some “plausible” assumptions:
# devtools::install_github("arcaldwell49/Superpower")
library(Superpower)
(design = ANOVA_design(design="3w*2b",
n=30/2,
mu=c(.00,.00,1.30,.30,1.20,.10),
sd=1,
r=.50)
)
## The design is specified as: 3w*2b
## Summary of means (mu) and standard deviations (SD)
## mu SD a b
## 1 0.0 1 a1 b1
## 2 0.0 1 a1 b2
## 3 1.3 1 a2 b1
## 4 0.3 1 a2 b2
## 5 1.2 1 a3 b1
## 6 0.1 1 a3 b2
##
## Correlation Matrix
## a1_b1 a1_b2 a2_b1 a2_b2 a3_b1 a3_b2
## a1_b1 1.0 0.0 0.5 0.0 0.5 0.0
## a1_b2 0.0 1.0 0.0 0.5 0.0 0.5
## a2_b1 0.5 0.0 1.0 0.0 0.5 0.0
## a2_b2 0.0 0.5 0.0 1.0 0.0 0.5
## a3_b1 0.5 0.0 0.5 0.0 1.0 0.0
## a3_b2 0.0 0.5 0.0 0.5 0.0 1.0pow = ANOVA_power(design)## Power and Effect sizes for ANOVA tests
## power effect_size
## anova_b 63.1 0.1812
## anova_a 98.9 0.2982
## anova_b:a 84.3 0.1894
##
## Power and Effect sizes for pairwise comparisons (t-tests)
## power effect_size
## p_a_a1_b_b1_a_a1_b_b2 4.8 0.01340
## p_a_a1_b_b1_a_a2_b_b1 99.8 1.39058
## p_a_a1_b_b1_a_a2_b_b2 14.0 0.33297
## p_a_a1_b_b1_a_a3_b_b1 99.3 1.28413
## p_a_a1_b_b1_a_a3_b_b2 7.0 0.11910
## p_a_a1_b_b2_a_a2_b_b1 91.6 1.34166
## p_a_a1_b_b2_a_a2_b_b2 20.3 0.32493
## p_a_a1_b_b2_a_a3_b_b1 88.2 1.23839
## p_a_a1_b_b2_a_a3_b_b2 8.0 0.10751
## p_a_a2_b_b1_a_a2_b_b2 75.0 -1.02109
## p_a_a2_b_b1_a_a3_b_b1 6.8 -0.09864
## p_a_a2_b_b1_a_a3_b_b2 87.0 -1.23346
## p_a_a2_b_b2_a_a3_b_b1 65.8 0.92260
## p_a_a2_b_b2_a_a3_b_b2 12.5 -0.21938
## p_a_a3_b_b1_a_a3_b_b2 82.9 -1.13140
##
##
## Within-Subject Factors Included: Check MANOVA ResultsExample: Group moderate continuous effect (2 x continuous)
Imagine studying how a relationship between x and y is different across two populations.
In model without interaction, R2 =
0.57. In model with interaction,
R2 = 0.59.
Overall, Cohen’s f = 0.25
(Cohen’s f 2 =
0.06).
But note how much the effects must vary across conditions to get there:
the x-y correlation in sample w = 0 is Pearson’s r =
0.30the x-y correlation in sample w = 0 is Pearson’s r =
0.66
…is this really a medium effect?! 😱
Example: Continuous relationship moderated by another continuous variable (continuous x continuous)
Imagine studying how a relationship between x and y is different across different levels of a continuous moderator z.
In model without interaction, R2 =
0.15. In model with interaction,
R2 = 0.20.
Overall, Cohen’s f = 0.25
(Cohen’s f 2 =
0.06).
The graphical representation above, however, is a little bit messy. We could simplify that by keeping just 3 lines at interpretable points (e.g., z-score of the moderator W equal -1, 0, +1):
Getting the estimates of the x-y relationship at given points of w, like in the previous case, can be done by looking at the coefficient “x” in the model summary, knowing that the “x:w” coefficient means how the effect of x on y varies for every unitary +1 increase of the moderator w.
Let’s have a look at the summary:
summary(fit1)##
## Call:
## lm(formula = y ~ x * w, data = dcc)
##
## Residuals:
## Min 1Q Median 3Q Max
## -3.1749 -0.6784 -0.0200 0.7015 2.7791
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 0.04000 0.04648 0.861 0.39
## x 0.33167 0.04700 7.056 6.08e-12 ***
## w 0.30607 0.04717 6.489 2.18e-10 ***
## x:w 0.25116 0.04590 5.472 7.20e-08 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 1.016 on 476 degrees of freedom
## Multiple R-squared: 0.1976, Adjusted R-squared: 0.1925
## F-statistic: 39.07 on 3 and 476 DF, p-value: < 2.2e-16In fact, y is not really on a z-score metrics here. Its residual SD that is about 1.0. Due to the very strong effects of x, w, and their interactions, the overall SD of y is 1.13.
So, the model coefficients here cannot be easily and directly interpreted in a correlation-like fashion.
Perhaps it’s better to look at the actual standardized coefficients:
lm.beta::lm.beta(fit1)##
## Call:
## lm(formula = y ~ x * w, data = dcc)
##
## Standardized Coefficients::
## (Intercept) x w x:w
## NA 0.2910461 0.2677671 0.2260456