Results

Data from Jakob et al. (2019). Collabra: Psychology, 5, 31. The authors investigated how the Hogwarts’ sorting hat test relates to some empirically established personality traits (in this example, Machiavellianism).

ANOVA

First, let's see if there are differences in Machiavellianism between the different houses using a one-way ANOVA.

As you can see in the output below, there is a significant difference in Machiavellianism between the different houses, F(3, 843) = 36.26, p < .001 $ $ (95% CI: 0.07, 0.15). Still, this does not tell us how any specific house scores on Machiavellianism compared to the others.

ANOVA - Machiavellianism
							95% CI for ω²
Cases	Sum of Squares	df	Mean Square	F	p	ω²	Lower	Upper
Sorting house	3935.101	3	1311.700	36.259	< .001	0.111	0.073	0.150
Residuals	30496.517	843	36.176

Note. Type III Sum of Squares

Descriptives

The raincloud and descriptives plots below seem to suggest that Slytherin does score higher on Machiavellianism, and we can do a formal hypothesis test for such differences using either post hoc tests (where we conduct all pairwise comparisons) or contrast analysis (where we test specific, planned comparisons).

Descriptives plots

Raincloud plots

Machiavellianism

Contrast Tables

For the contrast analysis we have different options. We can either specify three custom contrasts, where each contrast compares Slytherin to one other house, but we can also specify the weights in such a way that we have a single comparison between Slytherin on the one hand, and the other three houses combined on the other hand. Because it's just so much fun to specify custom contrasts (just make sure that the weights always sum to 0), I've done both! The results indicate that Slytherin scores significantly higher on Machiavellianism than the other three houses combined (t(843) = -9.67 , p < .001, d = -2.65), but also when compared to each house individually. For all comparisons, the effect size is quite large (and with a narrow confidence interval thanks to the large sample size).

Custom Contrast - Sorting house
		95% CI for Mean Difference							95% CI for Cohen's d
Comparison	Estimate	Lower	Upper	SE	df	t	p	Cohen's d	Lower	Upper
1	-15.923	-19.157	-12.690	1.647	843	-9.666	< .001	-2.647	-3.200	-2.095
2	-4.698	-5.915	-3.482	0.620	843	-7.582	< .001	-0.781	-0.987	-0.576
3	-4.569	-5.836	-3.303	0.645	843	-7.083	< .001	-0.760	-0.973	-0.546
4	-6.656	-7.923	-5.388	0.646	843	-10.307	< .001	-1.107	-1.324	-0.889

Custom Contrast Coefficients - Sorting house
Sorting house	Comparison 1	Comparison 2	Comparison 3	Comparison 4
Ravenclaw	1	1	0	0
Gryffindor	1	0	1	0
Slytherin	-3	-1	-1	-1
Hufflepuff	1	0	0	1

Post Hoc Tests

If we want to analyse all pairwise differences, a post hoc test (with Tukey's p-value correction) is more suitable. Based on the output below, we can see how each sorting house differs from the others. Focusing on Slytherin, we can see that it scores significantly higher on Machiavellianism compared to Ravenclaw t(843) = -7.58, p < .001, compared to Gryffindor t(843) = -7.08, p < .001, and compared to Hufflepuff t(843) = 10.31, p < .001 (note that the sign of the t-value changes depending on the reference point in the table, and how the t-statistics are identical to those from the contrast analysis). Because we look at all differences, we also see that Hufflepuff significantly scores lower than the other houses - how adorable!

Standard (HSD)

Post Hoc Comparisons - Sorting house
			95% CI for Mean Difference
		Mean Difference	Lower	Upper	SE	df	t	p_tukey
Ravenclaw	Gryffindor	-0.129	-1.542	1.284	0.549	843	-0.235	.995
	Slytherin	-4.698	-6.294	-3.103	0.620	843	-7.582	< .001***
	Hufflepuff	1.957	0.543	3.372	0.550	843	3.562	.002**
Gryffindor	Slytherin	-4.569	-6.230	-2.909	0.645	843	-7.083	< .001***
	Hufflepuff	2.086	0.598	3.574	0.578	843	3.609	.002**
Slytherin	Hufflepuff	6.656	4.993	8.318	0.646	843	10.307	< .001***

p < .01, * p < .001
Note. P-value and confidence intervals adjusted for comparing a family of 4 estimates (confidence intervals corrected using the tukey method).

Bayesian ANOVA

Model Comparison
Models	P(M)	P(M\|data)	BF_M	BF₁₀	error %
Sorting house	0.500	1.000	∞	1.000
Null model	0.500	1.508×10^-19	1.508×10^-19	1.508×10^-19	0.018

The Bayesian ANOVA results indicate very strong evidence in favor of the Sorting house model. The differences in Machiavellianism between the houses are so big, that the data are pretty much infinitely more likely under the model that includes Sorting house as a predictor, compared to the model that does not. To investigate the differences, we can look (as always..) at the descriptives plots and post hoc tests below.

Analysis of Effects - Machiavellianism
Effects	P(incl)	P(excl)	P(incl\|data)	P(excl\|data)	BF_incl
Sorting house	0.500	0.500	1.000	0.000	∞

Model Averaged Q-Q Plot

Our friend the Q-Q plot also returns in Bayesian ANOVA. Due to how the Bayesian framework works, we can even express uncertainty (credible intervals) in the Q-Q plot, to be even more nuanced about the normality assumption. Here we have so many data points though that the Q-Q plot looks OK in any case.

Post Hoc Tests

In Bayesian ANOVA, you can also get pairwise comparisons (i.e., t-tests between each condition). The correction for mutliple comparisons works a bit differently than in the frequentist ANOVA, since the Bayesian framework is not so focused on type 1/2 errors. The Bayes factors are unaffected by this correction, so the Bayes factor that compares Ravenclaw to Gryffindor (0.1014) is the same Bayes factor you would get from a t-test that compares these two conditions - here it indicates that we have moderate evidence that these two groups are similar in Machiavellianism. There are some huge Bayes factors that indicate that Slyterin differs from the other houses in their Machiavellianism.

Post Hoc Comparisons - Sorting house
		Prior Odds	Posterior Odds	BF_{10, U}	error %
Ravenclaw	Gryffindor	0.414	0.043	0.104	0.162
	Slytherin	0.414	5.298×10⁺⁹	1.279×10⁺¹⁰	1.121×10^-16
	Hufflepuff	0.414	15.453	37.308	6.415×10^-4
Gryffindor	Slytherin	0.414	1.036×10⁺⁹	2.501×10⁺⁹	5.267×10^-16
	Hufflepuff	0.414	27.153	65.554	3.821×10^-4
Slytherin	Hufflepuff	0.414	5.434×10⁺¹⁶	1.312×10⁺¹⁷	3.329×10^-24

Note. The posterior odds have been corrected for multiple testing by fixing to 0.5 the prior probability that the null hypothesis holds across all comparisons (Westfall, Johnson, & Utts, 1997). Individual comparisons are based on the default t-test with a Cauchy (0, r = 1/sqrt(2)) prior. The "U" in the Bayes factor denotes that it is uncorrected.

Descriptives

Descriptives plots

The raincloud and descriptives plots below seem to suggest that Slytherin does score higher on Machiavellianism, and in the Bayesian ANOVA we can look at the Bayes factors in the post hoc test (see above).