Results

Descriptive Statistics

(2) Below is the table that is displayed by default - it lists various statistics for the dependent variable, separately for each level of the "Split" variable. In order to obtain the boxplots below, you can tick "Boxplots" under the "Plots" tab (and "Use color palette" to spice them up a bit).

Bayesian Independent Samples T-Test - Two-Sided

(3) Below are the results for the Bayesian independent samples t-test, with a two-sided alternative hypothesis and default prior (a Cauchy distribution with the scale parameter set to 1/sqrt(2) = 0.707). The obtained 95% credible interval ranges from -1.173 to -0.006, which means that under this model, there is a 95% probability that the true effect lies between those two values. This interval does not include 0 (but it's very close!). The analysis block below shows the same analysis, but with a 99% credible interval. The 99% credible interval ranges from -1.367 to 0.162 and does include 0. The important dynamic to see here is that when you increase the percentage, the interval grows wider (just as with a confidence interval).

(4) based on the credible interval, we can conclude that there is probably a small effect of the reading activities, but there is quite a lot of undertainty about its magnitutde since the interval is quite broad (ranging from no effect, 0.006, to a very large effect, 1.173). One thing that could help here is to collect more participants, as another general dynamic of the credible interval is that it gets more narrow as more data is collected (again, just as with a confidence interval).

Bayesian Independent Samples T-Test - 99% credible

Bayesian Independent Samples T-Test - One-Sided

(5) The hypothesis of the teacher was that the reading activities would increase reading power. In order to test this hypothesis, we can use a one-sided hypothesis test. In this case, we want to test whether Control < Treatment. Since Treatment is group 2, we tick the option "Group 1 < Group 2" under "Alt. Hypothesis". This is not always clear, so when doing a fresh analysis, I would always just tick an option, look at the note underneath the table below and see if that is the alternative hypothesis you want to specify (if it's reversed, then you need "Group 1 > Group 2".

The resulting Bayes factor (BF_-0) is 4.328. This means that the data are 4.328 more likely under the alternative hypothesis than under the null hypothesis. If we compare this to the two-sided Bayes factor above, we can see that the Bayes factor has increased (almost doubled). This is because we now have a one-sided hypothesis, which makes a more specific prediction. This specific prediction turned out to be predicting the right direction of the effect (i.e., delta < 0), so it gets rewarded for its specificity. If the alternative hypothesis predicts the wrong direction, then it gets punished for it (you can change the alternative hypothesis in the settings to see the effect it has on the Bayes factor, in this case you would get BF₊₀ = 0.105 - so now the alternative hypothesis loses from the null hypothesis (because it predicted only positive values, while the observation was negative).

(6) We observe moderate evidence in favor of the one-sided alternative hypothesis, so it's probably a good idea to have all the kids do these reading activities! However, the evidence is not overwhelming, so it would not hurt to try to replicate this finding in future experiments.

(7) Below you find the robustness plot that explores the sensitivity of the Bayes factor to the prior specification.

a) the maximum Bayes factor in favor of H+ is 4.515

b) when setting the scale parameter to 0.4891

c) the default prior scale value is 0.707, so the prior distribution we used was wider (since 0.707 > 0.4891)

d) yes - the Bayes factor stays relatively stable across different scale values (i.e., it stays between 3 and 4.5). It only decreases towards 1 for scale values that lead to a very narrow prior distribution. What we see here is that as the prior grows more narrow, the alternative hypothesis starts to resemble the null hypothesis (since the null would have just a "spike" at 0), and therefore the Bayes factor converges to 1 (since then you are comparing two identical models).