Psychologists Tasting Beer

Bayesian Binomial Test - Two-Sided

(2) Below is the results table for the Bayesian binomial test. 42 out of 57 participants had a correct response (73.7%) and 15 out of 57 participants had an incorrect response (26.3%).

(3) In order to conduct a hypothesis test, we can specify our alternative model/hypothesis. There are two settings that determine how this model looks. First, we can specify if this hypothesis is two-sided or one-sided (and if so, which direction). This specifies which values the alternative model will bet on: if it is two-sided, it will bet on all possible values (from 0 to 1). If it is one-sided, it will only bet on values either lower than or greater than the test value. Second, we can specify the specific shape of the prior distribution. This will determine the "betting amount" that the model places on its values. For instance, if we specify a = b = 1, the prior will be uniform, and the model will bet equal amounts on each value of theta. By setting a = b = 5, the prior distribution will be more focused on the middle (0.5), which means that the model will bet more on values close to 0.5, and less on values close to 0 and 1. This setting implies a belief that the participants will perform closer to chance level (0.5), although this belief is not as strong as the null hypothesis' belief that the true value is 0.5.

(4) The observed Bayes factor is 112.646 in favor of the alternative hypothesis: the data are 112.646 times more likely under the two-sided alternative hypothesis than under the null hypothesis. This goes for both the proportion of incorrect responses and the proportion of correct responses. The reason for this is that, because we perform a two-sided test, deviations from 0.5 (the value postulated by the null hypothesis) to either side (higher or lower) lead to evidence against the null hypothesis (and in favor of the alternative hypothesis). Since the two proportions are each other's complement, both proportions are equally far away from 0.5 (both proportions have a difference of 0.237 with 0.5), and so both proportions offer equal evidence against the null hypothesis.

Bayesian Binomial Test - One-Sided

(5) In order to test whether psychologists have the ability to distinguish between the beers, we can perform a one-sided hypothesis test, where the alternative hypothesis postulates that the true proportion is greater than 0.5. In other words, it postulates that the participants performed better than chance level (if they have no ability, and would just guess, they would be correct 50% of the time).

By changing the option "Alt. Hypothesis", we specify the alternative hypothesis' direction. The corresponding Bayes factor (BF+0) is 225.258, which means that the data are 225.258 times morel likely under the one-sided alternative model than under the null model. This can be considered strong evidence in favor of the alternative hypothesis.

The Bayes factor is the ratio of the quality of the predictions made by two models (which we can express by our "betting" analogy). The alternative model predicted proportion values between 0.5 and 1 (one-sided), and placed equal bets on each of those values (uniform prior distribution). The null model went all in on the value 0.5. Since the observed proportion is 0.737, values close to it all received a boost in plausiblity (values between 0.62 and 0.82 all have a higher posterior probability than prior probability). All of these values are values that the alternative model placed bets on, so it wins pretty big (as reflected by the high Bayes factor in its favor).

(6) We can inspect the evolution of the Bayes factor as the data accumulated by looking at the Sequential analysis below. The x-axis lists the sample size, and the y-axis lists the corresponding Bayes factor. For instance, after the first 10 data points, the Bayes factor was still fluctuating around 1. At no point did it show evidence in favor of the null hypothesis. The Bayes factor first crossed the threshold of 100 when we had around 48 observations.

Bayesian Binomial Test - Confident Prior

(7) Lastly, we can look at the case where someone has a pretty strong prior belief about psychologists beer tasting ability, based on a previous experiment where they observed 10 correct responses and 1 incorrect response. We can use these previous data points in our prior distribution specification, by setting a to 10, and b to 1 (for the binomial test, this is the intuition behind these two prior settings - you can think of them as previously observed succeses and failures - in the most uninformative case you have only 1 success and 1 failure).

By setting these values, we can see impact that they have. We look only at the proportion of correct responses. First, we can see that the prior distribution (dashed line in the figure below) is very much centred to the right: it places a lot of prior plausibility on the value 1, and values close to it - this is a pretty informative/strong prior! While the observed proportion is fairly high, it is not as high as predicted by the alternative hypothesis, and so the posterior distribution is situated a bit more to the left (centering around 0.76).

The Bayes factor BF+0 is 80.288 in favor of the alternative hypothesis, so with this informed prior we still find evidence in favor of our alternative hypothesis. However, this Bayes factor is less strong than the previous Bayes factor. This means that the one-sided alternative hypothesis that spread its bets uniformly did better than the one-sided alternative hypothesis that placed most of its bets on values close to 1.