| Case 1 | Case 2 | |
|---|---|---|
| expected value | 2.1% | 3.3% |
| 65.0% credible interval | [1.6% , 2.4%] | [2.9% , 3.6%] |
| 85.0% credible interval | [1.4% , 2.7%] | [2.7% , 3.9%] |
| 95.0% credible interval | [1.2% , 3.0%] | [2.5% , 4.1%] |
Given a finite amount of trials the conversion rate can never be known exactly. But we can use statistical inference to estimate it. This tool uses Bayes Theorem to compute the probability distribution (PDF) of the conversion rate given the data (number of trials and conversion). This is shown in the graph above, along with the expected value (dashed line) and credible intervals (bars below the graph). Credible intervals are the Bayesian version of confidence intervals (here is a comparison) and can be interpreted as a measure of uncertainty about the conversion-rate.
For example, if you compare two versions of a web-site (A/B-Test) and the 95% credible intervals do not overlap, you can be quite sure that the two versions have indeed distinct conversion-rates (85% and 65% give you less confidence). On the other hand, if these intervals overlap considerably, you need to gather more data to make a well informed decision.
The computations are performed with a uniform prior and a binomial likelihood. Under this assumptions the posterior PDF is a beta distribution. The assumption of a uniform prior is not perfect and might be removed in the next version, contact the author if you want this.