Determining Statistical Significance

The first step towards meaningful, actionable results is understanding statistical significance and having confidence in test results.

Intelligems uses Bayesian statistics to analyze A/B tests. Because groups are randomized, we are able to measure significance through standard statistical methods, such as z-tests for Bernoulli distributions, such as conversion rates, and t-tests for more discrete metrics, such as revenue and profit per visitor.

As a rule of thumb, we typically need about 300 orders per group to accurately measure a 10% change in conversion rate. Once your test has reached these order volumes and has run for at least one full week, we recommend using the Stat Sig tab in the analytics dashboard in the app to get a read on confidence in the results.

See below for an example of what our statistical significance opportunity cost analysis looks like on a revenue and profit basis! On the x-axis is 'Risk Tolerance' and the y-axis 'Probability to be the best' — so the plot shows the probability that the winning group is truly the winner from a revenue perspective, or, if it’s not better, then at least not worse by more than [x]%. For example, in the below analysis, we’re 72.6% confident that the current winner of this test is a winner (or not worse by more than 1%) from a revenue standpoint and 73% confident that the current winner of this test is a winner (or not worse by more than 1%) from a profit standpoint. We usually like to aim to see this value around 90%.

Note that the stat sig tab will not contain any information until each test group in your test has had at least 100 visitors, 10 orders, and at least $1 in revenue.