Determining Statistical Significance

The first step towards meaningful, actionable results is understanding statistical significance and having confidence in test results.

Intelligems uses a Bayesian statistical model and Monte Carlo simulations to analyze A/B tests. Note that the model does not account for intra-week (daily) seasonality, or other affects that may be specific to your store. We recommend that in addition to using Intelligems' provided probabilities to determine the significance of your test, you also ensure the test reaches a pre-defined minimum number visitors, orders, and full weeks live.

As a simple rule of thumb, we typically recommend at least 300 orders per group before analyzing results (note this is just a rule of thumb, and the right number for you may vary significantly — also note that best practice would be to set a minimum number of visitors, orders, and time). Once your test has reached your pre-defined number of visitors, orders, and run for a minimum of one full week, you can use the Stat Sig tab in the analytics dashboard in the app to get a read on confidence in the results.

See below for an example of what our statistical significance opportunity cost analysis looks like on a revenue and profit basis! On the x-axis is 'Risk Tolerance' and the y-axis 'Probability to be the best' — so the plot shows the probability that the winning group is truly the winner from a revenue perspective, or, if it’s not better, then at least not worse by more than [x]%. For example, in the below analysis, we’re 72.6% confident that the current winner of this test is a winner (or not worse by more than 1%) from a revenue standpoint and 73% confident that the current winner of this test is a winner (or not worse by more than 1%) from a profit standpoint. We usually like to aim to see this value around 90%.

Note that the stat sig tab will not contain any information until each test group in your test has had at least 100 visitors, 10 orders, and at least $1 in revenue.