Stat Sig

The first step towards meaningful, actionable results is understanding statistical significance and having confidence in test results. This tab provides charts to help you understand which group is a 'winner' and how confident you can be in that. To learn more on stat sig, check out this article.

Revenue per Visitor Probability to be Best: The percent probability that each group is the best performing group on a revenue per visitor basis.

Profit per Visitor Probability to be Best: The percent probability that each group is the best performing group on a profit per visitor basis.

CVR Probability to be Best: The percent probability that each group is the best performing group on a conversion rate basis.

AOV Probability to be Best: The percent probability that each group is the best performing group on an average order value basis.

Revenue Opportunity Cost to be Best: On the x-axis is % Revenue per Visitor Opportunity Cost and the y-axis % Probability to be Best — so the plot shows the probability that the winning group is truly the winner from a revenue perspective, or, if it’s not better, then at least not worse by more than [x]%. You can switch test groups using the drop down in the top right.

Profit Opportunity Cost to be Best: On the x-axis is % Profit per Visitor Opportunity Cost and the y-axis % Probability to be Best — so the plot shows the probability that the winning group is truly the winner from a profit perspective, or, if it’s not better, then at least not worse by more than [x]%. You can switch test groups using the drop down in the top right.

Note that the stat sig tab will not contain any information until each test group in your test has had at least 100 visitors, 10 orders, and at least $1 in revenue.