Intelligems Docs
  • Welcome to Intelligems
  • Getting Started
    • Getting Started
    • Adding Intelligems Script to your Theme
    • Updating the Intelligems Script
    • Common Use Cases
      • Price Test Common Use Cases
        • The Straddle
        • The Double Down
        • The Strikethrough
        • The Great Discount Debate
        • Savings Showdown: Volume Discount vs. Price Discount
      • Shipping Test Common Use Cases
        • The Flat Fee Face Off
        • The Threshold Trials
      • Content Test Common Use Cases
        • Landing Page Testing
        • Testing a Brand New Theme
        • Testing Different Imagery
        • Testing Cart Elements
        • Testing Announcement Bar Text
        • Navigation Menu
        • Testing Checkout Blocks
      • Offer Test Common Use Cases
        • The Volume Discount Duel
        • Gifting Games
    • Best Practices
      • 🧪Test Design Best Practices
      • ✅Best Practices During a Test
    • General FAQs
  • Price Testing
    • Price Testing - Getting Started
    • Price Testing Integration Guides
      • Integration Guide using Shopify Functions
        • Step 1: Add Intelligems JavaScript
        • Step 2: Tag product prices
        • Step 3: Update your cart
        • Step 4: QA your integration, and publish your changes
      • Integration Guide using Checkout Scripts
        • Step 1: Add Intelligems JavaScript
        • Step 2: Tag product prices
        • Step 3: Add the Checkout Script
        • Step 4: Update your cart
        • Step 5: QA your integration, and publish your changes
      • Integration Guide using Duplicate Products
        • Step 1: Add Intelligems JavaScript
        • Step 2: Tag product prices
        • Step 3: Hide duplicate products from collections pages
        • Step 4: Configure duplicate products
        • Step 5: QA your integration, and publish your changes
      • Troubleshooting
        • How to Add the data-product-id and/or data-variant-id Attribute to an Element
      • Replo Page Builder
    • How to Set Up a Price Test
    • Price Test QA Checklist
    • Starting a Price Test
    • Ending a Price Test
    • Testing Prices with Subscriptions
      • Testing Prices with Recharge 2.0 or Stay.Ai
      • How to Set Up a Price Test using Duplicate Products and Recharge Subscriptions
      • How to Set Up a Price Test using Duplicate Products and Skio Subscriptions
      • Managing Duplicate Products when Redirecting to Duplicate PDPs
    • Multi-Currency Testing
    • Price Testing FAQs
  • Shipping Testing
    • Shipping Testing - Getting Started
    • How to Set Up a Shipping Test
    • Shipping Test QA Checklist
    • Starting a Shipping Test
    • Ending a Shipping Test
    • Shipping Progress Bar Integration
    • Shipping Testing FAQs
  • Content Testing
    • Content Testing - Getting Started
      • How to Set Up a Split URL Test
      • How to Set Up an Onsite Edits Test
      • How to Set Up a Template Test
      • How to Set Up a Theme Test
      • How to Set Up a Test using our JavaScript API
    • Content Test QA Checklist
    • Ending a Theme Test
    • Content Testing FAQs
  • Personalizations
    • Personalizations - Getting Started
    • Personalization Modifications
      • Offer Modifications
      • Progress Bars
      • Offers: Integrating Widgets
      • Offers: Running a Large Number of Offer Personalizations with Shopify Functions
      • Theme Personalization Precautions
    • Targeting Personalizations
    • Targeting Modes for Personalizations
    • Previewing Personalizations
    • Testing Offer Personalizations
    • Offers Limits
    • Offer Combinations
    • Scheduling Personalizations
    • Rolling Out Tests
    • Personalizations FAQs
  • General Features
    • Targeting
      • Audience Targeting
      • Currency Targeting
      • Page Targeting
      • Mutually Exclusive Experiments
      • Targeting FAQs
    • Onsite Editor
    • Image Onsite Editor
    • CSS and JavaScript Injection
  • Analytics
    • Overview
      • How orders and sessions are attributed to experiments
      • Order and revenue accounting
      • How experiment targeting affects analytics
    • Analytics FAQs
    • Metric Definitions
      • Revenue
      • Conversion Funnel
      • Profit
      • Subscriptions
    • Filters
    • Statistical Significance
    • Timeseries
    • Custom Events
      • Overview
      • CSS Selectors
      • Scoping to specific pages
      • Custom Javascript Events
      • Testing Custom Events
      • Using custom events in experiment analytics
    • How to Add Profit to Intelligems Analytics
    • How to Add Product Groups to Intelligems Analytics
    • Tagging Orders by Test Group in Shopify
    • Exporting Data
    • Data Sharing
  • Performance Optimization
    • Site Performance
    • Optimizing Your Price-Test Integration
    • Anti-Flicker Modes
    • Edgemesh
  • Integrations
    • Google Analytics 4 Integration
    • Amplitude Integration
    • Heap Integration
    • Segment Integration
    • Heatmap Integrations
      • Integrating with Microsoft Clarity
      • Integrating with Heatmap.com
      • Integrating with HotJar
    • Navidium Testing
  • Developer Resources
    • Javascript API
      • User Object
      • Price Object
      • Campaigns Object
        • campaigns.getAll()
        • campaigns.getGWP(options)
        • campaigns.setHistoryStatus(params)
    • Intelligems Theme Snippets
    • Advanced Settings
    • Cart Permalinks
    • Targeting By Customer Parameters
    • Custom Add to Cart and Order Completed Events
Powered by GitBook
On this page

Was this helpful?

  1. Analytics

Statistical Significance

PreviousFiltersNextTimeseries

Last updated 18 days ago

Was this helpful?

Intelligems uses a Bayesian statistical model and Monte Carlo simulations to analyze A/B tests. Note that the model does not account for intra-week (daily) seasonality, or other store-specific factors. We recommend that in addition to using Intelligems' provided probabilities to determine the significance of your test, you also ensure the test reaches a pre-defined minimum number visitors, orders, and full weeks live.

As a general rule, we typically recommend at least 300 orders per group before analyzing results (note this is just a rule of thumb, and the right number for you may vary significantly. Also note that best practice would be for you to set a minimum number of visitors, orders, and time).

Once your test has reached your predefined number of visitors, orders, and run for a minimum of one full week, you can use the Stat Sig tab in the analytics dashboard in the app to get a read on confidence in the results.

Probabilities and confidence intervals

Within Intelligems analytics, you’ll see a few metrics derived from this statistical model:

  • Probability to beat control: this is the probability that a test group is better than the control group, on the basis of the selected metric

  • Probability to be best: this is the probability that a test group is best, on the basis of the selected metric. In tests with a control group and only one other test group (i.e., A vs. B), the probability to be best will equal the probability to beat control

  • Uplift confidence interval: this is a 95% confidence interval around the uplift percent (i.e., what percent better or worse a test group is vs. control for a specific metric). For example, in this screenshot, the uplift for profit per visitor is +3.50% +/- 2.50. This means the 95% confidence interval around profit per visitor uplift is [+1.00%, 6.00%], that is, there is a 95% probability that the “true” profit per visitor difference is between +1.00% and +6.00%.

  • Value confidence interval: this is a 95% confidence interval around the value measurement for a specific metric. There is a 95% probability that the “true” value of the metric for the test group lies in this interval.

As your experiment accumulates more data, the confidence intervals around uplift and value will narrow. Note that your experiment may reach statistical significance (i.e., the probability to be best or beat control is above a threshold, like 95%), but the confidence interval around uplift and value may still be very wide. This is because for a test group to have a high probability to be best/beat control, we only need high confidence that the uplift for that test group is > 0%. We may have high confidence that one test group is better than another, but still need to collect more data to have a good specific estimate of how much better.