Intelligems Docs
  • Welcome to Intelligems
  • Getting Started
    • Getting Started
    • Adding Intelligems Script to your Theme
    • Updating the Intelligems Script
    • Common Use Cases
      • Price Test Common Use Cases
        • The Straddle
        • The Double Down
        • The Strikethrough
        • The Great Discount Debate
        • Savings Showdown: Volume Discount vs. Price Discount
      • Shipping Test Common Use Cases
        • The Flat Fee Face Off
        • The Threshold Trials
      • Content Test Common Use Cases
        • Landing Page Testing
        • Testing a Brand New Theme
        • Testing Different Imagery
        • Testing Cart Elements
        • Testing Announcement Bar Text
        • Navigation Menu
        • Testing Checkout Blocks
      • Offer Test Common Use Cases
        • The Volume Discount Duel
        • Gifting Games
    • Best Practices
      • 🧪Test Design Best Practices
      • ✅Best Practices During a Test
    • General FAQs
  • Price Testing
    • Price Testing - Getting Started
    • Price Testing Integration Guides
      • Integration Guide using Shopify Functions
        • Step 1: Add Intelligems JavaScript
        • Step 2: Tag product prices
        • Step 3: Update your cart
        • Step 4: QA your integration, and publish your changes
      • Integration Guide using Checkout Scripts
        • Step 1: Add Intelligems JavaScript
        • Step 2: Tag product prices
        • Step 3: Add the Checkout Script
        • Step 4: Update your cart
        • Step 5: QA your integration, and publish your changes
      • Integration Guide using Duplicate Products
        • Step 1: Add Intelligems JavaScript
        • Step 2: Tag product prices
        • Step 3: Hide duplicate products from collections pages
        • Step 4: Configure duplicate products
        • Step 5: QA your integration, and publish your changes
      • Troubleshooting
        • How to Add the data-product-id and/or data-variant-id Attribute to an Element
      • Replo Page Builder
    • How to Set Up a Price Test
    • Price Test QA Checklist
    • Starting a Price Test
    • Ending a Price Test
    • Testing Prices with Subscriptions
      • Testing Prices with Recharge 2.0 or Stay.Ai
      • How to Set Up a Price Test using Duplicate Products and Recharge Subscriptions
      • How to Set Up a Price Test using Duplicate Products and Skio Subscriptions
      • Managing Duplicate Products when Redirecting to Duplicate PDPs
    • Multi-Currency Testing
    • Price Testing FAQs
  • Shipping Testing
    • Shipping Testing - Getting Started
    • How to Set Up a Shipping Test
    • Shipping Test QA Checklist
    • Starting a Shipping Test
    • Ending a Shipping Test
    • Shipping Progress Bar Integration
    • Shipping Testing FAQs
  • Content Testing
    • Content Testing - Getting Started
      • How to Set Up a Split URL Test
      • How to Set Up an Onsite Edits Test
      • How to Set Up a Template Test
      • How to Set Up a Theme Test
      • How to Set Up a Test using our JavaScript API
    • Content Test QA Checklist
    • Ending a Theme Test
    • Content Testing FAQs
  • Personalizations
    • Personalizations - Getting Started
    • Personalization Modifications
      • Offer Modifications
      • Progress Bars
      • Offers: Integrating Widgets
      • Offers: Running a Large Number of Offer Personalizations with Shopify Functions
      • Theme Personalization Precautions
    • Targeting Personalizations
    • Targeting Modes for Personalizations
    • Previewing Personalizations
    • Testing Offer Personalizations
    • Offers Limits
    • Offer Combinations
    • Scheduling Personalizations
    • Rolling Out Tests
    • Personalizations FAQs
  • General Features
    • Targeting
      • Audience Targeting
      • Currency Targeting
      • Page Targeting
      • Mutually Exclusive Experiments
      • Targeting FAQs
    • Onsite Editor
    • Image Onsite Editor
    • CSS and JavaScript Injection
  • Analytics
    • Overview
      • How orders and sessions are attributed to experiments
      • Order and revenue accounting
      • How experiment targeting affects analytics
    • Analytics FAQs
    • Metric Definitions
      • Revenue
      • Conversion Funnel
      • Profit
      • Subscriptions
    • Filters
    • Statistical Significance
    • Timeseries
    • Custom Events
      • Overview
      • CSS Selectors
      • Scoping to specific pages
      • Custom Javascript Events
      • Testing Custom Events
      • Using custom events in experiment analytics
    • How to Add Profit to Intelligems Analytics
    • How to Add Product Groups to Intelligems Analytics
    • Tagging Orders by Test Group in Shopify
    • Exporting Data
    • Data Sharing
  • Performance Optimization
    • Site Performance
    • Optimizing Your Price-Test Integration
    • Anti-Flicker Modes
    • Edgemesh
  • Integrations
    • Google Analytics 4 Integration
    • Amplitude Integration
    • Heap Integration
    • Segment Integration
    • Heatmap Integrations
      • Integrating with Microsoft Clarity
      • Integrating with Heatmap.com
      • Integrating with HotJar
    • Navidium Testing
  • Developer Resources
    • Javascript API
      • User Object
      • Price Object
      • Campaigns Object
        • campaigns.getAll()
        • campaigns.getGWP(options)
        • campaigns.setHistoryStatus(params)
    • Intelligems Theme Snippets
    • Advanced Settings
    • Cart Permalinks
    • Targeting By Customer Parameters
    • Custom Add to Cart and Order Completed Events
Powered by GitBook
On this page
  • Why make tests mutually exclusive?
  • How to Make Tests Mutually Exclusive
  • How does Intelligems decide which visitors see which tests?
  • What about pending, ended, or paused experiments?
  • Ended Tests
  • Paused & Pending Tests
  • Caveats about Price and Shipping Tests

Was this helpful?

  1. General Features
  2. Targeting

Mutually Exclusive Experiments

Prevent visitors from being exposed to similar experiments at the same time or one after another to reduce interaction effects, minimize risks from overlapping experiences, and ensure good UX.

PreviousPage TargetingNextTargeting FAQs

Last updated 7 months ago

Was this helpful?

Note that this feature is not enabled by default. If you would like to use this feature, please contact us .

Why make tests mutually exclusive?

  • Reduce interaction effects: If someone is exposed to several experiments at the same time, it may become harder to ascertain which experience led to a change in their behavior. For example, if a visitor is exposed to one experiment that changes the copy on a button and another that repositions the button, you can’t be sure whether a change in conversion rate was caused by the first change, the second, or a combination of both.

  • Minimize UX risks: If you’re concerned about bugs or unexpected site behavior resulting from two experiments that affect the same elements or pages in your site, you may consider making them mutually exclusive.

  • Prevent users from being exposed to a test after seeing another: In case you’re running frequent tests, you may be concerned about a visitor seeing dramatic changes to colors, prices, and layouts in quick succession.

How to Make Tests Mutually Exclusive

Intelligems uses Mutually Exclusive Test Groups, or “Exclusion Groups” to ensure that tests are mutually exclusive.

  • An exclusion group holds one or more tests that will not be shown to the same visitor.

  • If a user has never been exposed to any of the active tests in the group, Intelligems will randomly place them in one of the active tests. You can also set a test to have higher probability than others of being served to a user.

  • If a user has been exposed to one of the active tests in the exclusion group, they will be kept in that test going forward - as long as it remains in the exclusion group.

  • Tests of any type and any status can be in an Exclusion Group - though the status of a test will affect the behavior of the group. See below for details.

  • A test can be in at most one exclusion group at the same time. This means there is no way for test A to be mutually exclusive with test B and C, but for B and C not to be mutually exclusive.

Follow these steps to place a test in an Exclusion Group:

  • On the Targeting page of an experiment, go to the Mutually Exclusive Tests section. Click to add the test to an existing Exclusion Group or create a new group.

  • You may do this regardless of test status, though it’s safest to create a number of tests and activate them only once they’ve been added to the same Exclusion Group.

  • When you no longer want a test to be in the group, you can remove it from the group or place it in another.

How does Intelligems decide which visitors see which tests?

A new visitor to your site will be exposed to at most one active test in an Exclusion Group. Intelligems uses one of two strategies to decide which test to show to a new visitor. You can set this option manually on each Exclusion Group at any time.

  • Option 1 - Random: a new visitor has an equal chance of being bucketed into any of the active tests. This is the default behavior that should serve well in most situations.

  • Option 2 - Weighted: a new visitor is more likely to be exposed to some tests than others. You can set the relative ‘weights’ of each test to indicate which tests are more important. This is useful in case you want specific tests to fill up with visitors first - in order to reach statistical significance faster - or if you are unsure about a test’s effects and want to ease users into it.

Weighted non-active tests: Note that, since visitors cannot be shown a paused or pending experiment, the weights shown for experiments are approximate and relative. For example, consider a group that contains the following tests:

A (active - 40% weight), B (active - 40%), C (pending - 10%), D (paused - 10%)

C and D cannot be shown to a visitor until activated, so A and B will each have a 50% chance of being shown to a visitor, rather than a 40% chance. This ensures that tests receive as much traffic as possible.

What about pending, ended, or paused experiments?

Aside from active tests, your Exclusion Groups may contain tests that are pending (not yet started), paused, or ended. Intelligems treats these as follows:

Ended Tests

New Visitors: If a group contains an Ended test, new visitors will be placed in one of the active tests in the group, as usual.

Returning Visitors: If a group contains an Ended test, Intelligems uses one of two strategies to decide what should happen to any returning visitors that have previously been exposed to that test. You can set this option manually on each Exclusion Group at any time.

  • Option 1 - Include : the returning visitor will be placed into one of the active tests (if any). This is the default behavior and helps maximize traffic to your tests.

  • Option 2 - Exclude: the returning visitor will not be placed into any of the active tests in the group, as long as those tests remain in the group. You may choose this option in order to limit the number of changes visitors see in a short period of time, or to prevent visitors from seeing a certain experience once they’ve been exposed to another in the past.

Note that, if a test ends and it’s in a group that uses relative weights, its weight is automatically set to zero. The weights of all other tests are scaled up proportionally to add up to 100%.

Paused & Pending Tests

  • New Visitors: If a group contains an Paused or Pending test, new visitors will be placed in one of the active tests in the group, as usual.

  • Returning Visitors in Paused tests: If a group contains a Paused test, returning visitors that have been exposed to that test will be excluded from all tests in the group. When the experiment is unpaused, they will once again see it.

Caveats about Price and Shipping Tests

There are a few things to consider when adding a Price or a Shipping Test to an exclusion group:

  • Price Tests: when you start a price test, Intelligems will automatically update your product prices in Shopify to the highest price in the test for each product. Therefore, any visitors excluded from the test will see the highest price on those products for the duration of the test.

  • Shipping Tests: when you start a shipping test, Intelligems will automatically remove the rates that you select to test when you start your test, and replace those with the rates configured in the test. Any visitors excluded from the test will see the rates you've set for the Control Group.

here
Making a test mutually exclusive