How to Set Up a Shipping Test
This guide will walk you through the steps to set up a shipping test in Intelligems.
Step 1: Create a new Shipping Test
Navigate to the A/B Tests tab in the menu on the left-hand side of the Intelligems app. Once there, click Create New Test above the experiments table. Select Shipping Test and Create New Test.
Step 2: Select shipping profiles and zones to test
Using the expander for each shipping profile, select the zone(s) where the Intelligems rate will be added. The Intelligems rate will only apply to the profiles and zones selected here and will apply to all orders that meet the criteria for the selected profiles and zones, subject to other cart value and weight criteria which you'll configure in the next step.
Consider carefully which profile(s) you add the Intelligems rate to. If your store's products are spread across multiple profiles, adding the Intelligems rate to a subset of profiles may lead to unexpected behavior at checkout. Learn more about how Shopify combines rates here.
Step 3: Select the rates you want to include for this test
Once you have selected the profiles and zones you want Intelligems rates to apply to, you will select which rates from those zones you want to test. These rates will be hidden during the test and replaced by an Intelligems rate, which will vary by test group.
Step 4: Enter the test details
Fill in the Test Name and Test Description for the experiment you are creating. This information is all internal - the more detail you include here the better! Tests can be live for several weeks, and your future self will thank you for including the details here.
You will also select what your primary goal is - this will not impact what data is tracked or available to view, but will allow Intelligems to display analytics so that the most important information is surfaced first.
Step 5: Create your test groups
Create between two and five groups to include in the test by clicking on the ‘+’ button. Name the groups for the experiment and use the slider to allocate what percentage of traffic will go to each group.
The more groups you have, the longer it will take to get statistically significant results. You’ll need about 300 orders for each group in the test to detect a 10% change in conversion with 90% confidence.
Step 6: Configure the Intelligems rate(s)
In this step, you will configure the calculated carrier rate to be added to the previously selected profiles and zones.
For each group, choose the rate type that most closely aligns with what you're testing (i.e. what you want to change in each test group). See below for some tips on when to select each rate type and examples of combined rates:
Legend
Flat Rate: You are testing rate(s) that are not set up with conditions in Shopify ("flat rates")
Flat Rate with Threshold: You are testing a flat rate and the threshold for free shipping
Threshold Only: You are testing a free shipping threshold but want to keep your existing rates for those that do not reach the threshold (Note: our threshold is based on pre-discount subtotal)
Tiered by Price or Weight:
Custom: Have something else you want to test? Let us know and we'll see what we can do!
Step 7: Review your rates
Use the drop-downs to review profiles, zones, and test groups.
Select profile and zone to preview
Select test group for cart preview
Input sample cart totals and weight to see what shipping options will be available at checkout
Rates displayed at checkout during the test may differ from preview if order products are included in multiple shipping profiles.
Step 8: Configure advanced settings
There are four advanced settings steps, all of which are optional to set up.
The first is Targeting. This tool allows you to apply specific conditions to certain site visitors.
You can set up currency and country targeting that allows you to limit your test to a single currency and/or a list of specific countries. This feature is defaulted to your store currency for price test.
You can use UTM parameters to customize your user experience under the Audience option.
You can filter traffic based on device type (i.e. mobile or desktop).
You can filter traffic based off of whether a visitor is new or returning.
You can learn more about targeting here!
The second advanced setting is enabling Google Analytics. Enabling this option will send all test data to your GA4 account.
The third is Find & Replace. This tool allows you to adjust site copy for site visitors based on their test group. Check out this article for more details on configuring this option.
The fourth is a Shipping Progress Bar. If you currently show a shipping progress bar on your site or if you would like to include one for your test and you are testing shipping thresholds, it is recommended to use the built-in option from Intelligems so that the bar will update with the test group. Adding the Intelligems progress bar typically requires adding a component to your theme's liquid code and customizing the bar's style in the Intelligems app.
Learn more about adding the Intelligems shipping progress bar here.
See below for a few examples of a configured shipping progress bar during a live test.
Step 9: Save your test
Once you have completed all the above steps, you will be able to save your test with the button in the bottom right. Once the test is saved, you should be able to see Intelligems as a rate carrier in your Shopify shipping settings for each profile and zone you selected in step 2. If you do not see it there automatically, you may need to click 'Add rate' and select it from the carrier options.
Don’t worry, this won’t set the test live yet and you can come back and edit if you need to make changes!
What happens next?
Now that you've created your shipping test, you can QA your test using this checklist!
Last updated
