General FAQs
Plan, Capabilities & Access FAQs
How much does Intelligems cost?
Intelligems offers monthly and annual pricing options to meet your organization's needs. Prices vary depending on your store's order volume, what type of tests you'd like to run, and how long you are planning to test with us. Please see our pricing page for more information.
What eCommerce platforms does Intelligems integrate with?
Currently, Intelligems only supports Shopify stores. However, if you are on another platform and interested in using Intelligems, please reach out to our support team! We'd love to keep you updated if this changes in the future.
How long does Intelligems take to set up?
This depends on a few factors, including:
What test type or types you are planning to run
Whether you choose to integrate Intelligems on your own, or have the Intelligems team complete it on your behalf
What version of Shopify you are on and how many other apps and customizations your store has
That said, if you choose to have Intelligems do the integration, our team typically completes it within 3-5 business days, depending on backlog. If you are starting with something simple like content testing, you can get up and running within about 30 minutes!
What types of tests can I run with Intelligems?
Intelligems enables organizations to test just about anything on their Shopify store, including themes, page templates, pages, other content, discounts, product pricing, shipping and more.
The types of tests that you can run depend on the plan that you are on, so we recommend checking out our pricing page for a full look at what each plan includes. Please don't hesitate to reach out to our support team if you have any questions!
What kind of test should I start with?
The answer to this depends on your goals! If you are starting with price testing, most of our customers start off by testing the prices across their store or on their most popular product or collection as that will generate the fastest results and the most impact. When running a price test, we usually recommend testing both a lower and a higher price vs. the current price.
If you are more interested in testing around shipping, another popular strategy is to test various free shipping thresholds to see what finds the “sweet spot” of AOV and conversion rate.
You can see more test suggestions here!
Can I run multiple tests at once?
Technically, there is no limit to the number of parallel tests you can have running on Intelligems. Strategically, you run the risk of creating interference between your tests if they are running on similar things at similar times.
For example, maybe you are testing pricing on a particular product while also testing the layout of the PDP. You may see these as two separate tests, but in reality customers are being exposed to both. So you end up with a combination of experiences that you don’t have full visibility into since visitors are being randomly assigned within each test.
To control for interference, you can:
Set this up as one large test. Make a group for each combination of variables so that you can properly measure and compare (a multivariate test). You can have multiple products in the same test, which would keep users in the same group across all products being tested.
Run tests sequentially, one after another
Use our Mutually Exclusive Experiments feature (beta). If you don't see this as an option in your test setup, please contact us and we'll enable it.
EXCEPTIONS: You can run tests in parallel if there is not meaningful overlap in their impact to the customer experience.
For example, if you wanted to test a PDP layout for one product, and separately have a test running on a landing page for a completely different product and funnel - that should have minimal interference, as the customers won’t likely experience both.
Or if you ran a pricing test on women’s underwear at the same time you were testing men’s overcoats - those are likely to be pretty distinct sets of customers who won’t be exposed to both tests.
Note: We do only allow one shipping test at a time for technical reasons.
Will Intelligems continue to function if I update to Shopify's One-page checkout?
Overall, yes. Intelligems will continue to function if you update to Shopify's One-page checkout! That said, the full answer depends on the type of test you are looking at:
Price Test: this will continue to work seamlessly! If you are on Shopify Plus, please ensure that your Line Item script still includes our Checkout Script. Note that hiding the added discount on the One-page checkout won't be possible, but we can customize the discount's name to your preference.
Shipping Test: Intelligems will continue to operate without issues.
Content Test: Running content tests on the One-page checkout isn't possible as it doesn't utilize the checkout.liquid file and doesn't allow third-party alterations. However, you can continue to run content tests on the rest of your site without any issues. You can read further about this here.
Why does Intelligems need collaborator access to your store?
Intelligems may ask for Shopify collaborator access for a few reasons:
To complete the integration on your behalf. The integration requires a few theme changes or other Shopify configurations to be made, so we need access to your Shopify account in order to complete those.
To help troubleshoot. If something is not working as expected, it is helpful for our team to have access to your Shopify account to solve the issue quickly.
To take a look at your analytics. This helps us understand if there are an discrepancies with the data we have captured.
Permissions Intelligems requires:
Orders: Necessary to track test impact data such as AOV, product mix, etc.
Products: Necessary to find detail on products and variants that will be part of a test.
Apps: Necessary to access Shopify Checkout Scripts, in addition to apps that may play a role in things like managing the cart, collections pages, reviews, etc., which will need to be updated as part of a test.
Online Store Themes: Necessary to duplicate a theme and update the code to integrate a test.
Settings & Locations: Necessary to run shipping tests and install Intelligems as a third party rate carrier.
Permissions Intelligems prefers to have:
Discounts: In general, we want to be aware of what discounts may be impacting products that are in the test. Note that this is required if we are duplicating products.
Customers/Reports/Dashboards: Helpful to (a) check our data vs. historical data, and (b) confirm there are no discrepancies between the data we are presenting vs. what you see in Shopify.
Permissions Intelligems does not request access to:
Apps developed for you
Marketing / gift cards
Edit Orders / Draft Orders
How can I add or remove users from my Intelligems account?
The easiest way to access Intelligems is directly through Shopify - you can find us by clicking on Apps in the left hand navigation menu -> searching for Intelligems and selecting our app. Anyone who has the permissions to access apps in Shopify should be able to log in that way - you can add users via the Shopify admin by going to Settings -> Users and Permissions -> Add staff. Be sure to give the user permission to manage Intelligems in the App permissions section. If you need to remove users from Intelligems, you can do so by removing that permission. Please see Shopify's documentation for additional details.
In the rare case that you would like to provide a user access to Intelligems, but will not be providing them access to Shopify, please reach out to our support team here and we can help get that set up.
Testing FAQs
Can I schedule tests to start, pause or end at a specific time?
Yes! Intelligems allows you to schedule start times for all types of tests, including Price, Shipping, Content and Campaigns. Intelligems also allows you to schedule pause and end times for Content and Campaigns tests, but does not currently support scheduling pause or end times for Price or Shipping tests due to rollout requirements for ending those types of tests.
To schedule a start, pause or end time, follow these steps:
Click on the More Options menu (the three dots) next to the test you'd like to set up a schedule for.
Click "Schedule Test". If you do not see this option, please reach out to our support team here to request this feature be turned on for your account!
If you are setting up a schedule for a test that has not yet been started, this will open a modal that will ask you to confirm that you have completed the integration & QA'd your test. If you have not already done so, please be sure to complete both of these items before proceeding.
Once you have confirmed those items are complete, you will be taken to a modal where you can set up a scheduled start, pause, and/or end time.
A few things to keep in mind as you set these up:
You must schedule items for times in the future only.
Test can only be scheduled at 5 minute increments.
The times will be based off of your device's current time zone, which will also be listed at the top of the modal.
You can only schedule pause and end times for Content and Campaign tests that are either live, or that have a scheduled start time.
Once you have scheduled a test to start, pause or end at a specific time, you will see a clock symbol next to the status, and will be able to hover over that to get more information on the scheduled times like in the below screenshot.
How long does it take to run a test? When should I end my test?
Determining how long to run a test for relies on 3 major things:
How big is the effect of this change? The smaller the effect, the harder it is to detect and be certain that the result is not just noise.
How many people visit your website every day (and more specifically, how many orders are being placed in a day).
How confident do you want to be? The more confident you want to be, the more data you need.
Our rule of thumb is that if you want to detect a 10% change in conversion with 90% confidence, you need about 300 orders per group.
Smaller changes will likely require more data to pick out the signal from the noise
Larger changes will likely require less data to see a significant pattern
Here are the steps we recommend you take when your test goes live to monitor results and determine when to end the test:
Always check in on your test after ~4 hours to make sure your data is flowing - this can be a helpful time to catch any errors in the test configuration (don't launch a test end of day on a Friday, when you won't be able to monitor it)
Try not to end any tests before a full week - it helps to get an even set of weekdays in the mix for your analysis, and early results often change
Take a look at the "Trend" tab in the Intelligems analytics dashboard. Have those charts shown consistent results, or are they still varying from day to day? If there is still a lot of variability, it may make sense to run the test for a few more days.
Once a week has passed, you can check the “statistical significance” tab to get a read on the “probability to be best” for each group. Check out our article on statistical significance to understand what you are looking for here!
You should have a risk tolerance in your head.
You can wait for everything to hit 95 or 99% confidence, but for smaller brands, that means a lot of time waiting around for results. Waiting around presents a big opportunity cost since you are not running other tests - which could be delivering more value! - during that time
We wouldn’t recommend making decisions with less than 75% confidence
Many of our customers look for confidence somewhere around 90%
Some tests will not ever hit confidence - they’ll waffle around in the 40-60% range - that means there is no real difference between the versions. At that point, pick one to move forward with based on intuition or other strategic considerations, and move on
Pro tip: you’re able to change traffic allocation in the middle of a test. So if you have one group that is a clear loser, you can “shut it off” in the middle of your test and send more traffic to the more viable options.
How do I edit something in a live test?
There are a few steps to safely edit your live test:
Step 1: Pause your test.
The pause button can be found next to the status column within your A/B Tests overview tab.
If you are pausing a shipping test, we will automatically restore your shipping profiles while the test is paused.
If you are pausing a price test, we will ask you to select which prices you would like to roll out while the test is paused.
Step 2: Make your edits.
The edit button can be found under the three dots to the right of your test. Select this and make any changes you need, such as adding a new product, removing a test group (you can do this by allocating 0% of traffic to that group), changing a price, or changing a shipping rate. Make sure you click through the entire edit flow and select the save button at the end to save your changes!
Step 3: Resume your test.
Now that you've made your edits, you can resume your test and keep getting results! The button to do so is right next to the test status.
Keep your edits in mind when looking at the analytics dashboard if the change may have affected the results. Once a test has started, test group names cannot be changed and you can no longer add/remove test groups, but you can set traffic to 0% for a group to effectively remove that group from the test. We do this so that analytics are maintained for that old group that now has no traffic going to it. You can filter your results by date to see the results before and/or after edits were made.
How does the test status work in the test dashboard?
There are several different statuses that a test can have:
Pending: You've set this test up, but have not yet started it.
Gathering Data: You've started this test within the last 21 days.
Ready For Analysis: This test has been running for at least 21 days. While this is a good baseline, there is ultimately no general rule on time. That said, we often start seeing significant results after about 300 orders per group.
Paused: You've paused this test; it is not active on the site, but you can resume it.
Ended: You've ended this test.
What will my orders look like in Shopify?
Customers on your site will not see anything to indicate that an Intelligems price test is running, but there are three ways that Intelligems will show up on orders within your Shopify admin portal:
Each visitor gets tagged with a unique id that sorts them into a group. This id is passed to the order via cart.attribute for tracking purposes and will show under “Additional Details” on the order page. This is hidden from the customer.
In cases where we use a checkout script, we add a line_item.property to each item so that the checkout script provides the appropriate price.
The checkout script applies a “line-item discount” in order to generate the correct price for a customer’s product. While this shows as a discount in the admin view, this is always hidden from the customer (i.e. in this case they would just see “$13.99” for the product price).
Points 2 and 3 above only apply if a price test is implemented using checkout scripts, which are used for Shopify Plus customers only.
Preview FAQs
Can I preview a Shopify theme and Intelligems test at the same time?
Yes, you can! This is a very common practice we use here at Intelligems. You can do this by entering preview mode for your theme, and then entering preview mode for your test. The theme should be cached when you open your test preview.
Traffic FAQs
How does Intelligems split the traffic and keep users in the same test group for the entirety of the test?
All of our tests are run as true split tests where we segment your traffic in real time between test variants. Intelligems randomly assigns users to a test group when they visit your site for the first time during a test.
Intelligems then saves a unique string of numbers and letters know as the Intelligems ID in a first-party cookie, which we then use to remember your site visitors and keep them in the same test group(s) for the entirety of your test(s). Because of this, every time they come back to the site on the same browser on the same device, the user will have a consistent experience.
Cookies do expire - when they expire depends somewhat on the browser, since different browsers have different max allowed cookie ages. For example, on Chrome, our cookie life is about 1 year. On Safari (including all iOS browsers), cookies are deleted if the user does not visit the site for 7 days.
How does Intelligems handle customers who switch devices?
Intelligems randomly assigns users to a test group when they visit your site for the first time during a test. We then use a cookie to ensure they remain in the same test group every time they come back to the site on the same browser and device.
When a user visits your site on a new device, we cannot rely on that cookie to place them in the same group, so there is a chance they will end up in a different group. That said, in our testing, we have found that less than 1% of typical site traffic may be impacted by switching devices during a test.
We recommend making your customer support team aware of any tests you may be running, and in the rare case that a customer does switch devices and notices a differing price, having a plan of action (such as a discount code) to create a positive customer experience.
Targeting FAQs
Can I filter out blog traffic from my test(s)?
Yes! Some brands use their blog pages for SEO. This generates a lot of traffic for their site, but does not result in many conversions or revenue. Because of this, blog traffic can create a lot of noise in analytics, therefore it might be better to filter blog traffic out of your tests to keep your results clean and actionable. If you do not have any UTMs set up for your blog pages, this article will walk you through how you can exclude that traffic from your test.
Step 1: Add snippet to theme.liquid file.
Add the below code to your theme.liquid file directly above your Intelligems JavaScript snippet. An example Intelligems JavaScript snippet is included at the bottom of the snippet below to illustrate where the new snippet should be placed.
<script>
if (window.history.pushState && window.location.pathname.includes('blog')) {
const newURL = new URL(window.location.href);
newURL.search = '?blog=1';
window.history.pushState({ path: newURL.href }, '', newURL.href);
}
</script>
<script src="https://cdn.intelligems.io/<your_customer_id>.js"></script>Step 2: Set up UTM filter.
Set up the below Audience Targeting on your test. Note that this will need to be added to each test that you want to exclude blog traffic, and will only exclude that traffic moving forward.
How can I force myself into a specific test group?
You can force yourself into a specific test group by adding the below to the end of your site's URL:
/?igTg=TEST-GROUP-ID
where TEST-GROUP-ID is the ID for the test group you would like to be forced into. You can find the test group ID by heading to the A/B Tests tab in the Intelligems app, clicking on the three dots / more options menu next to the test you are working on, and selecting "Show Info". This will bring up the experiment ID, as well as the test group ID for each test group. Click on the long ID for the group you'd like to be forced into to copy it to your clipboard.
The final results should look something like this:
www.mywebsite.com/?igTg=44bae2e6-dbc3-4fc1-a68d-4218ac04f99c
Will Intelligems remove UTM parameters when a visitor lands on my site and is part of a test?
No, UTM parameters won't be removed from your URLs by Intelligems! While there are a few test types that involve redirects, such as Theme, Template and Split URL Tests, URL parameters are persisted, and therefore traffic attribution is preserved.
How do I target or exclude new vs. returning customers?
Targeting new vs. returning customers is a bit tricky because Shopify does not share whether a customer is new or returning with Intelligems. There are, however, a few ways you can set up audience targeting to help target or exclude new vs. returning visitors.
The first option is to use UTM parameters. In order to target only new visitors, we typically recommend applying the test to users visiting the site through specific UTM or media campaigns that you use for targeting new customers. There is a small chance that returning customers could come in through these campaigns.
The second option is to use our New / Returning Visitor targeting. It is important to note that this is looking at whether someone is a new or returning visitor, and not necessarily whether they are a new or returning customer. It is also important to note that we determine whether a visitor is new or returning based off of whether they have an Intelligems ID assigned to them. We assign all site visitors an Intelligems ID when they come to your site, as long as our JavaScript is in your live site. Because of this, if you have just started working with Intelligems, most visitors will be seen as new since they don't have an Intelligems ID assigned to them yet.
Both of these options can be set up in the Targeting step of test set up, which you can read more about here.
Uninstalling FAQs
Can I remove the Intelligems script once my test is complete?
While there are technically no issues with removing the Intelligems script from your site once your test is complete, we only recommend doing so in the rare case that the site is experiencing performance issues. Otherwise, we recommend leaving the lightweight script installed in your theme as testing shouldn't be one-and-done. We find that stores that approach testing with a roadmap and plan are the stores that unlock the most value. If you are stuck on coming up with an idea on what to test next, check out this article!
Additionally, keeping the Intelligems script installed on your store allows Intelligems to continue assigning users an Intelligems ID - this will make New vs. Returning Visitor targeting much more powerful.
If you do need to remove the Intelligems script from your site, delete or comment it out in your theme.liquid file. For Shopify Plus customers, this may also be located in your checkout.liquid file. The JavaScript to remove or comment out will look something like this:
<script src="https://cdn.intelligems.io/<your_customer_id>.js"></script>How do I uninstall Intelligems?
✨ We'd hate to see you go! If you plan on testing again any time in the future, we recommend moving to a Pause Plan instead of uninstalling - this will allow you to keep the integration live so it's easy to start a new test down the road, as well as maintain access to the analytics from any tests you've already run. If this sounds like a better option, please reach out to sales@intelligems.io to get set up with a Pause Plan.
Step 1: Stop any tests.
Confirm that you've stopped any tests or campaigns from within the app! You'll want to make sure you also rolled out the winners while doing this - see more details on how to end a Price Test and a Shipping Test at each of those links.
Step 2: Uninstall the app via Shopify.
Follow Shopify's steps here to uninstall the Intelligems app - this will cancel billing.
Step 3: Remove Intelligems JavaScript.
During the integration, you or the Intelligems team added JavaScript as a source into your theme.liquid file - you can now remove the Intelligems script from your site. To do so, delete or comment it out in your theme.liquid file. For Shopify Plus customers, this may also be located in your checkout.liquid file. The JavaScript to remove or comment out will look like this:
❗Note that if you do not complete this step, we will continue to append an igId onto your orders in the 'Additional Details' section in Shopify. Customers will not see this, and it will not cause any issues, but may cause confusion for your team if they do not know where it is coming from!
Step 4: Remove Checkout Script or manage Duplicate Products, if applicable.
❗This step is only applicable if you have run Price Tests!
If you are on Shopify Plus and ran your test using a Checkout Script, you can now stop that script by either unpublishing the entire Checkout Script (if nothing else is in it), or by removing the Intelligems script if you use the Checkout Script for anything else.
If you are not on Shopify Plus and ran your test using Duplicate Products, confirm those products are archived. See more details on managing Duplicate Products from previous tests here.
Step 5: Remove Intelligems as a shipping rate provider, if applicable.
❗This step is only applicable if you have run Shipping Tests!
Confirm that Intelligems is removed as a shipping rate provider. You can check this by going to your Settings in Shopify → Shipping and delivery → Manage by General Shipping Rates. Once there, confirm that 'Intelligems Shipping (Rates provided by app)' is not listed anywhere. If it is, please delete it using the three dots to the right and add your own rate to replace it if needed.
Step 6: Send us your feedback!
We are always looking to make improvements to our platform and processes! If there is something we could have done better, you have feature requests, or you just want to chat all things ecomm, please don't hesitate to reach out to support@intelligems.io.
Last updated
