Content Testing FAQs
Why do I see the Shopify theme preview bar during my theme test?
During a properly functioning Intelligems theme test, visitors will never see the Shopify theme preview bar. If you're seeing the preview bar, it's likely because the Intelligems JavaScript is not installed in all themes being tested. The Intelligems script needs to be in the live theme, plus all themes in the test.
If the script is in only the live theme, for example, visitors will be put into test groups and will be redirected to a draft theme, but since the Intelligems script is not in the draft theme, the preview bar will not be hidden once they're there, and Intelligems' analytics tracking will not function properly.
To fix this issue, install the Intelligems script in the live theme and all themes that you're testing. Since theme tests require a redirect, we recommend installing the Intelligems script synchronously by adding the script tag to your theme's head in the theme.liquid and all theme.*.liquid files (learn more here).
Can I test the One Page vs Three Page checkout, or test specific components on the checkout page, such as whether to include upsells?
Unfortunately we cannot A/B test the One Page vs Three Page checkout currently - this option has to be either on or off, so there isn't a way for us to turn it on for some visitors and not others.
We can test other components on the checkout page in limited cases - this is only possible if you are on Shopify Plus and are still using the checkout.liquid file (i.e. you have not moved to Checkout Extensibility yet). You can test components on that page by using our Find & Replace tool, which you can read more about here. To use Find & Replace on the checkout page, you will need to add our JavaScript to the head of your checkout.liquid file, which you can read more about here.
If you are not on Shopify Plus, or you no longer have access to the checkout.liquid file, you'll not be able to use Intelligems to A/B test any components on your checkout page currently, though this may change in the future if Shopify allows third parties, like Intelligems, access to make changes to that page. That said, we can A/B test pretty much any other content, such as different themes, page templates or pages if there is other content testing you are looking to do. We do anticipate the ability to test checkout layout/UI in the near future, but don't have a specific date for this feature just yet.
I'm running a content test and noticed the control content (i.e. URL, theme, language, etc.) flashes for a second before I see the test group content. How can I fix this?
Intelligems has two options for installing our JavaScript in your theme:
By using our app embed
By manually installing it into the theme code
The app embed works really well in most cases, but it can cause some flashing for certain tests and sites. If you notice this happening on your site, we recommend following these steps:
Turn the app embed off in all theme included in the test. You can do this by going to Shopify and in the online store section, click "Customize" next to each theme. In the left side menu, click the bottom option, which is "App Embeds". Turn the toggle off next to Intelligems and hit save.
Follow these steps to manually add the Intelligems script tag to your theme's head in the theme.liquid and all theme.*.liquid files.
If you still notice any flashing after making that change, please reach out to our support team here - there are a few settings we can turn on that allows us to load more information more quickly to avoid any flashing.
How does a theme test impact page load speeds?
Theme tests are relatively high performing, however, there may be a performance impact on a visitor's first page view. If Intelligems detects that a visitor is in the wrong theme, it will immediately reload the page in the correct theme. From then on, that visitor will be in the correct theme for their test group, which may be an unpublished theme. Our benchmarking shows that there is no performance impact on these subsequent page views, even for unpublished themes. If you're concerned abotu the potential performance impact of a reload on the first page view, you might consider using an onsite edit test instead.
Why should I run a template or content replacement test instead of a theme test?
Theme tests are a great way to test large changes to your store, or experiments that would otherwise require a developer to implement. There are a few drawbacks to keep in mind, however, so you may choose to run a template test or content replacement test instead:
A theme test means you have multiple live themes. Your customers are accessing your store via multiple themes, so you need to ensure any changes unrelated to the test (e.g., adding a landing page, editing copy or imagery) are made in all themes in the test.
Every visitor goes through a redirect. As mentioned above, theme tests are relatively high performing, but every visitor placed into a non-control group will still go through a one-time redirect to bring them to the correct theme for their group. While the UX impact of this one-time redirect is generally low, you may be able to avoid it altogether by using a content replacement test to dynamically edit content or inject HTML/CSS/JS.
You won't be able to delete any themes in the test for a while. During the experiment, visitors are sent to draft themes, and their browser will "remember" which theme to open the next time they visit your store via a session cookie. Once the test is over, if a visitor who was in one of the test groups re-visits your store and the cookie is still active, Shopify will first load the draft theme, and Intelligems will then immediately reset them back to the live theme. However, if the draft theme that was in the test has since been deleted, Intelligems won't be loaded and won't have the opportunity to reset the visitor's theme. Instead, the visitor see an error from Shopify, since they're trying to load a theme that does not exist. So, it's important to leave any themes that were tested in draft mode (rather than deleting them) for a while after the experiment (we recommend at least 1 week), to ensure any returning visitors are reset back to the live theme.
I got an error that said I am on the wrong destination URL for my test group while previewing my test - what does this mean?
In a split URL test, Intelligems monitors site visitors who visit your Origin URL, then either keeps them there or redirects them based on their assigned test group. Our preview widget detected that you're on a destination URL for a test group that you're not currently previewing. Please note, Intelligems will not redirect you away from a destination URL, as our redirects only function in one direction: from the origin to the destination.
To continue testing your split URL test, please navigate to the Origin URL again.
I got an error that said I have enabled the setting for "Redirect One Time" for this test, and I have already been redirected while previewing my test - what does this mean?
Intelligems has two options for how a visitor can be redirect in a split URL test -
Redirect One Time: A visitor will only be redirected the first time they land on the origin URL
Redirect Every Time: A visitor will be redirected every time they land on the origin URL
Because you have enabled "Redirect One Time" for your test, and you have already been redirected one time, Intelligems will not redirect you again, unless you clear your redirect history by clicking the button to do so in the preview widget.
If you'd prefer to change this setting, please see our help guide here on where you can do so.
Can we test an image vs. a video (MP4)?
Yes! This can be done using an HTML Onsite Edit.
How can I roll out a winning homepage template?
Shopify doesn't allow you to switch homepage templates. The winning template will have to be recreated in the original template.
An easy way to do this is within the theme editor. Find the test template and copy its contents into the default template (usually index.json).
What is a query selector and why does Intelligems use them?
A query selector is a way to select an element on a webpage using CSS-style syntax in JavaScript. In this case, we created a query selector based on the element you clicked on, which means it automatically targets that specific part of the page. However, this selector is just a starting point—you can edit it to be more general, more specific, or to target a different element altogether, depending on what you need your code to do.
Why must visitors visit the Origin URL to be included in the test?
Visitors must visit the Origin URL because this is the only page where Intelligems can detect their arrival and assign them to a test group. When a visitor lands on the Origin URL, our system:
Identifies that they've entered the test
Assigns them to either the Control Group or a test variation
Redirects test group visitors to their designated destination URL
Without visiting the Origin URL first, there's no trigger point for test group assignment, so visitors remain outside the test entirely. This also keeps the test's denominator limited to people who have been exposed to the test, keeping your data as actionable as possible.
Another thing to note is that URL redirect tests work in one direction only: from the Origin URL to the test destination URLs. Therefore, if a visitor lands directly on a destination URL (bypassing the Origin URL), they won't be included in the test. This is because:
They never triggered the test assignment process
Intelligems doesn't know they should be part of the test
They won't be tracked in test results or analytics
To ensure accurate test results, visitors must enter through the Origin URL to be properly allocated and tracked.
Why must visitors visit a page assigned to the Control Group's template in Shopify to be included in a Template test?
Visitors must visit a page that uses the Control Group's template because this is the only way Intelligems can detect their arrival and assign them to a test group. When a visitor lands on a page using the control template, our system:
Identifies that they've entered the test
Assigns them to either the Control Group or a test variation
Dynamically serves the appropriate template based on their test group
Without visiting a page that uses the control template, there's no trigger point for test group assignment, so visitors remain outside the test entirely.
You can check which pages are assigned to a template in Shopify by following these steps, or in your Shopify admin.
If your control template isn't assigned to any pages in Shopify, no visitors will be included in your test. To fix this:
Assign your control template to relevant pages in Shopify
Or choose a different template to use as your Control that's already assigned to pages you want to test
Can we test our Pop-Ups with Intelligems?
Yes!
Overview
Testing pop-ups is a common way to evaluate the impact of on-site messaging, promotions, or lead capture strategies without making permanent code changes.
With Intelligems, you can run A/B or multivariate tests on pop-ups to measure their effect on conversion rates, email capture rates, or profit per visitor.
Pop-ups are often powered by third-party apps (e.g. Klaviyo, Justuno, Alia, Privy, Postscript etc.), or sometimes implemented natively in a Shopify theme. Intelligems can generally integrate with either approach by letting you toggle visibility, edit content, or inject custom logic on who sees the pop-up.
Test Setup Types
Pop-ups can be tested using multiple Intelligems test setups. The right choice depends on how the pop-up is built and what aspect you want to test:
1. Onsite Edits Test - CSS or Javascript Injection
Best for:
Turning a third-party app’s pop-up hide/show.
Swapping content or offers inside an existing pop-up container.
Changing timing (e.g., show at 5s vs. 20s).
How it works: Use Intelligems’ editor to show, hide, or modify a pop-up on-page.
Example to show the Pop Up:
#my-popup {
display: block;
}Example to hide the Pop Up:
#my-popup {
display: none !important;
}Notes: We generally recommend having the Pop-Up on your theme and using Intelligems to Show/Hide it rather than "injecting" the popup itself. If you want to change which popup is displaying, you should either:
Have multiple present in your theme, with all but one hidden by default, then use CSS to swap which one is hidden
Use JS to "tell" the popup provider which version to render
If you would like us to take a deeper look at your set up, please reach out to our support team for help!
2. Split URL Test
Best for:
Comparing two entirely different pop-up designs built in Shopify or a page builder (e.g. Replo, Shogun).
Testing different pop-up apps without overlap.
How it works: Control and variant each point to different Shopify pages or landing pages, each with their own pop-up configuration.
Notes: Each page must be self-contained with its own tracking and pop-up triggers.
Setup
To test pop-ups with Intelligems, you’ll generally follow one of these approaches:
Follow the steps to set up an Onsite Edits Test
In Control: no pop-up (or your original pop-up).
In Variant: show new pop-up design, offer, or timing.
Swap Pop-Up Content
In your new group, adjust the pop up using the one of the methods described above. This can be useful for testing offers (e.g. “10% Off” vs. “Free Shipping”).
Adjust Targeting & Triggering
Combine Intelligems audience targeting with your pop-up (e.g. first-time visitors vs. returning customers).
Change when or how the pop-up fires (e.g. on scroll, after X seconds, exit intent).
Compare whether different timing increases conversions without disrupting browsing.
Use Custom Events to Filter Results
Tracking Engagement with Custom Events
For deeper analysis, you can use custom events to track whether a visitor actually engaged with a pop-up. This allows you to segment and filter results in reporting (e.g., conversion rates among users who saw or clicked the pop-up vs. those who did not).
Why this matters:
By tagging these events, you can evaluate whether the pop-up itself is influencing conversion behavior — not just whether the variant group had higher performance. This makes results easier to interpret and helps confirm if engagement with the pop-up is the driver of uplift.
Best Practices
Isolate one variable at a time: Test either the offer, design, or timing, but not all at once, to avoid confounding results.
Connect to downstream KPIs: Go beyond email sign-ups—measure profit per visitor by adding your COGS in Intelligems to confirm the pop-up improves business results.
Consider device experience: Pop-ups can behave differently on mobile vs. desktop. Always QA on both.
Limitations
Third-party app restrictions: Some pop-up apps don’t expose all timing/triggering logic to external scripts. In these cases, Intelligems can usually control visibility or injection, but not the app’s internal scheduling.
Theme-level vs. app-level: If the pop-up is baked into the theme code, edits must be structured as content tests. If powered by an app, you may need to coordinate app settings with Intelligems settings and targeting.
Last updated
Was this helpful?