Content Testing FAQs
Why do I see the Shopify theme preview bar during my theme test?
During a properly functioning Intelligems theme test, visitors will never see the Shopify theme preview bar. If you're seeing the preview bar, it's likely because the Intelligems Javascript is not installed in all themes being tested. The Intelligems script needs to be in the live theme, plus all themes in the test.
If the script is in only the live theme, for example, visitors will be put into test groups and will be redirected to a draft theme, but since the Intelligems script is not in the draft theme, the preview bar will not be hidden once they're there, and Intelligems' analytics tracking will not function properly.
To fix this issue, install the Intelligems script in the live theme and all themes that you're testing. Since theme tests require a redirect, we recommend installing the Intelligems script synchronously by adding the script tag to your theme's head in the theme.liquid and all theme.*.liquid files (learn more here).
Can I test the "New" Shopify Checkout vs. the "Old" Shopify Checkout, or test specific components on the checkout page, such as whether to include a tip modal?
Unfortunately we cannot A/B test Shopify's new checkout currently - it has to be either on or off, so there isn't a way for us to turn it on for some visitors and not others.
We can test other components on the checkout page if you are on Shopify Plus and are still using the checkout.liquid file (i.e. you have not moved to Checkout Extensibility yet). You can test components on that page by using our Find & Replace tool, which you can read more about here. In order to use Find & Replace on the checkout page, you will need to add our JavaScript to the head of your checkout.liquid file, which you can read more about here.
If you are not on Shopify Plus, or you no longer have access to the checkout.liquid file, you will not be able to use Intelligems to A/B test any components on your checkout page. That said, we can A/B test pretty much any other content, such as different themes, page templates or pages if there is other content testing you are looking to do.
Help! I'm running a theme test and noticed the live theme flashes for a second before I'm redirected. How can I fix this?
Intelligems has two options for installing our JavaScript in your theme:
By using our app embed
By manually installing it into the theme code
The app embed works really well in most cases, but it can cause some flashing for theme tests. If you notice this happening on your site, we recommend following these steps:
Turn the app embed off in all theme included in the test. You can do this by going to Shopify and in the online store section, click "Customize" next to each theme. In the left side menu, click the bottom option, which is "App Embeds". Turn the toggle off next to Intelligems and hit save.
Follow these steps to manually add the Intelligems script tag to your theme's head in the theme.liquid and all theme.*.liquid files.
How does a theme test impact my loading time?
Theme tests are relatively high performing. The lowest performance will be on the first page view. If we detect they’re in the wrong theme then then we’ll reload the page in the correct theme. From then on, they will be in the preview theme. Our benchmarking shows that there should be no performance impact on these page views, but have the impact of the redirect on the first page view. A more highly performing test option is to use our HTML editor and CSS/JS injection, but the theme tests are very convenient and are definitely a good option for testing large changes.
Why should I run a template or content replacement test instead of a theme test?
Theme tests are a great way to test large changes to your store, or experiments that would otherwise require a developer to implement. There are a few drawbacks to keep in mind, however, so you may choose to run a template test or content replacement test instead:
A theme test means you have multiple live themes. Your customers are accessing your store via multiple themes, so you need to ensure any changes unrelated to the test (e.g., adding a landing page, editing copy or imagery) are made in all themes in the test.
Every visitor goes through a redirect. As mentioned above, theme tests are relatively high performing, but every visitor placed into a non-control group will still go through a one-time redirect to bring them to the correct theme for their group. While the UX impact of this one-time redirect is generally low, you may be able to avoid it altogether by using a content replacement test to dynamically edit content or inject HTML/CSS/JS.
You won't be able to delete any themes in the test for a while. During the experiment, visitors are sent to draft themes, and their browser will "remember" which theme to open the next time they visit your store via a session cookie. Once the test is over, if a visitor who was in one of the test groups re-visits your store and the cookie is still active, Shopify will first load the draft theme, and Intelligems will then immediately reset them back to the live theme. However, if the draft theme that was in the test has since been deleted, Intelligems won't be loaded and won't have the opportunity to reset the visitor's theme. Instead, the visitor see an error from Shopify, since they're trying to load a theme that does not exist. So, it's important to leave any themes that were tested in draft mode (rather than deleting them) for a while after the experiment (we recommend at least 1 week), to ensure any returning visitors are reset back to the live theme.
Last updated