Content Testing FAQs
Last updated
Last updated
During a properly functioning Intelligems theme test, visitors will never see the Shopify theme preview bar. If you're seeing the preview bar, it's likely because the Intelligems Javascript is not installed in all themes being tested. The Intelligems script needs to be in the live theme, plus all themes in the test.
If the script is in only the live theme, for example, visitors will be put into test groups and will be redirected to a draft theme, but since the Intelligems script is not in the draft theme, the preview bar will not be hidden once they're there, and Intelligems' analytics tracking will not function properly.
To fix this issue, install the Intelligems script in the live theme and all themes that you're testing. Since theme tests require a redirect, we recommend installing the Intelligems script synchronously by adding the script tag to your theme's head in the theme.liquid and all theme.*.liquid files (learn more here).
Unfortunately we cannot A/B test the One Page vs Three Page checkout currently - this option has to be either on or off, so there isn't a way for us to turn it on for some visitors and not others.
We can test other components on the checkout page in limited cases - this is only possible if you are on Shopify Plus and are still using the checkout.liquid file (i.e. you have not moved to Checkout Extensibility yet). You can test components on that page by using our Find & Replace tool, which you can read more about here. In order to use Find & Replace on the checkout page, you will need to add our JavaScript to the head of your checkout.liquid file, which you can read more about here.
If you are not on Shopify Plus, or you no longer have access to the checkout.liquid file, you will not be able to use Intelligems to A/B test any components on your checkout page currently, though this may change in the future if Shopify allows third parties, like Intelligems, access to make changes to that page. That said, we can A/B test pretty much any other content, such as different themes, page templates or pages if there is other content testing you are looking to do. We do anticipate the ability to test checkout layout/UI in the near future, but don't have a specific date for this feature just yet.
Intelligems has two options for installing our JavaScript in your theme:
By using our app embed
By manually installing it into the theme code
The app embed works really well in most cases, but it can cause some flashing for certain tests and sites. If you notice this happening on your site, we recommend following these steps:
Turn the app embed off in all theme included in the test. You can do this by going to Shopify and in the online store section, click "Customize" next to each theme. In the left side menu, click the bottom option, which is "App Embeds". Turn the toggle off next to Intelligems and hit save.
Follow these steps to manually add the Intelligems script tag to your theme's head in the theme.liquid and all theme.*.liquid files.
If you still notice any flashing after making that change, please reach out to our support team - there are a few settings we can turn on that allows us to load more information more quickly to avoid any flashing.
Theme tests are relatively high performing, however, there may be a performance impact on a visitor's first page view. If Intelligems detects that a visitor is in the wrong theme, it will immediately reload the page in the correct theme. From then on, that visitor will be in the correct theme for their test group, which may be an unpublished theme. Our benchmarking shows that there is no performance impact on these subsequent page views, even for unpublished themes. If you're concerned abotu the potential performance impact of a reload on the first page view, you might consider using an onsite edit test instead.
Theme tests are a great way to test large changes to your store, or experiments that would otherwise require a developer to implement. There are a few drawbacks to keep in mind, however, so you may choose to run a template test or content replacement test instead:
A theme test means you have multiple live themes. Your customers are accessing your store via multiple themes, so you need to ensure any changes unrelated to the test (e.g., adding a landing page, editing copy or imagery) are made in all themes in the test.
Every visitor goes through a redirect. As mentioned above, theme tests are relatively high performing, but every visitor placed into a non-control group will still go through a one-time redirect to bring them to the correct theme for their group. While the UX impact of this one-time redirect is generally low, you may be able to avoid it altogether by using a content replacement test to dynamically edit content or inject HTML/CSS/JS.
You won't be able to delete any themes in the test for a while. During the experiment, visitors are sent to draft themes, and their browser will "remember" which theme to open the next time they visit your store via a session cookie. Once the test is over, if a visitor who was in one of the test groups re-visits your store and the cookie is still active, Shopify will first load the draft theme, and Intelligems will then immediately reset them back to the live theme. However, if the draft theme that was in the test has since been deleted, Intelligems won't be loaded and won't have the opportunity to reset the visitor's theme. Instead, the visitor see an error from Shopify, since they're trying to load a theme that does not exist. So, it's important to leave any themes that were tested in draft mode (rather than deleting them) for a while after the experiment (we recommend at least 1 week), to ensure any returning visitors are reset back to the live theme.
In a split URL test, Intelligems monitors site visitors who visit your Origin URL, then either keeps them there or redirects them based on their assigned test group. Our preview widget detected that you're on a destination URL for a test group that you're not currently previewing. Please note, Intelligems will not redirect you away from a destination URL, as our redirects only function in one direction: from the origin to the destination.
In order to continue testing your split URL test, please navigate to the Origin URL again.
Intelligems has two options for how a visitor can be redirect in a split URL test -
Redirect One Time: A visitor will only be redirected the first time they land on the origin URL
Redirect Every Time: A visitor will be redirected every time they land on the origin URL
Because you have enabled "Redirect One Time" for your test, and you have already been redirected one time, Intelligems will not redirect you again, unless you clear your redirect history by clicking the button to do so in the preview widget.
If you'd prefer to change this setting, please see our help guide here on where you can do so.
Yes! This can be done using an HTML Onsite Edit.