Analytics FAQs
Frequently asked questions about analytics on Intelligems
My test is live! Why can't I see any analytics yet?
The analytics dashboard is on a one to two hour delay. If you have just launched a new test, check back in after two hours. If you're still not seeing anything, please reach out to Intelligems support.
Why doesn't the data align with Shopify or Google Analytics?
There are a few reasons why the data won't align perfectly with Shopify or GA in a given moment in time:
Our data goes through batch processing once an hour and has some caching layers to make the dashboards load quickly. So, the data you're looking at is on about a 1-2 hour lag.
We track conversion based on visitors instead of sessions.
We also filter out "outlier" orders (more than 3 standard deviations from the mean) to avoid huge orders skewing the data.
We do not include any orders not placed via the online sales channel (e.g. draft orders, subscription refills, app orders like Tapcart, etc.)
We are not able to track conversions that occurred through a dynamic checkout button - see the FAQ below this for more information on this.
For price tests, by default we filter only orders that contain a "test product", i.e. a product that we're testing prices on.
You may have targeting set up that only includes certain users in a test, and therefore the results, depending on their currency or location, UTMs, device types, whether they are new or returning, or whether they have hit a page you are targeting in the test.
For Split URL tests specifically, you may notice a large discrepancy between Intelligems and Shopify. Since Intelligems is redirecting the visitor client-side, Shopify may register the origin page as the landing page for some visitors who are ultimately redirected to the test page. So, you can't compare Shopify's "landing page" reporting to Intelligems' test group reporting in this case as the "landing page" is unreliable.
If you still think the numbers don't align, please send a few screenshots of what you're seeing in Shopify or another analytics app, and what test results you are looking at, and we can help look into it.
How does Intelligems track conversion rate?
Intelligems tracks our conversion rates based off of visitors instead of sessions. Here's the difference between the two: In Shopify, sessions are the total number of interactions a user has with a store, while visitors are the total number of unique people who access the website:
Sessions
A session is counted each time a user interacts with the store, such as by searching or entering the store's URL directly. A session can also be initiated when a user logs in to the website. However, a single user can have multiple sessions if they leave and return to the store, or if they use different devices to interact with the store. A session ends after 30 minutes of inactivity or at midnight UTC. Because of this, the number of sessions is usually higher than the number of visitors.
Visitors
Visitors are the total number of people who access the website, regardless of how many times they visit.
Can Intelligems track checkouts that occurred through dynamic checkout buttons, such as Apple Pay, Shop Pay or PayPal?
Intelligems requires a unique cart attribute known as the Intelligems ID, or igId for short, in order to track checkout information. Unfortunately, dynamic or express checkout buttons don't support cart attributes, so Intelligems is not able to track conversions that occurred through one of these buttons. You can find more information on this Shopify limitation at their help article here.
Can I export the data from an Intelligems test?
Yes! We support data exports of orders and line item sales. See more information on what’s included in those exports and how to get them here.
How can I remove outlier orders from the analytics?
Intelligems automatically removes outlier orders over a certain z-score, also known in statistics as a standard score. A z-score tells us the number of standard deviations a value is from the mean of a given distribution.
Negative z-scores indicate the value lies below the mean
Positive z-scores indicate the value lies above the mean
In Intelligems analytics, the z-score filter is applied to order value. We check if an individual order's value is an outlier (3 standard deviations greater than the mean) and if it is (based on where the filter is set), we exclude that order and visitor from analytics.
The mean we compare to is the mean of all orders that have been made since you installed Intelligems, plus a month of historical orders (the month before your installation). This only applies when we have ≥ 850 orders in our system (if we have recorded fewer orders than that, the z-score filter will not have any effect).
By default, the max z-score value is 3 (i.e., we filter out orders with a z-score more than 3 standard deviations above the mean), but it is configurable if you’d like to set a different threshold within the 'Filters' section in the top right corner of the analytics.
Can I view results for just a specific product or category?
Yes! You can add a product group for each product/variant in the Intelligems COGS upload - see more on downloading and providing this upload here. Once Intelligems has this information, splits of sales, revenue, profit and other metrics by product or collection will populate in the ‘Product Groups’ page of the Results & Goals dashboard in the Intelligems app.
Are orders reported if the user completes any checkout or only a checkout that has test products?
For price tests and content tests, we offer both options in our analytics. You can set your preferred analysis method
When setting up the test: before you start it
When viewing analytics: If you are in the Analytics tab, you can click on the 'Filters' button in the top right and then choose between 'Test Orders Only' or 'Sitewide Orders', followed by the 'Save' button.
Other types of tests do not offer this option - they report on all orders regardless of the products contained within.
Are orders reported if they happen somewhere other than the online store?
No. Intelligems data only includes what happens on the online store. Draft orders, orders that occur on Facebook, TikTok, Instagram, an app, or in physical stores will not be included in our results.
Does data from post-purchase upsells get counted in the analytics dashboard?
In most cases yes, the order will be updated with the upsell and we will count it. If you have any questions about this, contact our support team with information about the app you're using for post-purchase upsells, and include an example Shopify order number if possible. Note that we cannot track purchases that were created as draft orders.
How does the app determine New vs Returning visitor?
We assign every visitor an Intelligems ID when they come to the brand's website once our JavaScript is live in their site. We assign the Intelligems ID to the user's cookie. If a visitor has an ID already, they are returning. If we are assigning them an ID on that visit, they are new. Here's more information on how to use Audience Targeting.
Why is one of my test groups receiving more visitors even when the traffic is split evenly?
Intelligems randomly splits your website traffic between all of the groups in your test. Because the traffic is divided randomly, the traffic split may not be exactly even, especially early on in your test. The traffic split ratio has minimal statistical significance as Intelligems calculates results based on the conversion rates in each group, so as long as the visitor numbers are close, your results are still valid and reliable.
That said, if there is a large difference between the number of visitors in each test group for a test where the traffic should be split evenly, this could be a sign that there is something wrong with your set up. Please reach out to Intelligems support if you think this may be the case!
Are the numbers in the Analytics Dashboard inclusive of discounts?
Revenue and profit figures shown in the Analytics Dashboard are post-discount, including the “hidden” discount Intelligems applies for price tests for brands on Shopify Plus.
For details on discounts that occurred during the test, check out the Discounts tab of the Analytics Dashboard. These figures will still exclude the “hidden” discount Intelligems applies for price tests for brands on Shopify Plus, but include any others, including (if applicable) volume discounts that occurred through Intelligems.
How does the Google Analytics integration work?
If you have logged in and enabled the GA4 integration on a test in your Intelligems account, once you start a test, Intelligems will create an audience for each test group in Google Analytics. The names of the audiences will match the names of the test groups that you set up in Intelligems. From there, you can manipulate the data and create segments in GA4.
It is important to note that Intelligems cannot send data to GA4 retroactively, so you should enable the integration on your test before starting it.
The experiments we have run using Intelligems resulted in uplifts, but we have not seen our performance improve in Shopify since implementing the changes that we tested. Why might that happen?
The short answer is that Intelligems tests are measuring incrementality. For example, the data may confirm that experience B converted better than experience A while the test was running, but there are still a number of exogenous factors that go into overall conversion rate and site performance.
For example, consider site performance for most industries in January. Many brands see lower conversion rates that time of year due due to post-holiday demand.
Similarly, it's possible that your ad campaigns are generating different types of customers who have fundamentally different likelihoods to convert. It still is incrementally better, but that natural conversion rate can vary quite a bit.
These are all great reasons to continually test your site! Frequent testing will help to solve for seasonal changes and increase confidence in test results.
How does Intelligems calculate metrics when customers purchase in different currencies?
In our analytics dashboard, we only use your store's default currency. For example if your store's default currency is in USD and a customer makes a purchase in EUR, we convert it to USD in our analytics.
What time zone is used in analytics?
Analytics in Intelligems are based on the time zone in your Shopify store's settings. Learn how to access and edit your Shopify store's time zone here.
Does any data from post purchase upsell apps filter through to Intelligems' analytics?
Intelligems works with different post purchase upsell apps on an app by app basis. If the post purchase updates the original order, those are included in our analytics as it is for any other order. However, if the post purchase app creates the upsell as a draft order, then we are unable to include this in our analytics, as draft orders are excluded automatically.
How does Intelligems classify New vs. Returning customers?
Shopify classifies customers as "new" or "returning" based on their order history:
New customers are those who have placed their first order with your store. They are considered new customers because they haven't previously made a purchase from your store.
Returning customers are those who have placed an order and have a history of at least one previous order with your store. This means they have made more than one purchase from your store.
Unfortunately, Shopify does not share a visitor's order history, or whether a visitor is new or returning with Intelligems. So Intelligems classifies New vs. Returning as visitors who have an Intelligems ID assigned to them. Not necessarily whether they are a new or returning customer. It is important to note that we determine whether a visitor is new or returning based off of whether they have an Intelligems ID assigned to them. We assign all site visitors an Intelligems ID when they come to your site, as long as our JavaScript is in your live site. Because of this, if you have just started working with Intelligems, most visitors will be seen as new since they don't have an Intelligems ID assigned to them yet.
Last updated