Analytics FAQs
Frequently asked questions about analytics on Intelligems
My test is live! Why can't I see any analytics yet?
The analytics dashboard is on a one to two hour delay at this time! If you have just launched a new test, check back in after one to two hours - if you're still not seeing anything, please reach out to us at support@intelligems.io.
Why doesn't the data align with Shopify or Google Analytics?
There are a few reasons why the data won't align perfectly with Shopify or GA in a given moment in time:
Our data goes through batch processing once an hour and has some caching layers to make the dashboards load quickly. So, the data you're looking is on about a 1-2 hour lag.
We track conversion based on visitors instead of sessions.
We also filter out "outlier" orders (more than 3 standard deviations from the mean) to avoid huge orders skewing the data.
We do not include any orders not placed via the online sales channel (e.g. draft orders, subscription refills, app orders like Tapcart, etc.)
We are not able to track conversions that occurred through a dynamic checkout button - see the FAQ below this for more information on this.
For price tests, by default we filter only orders that contain a "test product", i.e. a product that we're testing prices on.
You may have targeting set up that only includes certain users in a test, and therefore the results, depending on their currency or location, UTMs, device types, whether they are new or returning, or whether they have hit a page you are targeting in the test.
For Split URL tests specifically, you may notice a large discrepancy between Intelligems and Shopify. Since Intelligems is redirecting the visitor client-side, Shopify may register the origin page as the landing page for some visitors who are ultimately redirected to the test page. So, you can't compare Shopify's "landing page" reporting to Intelligems' test group reporting in this case as the "landing page" is unreliable.
If you still think the numbers don't align, please send a few screenshots of what you're seeing in Shopify or another analytics app, and what test results you are looking at, and we can help look into it.
Can Intelligems track checkouts that occurred through dynamic checkout buttons, such as Apple Pay, Shop Pay or PayPal?
Intelligems requires a unique cart attribute known as the Intelligems ID, or igId for short, in order to track checkout information. Unfortunately, dynamic or express checkout buttons don't support cart attributes, so Intelligems is not able to track conversions that occurred through one of these buttons. You can find more information on this Shopify limitation at their help article here.
Can I export the data from an Intelligems test?
Yes! We support data exports of orders and line item sales. See more information on what’s included in those exports and how to get them here.
How can I remove outlier orders from the analytics?
Intelligems automatically removes outlier orders over a certain z-score, also known in statistics as a standard score. A z-score tells us the number of standard deviations a value is from the mean of a given distribution.
Negative z-scores indicate the value lies below the mean
Positive z-scores indicate the value lies above the mean
In Intelligems analytics, the z-score filter is applied to order value. We check if an individual order's value is an outlier (3 standard deviations greater than the mean) and if it is (based on where the filter is set), we exclude that order and visitor from analytics.
The mean we compare to is the mean of all orders that have been made since you installed Intelligems, plus a month of historical orders (the month before your installation). This only applies when we have ≥ 850 orders in our system (if we have recorded fewer orders than that, the z-score filter will not have any effect).
By default, the max z-score value is 3 (i.e., we filter out orders with a z-score more than 3 standard deviations above the mean), but it is configurable if you’d like to set a different threshold within the 'Filters' section in the top right corner of the analytics.
Can I view results for just a specific product or category?
Yes! You can add a product group for each product/variant in the Intelligems COGS upload - see more on downloading and providing this upload here. Once Intelligems has this information, splits of sales, revenue, profit and other metrics by product or collection will populate in the ‘Product Groups’ page of the analytics dashboard in the Intelligems app.
Are orders reported if the user completes any checkout or only a checkout that has test products?
For price tests, we offer both options in our analytics. If you are in the Analytics tab in the Intelligems app, you can click on the 'Filters' button in the top right and then choose between 'Test Orders Only' or 'Sitewide Orders', followed by the 'Save' button.
Are orders reported if they happen somewhere other than the online store?
No. Intelligems data only includes what happens on the online store. Draft orders, orders that occur on Facebook, TikTok, Instagram, an app, or in physical stores will not be included in our results.
How does the app determine New vs Returning visitor?
We assign every visitor an Intelligems ID when they come to the brand's website once our JavaScript is live in their site. We assign the Intelligems ID to the user's cookie. If a visitor has an ID already, they are returning. If we are assigning them an ID on that visit, they are new. Here's more information on how to use Audience Targeting for new and returning visitors.
Why is one of my test groups receiving more visitors even when the traffic is split evenly?
Intelligems randomly splits your website traffic between all of the groups in your test. Because the traffic is divided randomly, the traffic split may not be exactly even, especially early on in your test. The traffic split ratio has minimal statistical significance as Intelligems calculates results based on the conversion rates in each group, so as long as the visitor numbers are close, your results are still valid and reliable.
That said, if there is a large difference between the number of visitors in each test group for a test where the traffic should be split evenly, this could be a sign that there is something wrong with your set up. Please reach out to Intelligems support if you think this may be the case!
Are the numbers in the Analytics Dashboard inclusive of discounts?
Revenue and profit figures shown in the Analytics Dashboard are post-discount, including the “hidden” discount Intelligems applies for price tests for brands on Shopify Plus.
For details on discounts that occurred during the test, check out the Discounts tab of the Analytics Dashboard. These figures will still exclude the “hidden” discount Intelligems applies for price tests for brands on Shopify Plus, but include any others, including (if applicable) volume discounts that occurred through Intelligems.
How does the Google Analytics integration work?
If you have logged in and enabled the GA4 integration on a test in your Intelligems account, once you start a test, Intelligems will create an audience for each test group in Google Analytics. The names of the audiences will match the names of the test groups that you set up in Intelligems. From there, you can manipulate the data and create segments in GA.
It is important to note that Intelligems cannot send data to GA4 retroactively, so you should enable the integration on your test before starting it.
The experiments we have run using Intelligems resulted in uplifts, but we have not seen our performance improve in Shopify since implementing the changes that we tested. Why might that happen?
The short answer is that Intelligems tests are measuring incrementality. For example, the data may confirm that experience B converted better than experience A while the test was running, but there are still a number of exogenous factors that go into overall conversion rate and site performance.
A good example of this is looking at site performance for most industries in January - many brands see lower conversion that time of year due to it being after the holidays, and there being less urgency to spend. Similarly, it's possible that your ad campaigns are generating different types of customers who have fundamentally different likelihoods to convert. It still is incrementally better, but that natural conversion rate can vary quite a bit.
These are all great reasons to continually test your site! Frequent testing will help to solve for seasonal changes and increase confidence in test results.
Last updated