๐Ÿ…ฐ๏ธA/A Test

What is A/A testing?

While A/B testing involves driving traffic to two (or more) different variants to see which version converts better, an A/A test pits two identical variants against each other. Instead of learning which version performs the best, the goal of an A/A test is to start getting comfortable with Intelligems, while also confirming that all the data and analytics are tracking smoothly.

How do I set up an A/A test in Intelligems?

There are a lot of different ways that you could set up an A/A test in Intelligems, but here is what we recommend doing to keep things simple:

  1. Click "Create New Test" in the top right corner of Intelligems > Content Test > Create Test > Onsite Edits > Create Test.

  2. Enter your test name, description and goal - this only impacts how you view the results!

  3. Create your test groups - in most cases, it makes sense to leave traffic split 50/50 for an A/A test.

  4. You can skip setting up any targeting, and feel free to enable the integration with Google Analytics if you would like.

  5. Click "Go to Builder" in the bottom right to save your test - this will bring you to your site in our Builder Mode, but no need to set anything up here since you are running an A/A test!

  6. Return to the app and hit the play button to take your A/A test live! ๐ŸŽ‰

Data will start to be available for this test within 2 hours of starting your test, and will update that often as well! Because the groups in this test are identical, you should see nearly identical results across the board. You may notice differences in the results for each group early on, but the longer you run your A/A test, the closer the results should get.

We recommend ending an A/A test after:

  • 2 full weeks of testing

  • 300+ orders per test group

If the above has been hit, and Stat Sig shows 95%+ confidence that one A group is better than the other, this may be a sign that something is wrong, since equal groups should perform similarly. Please reach out to support@intelligems.io for help troubleshooting your test setup.

One note to keep in mind: running other tests simultaneously can cause some noise in these results. You can read more about interference between tests here.

Last updated