# Automated Test Alerting for Slack

## Overview:

This guide walks you through creating a daily automation that checks running tests for traffic or data issues (low visitors, SRM failures, large conversion drops, minimum orders per variant) and posts a Slack alert when a test fails health thresholds.

<figure><img src="/files/GFsSWegmz3uypm0BMwjk" alt=""><figcaption></figcaption></figure>

### Softwares Used <a href="#softwares-used" id="softwares-used"></a>

To build this automated reporting pipeline, you will need the following tools:

* [n8n](https://n8n.io/): The primary workflow automation platform used to connect APIs and schedule tasks.
* [Slack](https://api.slack.com/apps): The final destination where the AI-generated health checks and reports will be posted.

## How to Create Your Automated Test Alerts <a href="#how-to-create-your-scheduled-test-analysis" id="how-to-create-your-scheduled-test-analysis"></a>

### Step 1: Get Your Intelligems API Keys <a href="#step-1-get-your-api-keys" id="step-1-get-your-api-keys"></a>

\[[How to get Intelligems API Key](https://docs.intelligems.io/developer-resources/external-api#getting-started)]

### Step 2: Create the Workflow in n8n <a href="#how-to-create-an-automated-test-monitoring-integration-for-slack" id="how-to-create-an-automated-test-monitoring-integration-for-slack"></a>

#### **Node 1: Schedule Trigger**

1. Click the **"+"** button to add a node
2. Search for **"Schedule Trigger"**
3. Configure it:
   * **Trigger Interval**: "Hours"
   * **Hours Between Triggers**: 8 (runs 3x daily: morning, afternoon, evening)
   * Or set to **12** for 2x daily

<figure><img src="/files/dqKd9pGocqh6gtE98b6J" alt=""><figcaption></figcaption></figure>

#### **Node 2: Create Organization List**

* Add **"Code"** node
* Select **"Code in JavaScript"**
* Select **"Run Once for All Items"**
* Paste this code. Update it with a display name for your client & their API key:

```
// Define all your Intelligems organizations
const organizations = [
  {
    name: "Client 1",
    apiKey: "YOUR_API_KEY_1"
  },
  {
    name: "Client 2", 
    apiKey: "YOUR_API_KEY_2"
  },
  {
    name: "Client 3",
    apiKey: "YOUR_API_KEY_3"
  }
];

// Return each org as a separate item for n8n to process
return organizations.map(org => ({ json: org }));
```

<figure><img src="https://docs.intelligems.io/~gitbook/image?url=https%3A%2F%2F2052204893-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F2SvefuMLsJyJPAcVXeWc%252Fuploads%252FDRBeDys0ClEIoItfUdsH%252FScreenshot%25202026-01-30%2520at%252010.58.08%25E2%2580%25AFAM.png%3Falt%3Dmedia%26token%3Dfdc79f48-af06-4251-809a-3fd2a5864bc9&#x26;width=768&#x26;dpr=3&#x26;quality=100&#x26;sign=b4f3eeab&#x26;sv=2" alt=""><figcaption></figcaption></figure>

#### **Node 3: Get All Running Tests**

1. Click **"+"** after the Schedule node
2. Search for **"HTTP Request"**
3. Configure:
   * **Method**: GET
   * **URL**: `https://api.intelligems.io/v25-10-beta/experiences-list`
   * **Authentication**: None
   * Enable **Send Headers**
     * **Name**: `intelligems-access-token`
     * **Value**: `{{ $json.apiKey }}`
4. Click "Execute step" to verify it works

<figure><img src="https://docs.intelligems.io/~gitbook/image?url=https%3A%2F%2F2052204893-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F2SvefuMLsJyJPAcVXeWc%252Fuploads%252F4VQawiu8Aj6jmzn9W7Tt%252FScreenshot%25202026-01-30%2520at%252010.59.12%25E2%2580%25AFAM.png%3Falt%3Dmedia%26token%3Dc21a9cec-3937-429a-8eb7-ca459dedb53a&#x26;width=768&#x26;dpr=3&#x26;quality=100&#x26;sign=c1bd2f9&#x26;sv=2" alt=""><figcaption></figcaption></figure>

#### **Node 4: Filter Tests by Status = "started"**

* \
  Add **"Code"** node
* Select **"Code in JavaScript"**
* Select **"Run Once for All Items"**
* Update the below code so that `InsertOrgId` maps to match the Intelligems organization IDs of your clients (you can find these in Intelligems under Settings > General > Organization Settings), the `name` value is display name for your clients, and the `apiKey` value is the Intelligems API Key for those clients. Then paste this code into the Code section in n8n.

```
// Filter for running tests with status = started
const items = $input.all();
const today = new Date();
const filteredTests = [];

// Static mapping of organization IDs to client names AND API keys
const clientMapping = {
  "InsertOrgId": {
    name: "Client 1",
    apiKey: "ig_live_11111"
  },
  "InsertOrgId": {
    name: "Client 2",
    apiKey: "ig_live_22222"
  }
};

for (const item of items) {
  // Get organization ID from the API response
  const orgId = item.json.organizationId || item.json.organization?.id || (item.json.experiencesList && item.json.experiencesList[0]?.organizationId);
  const clientData = clientMapping[orgId] || { name: "Unknown Client", apiKey: "" };
  
  // Access the experiencesList array
  const experiencesList = item.json.experiencesList;
  
  // Iterate through each test in the experiencesList
  if (experiencesList) {
    for (const test of experiencesList) {
      // Only check if status is "started"
      if (test.status === "started") {
        // Calculate days running for reporting purposes
        const startDate = new Date(test.startedAtTs);
        const daysRunning = Math.floor((today - startDate) / (1000 * 60 * 60 * 24));
        
        filteredTests.push({
          json: {
            clientName: clientData.name,
            apiKey: clientData.apiKey,
            experienceId: test.id,
            testName: test.name,
            daysRunning: daysRunning,
            weekNumber: Math.ceil(daysRunning / 7),
            startDate: startDate.toISOString().split('T')[0],  
            startDateReadable: startDate.toLocaleDateString('en-US', { month: 'short', day: 'numeric', year: 'numeric' }),
            endDate: today.toISOString().split('T')[0],
            endDateReadable: today.toLocaleDateString('en-US', { month: 'short', day: 'numeric', year: 'numeric' })
          }
        });
      }
    }
  }
}

return filteredTests;
```

<figure><img src="/files/VAlNDaRVcsd6mPIeoxon" alt=""><figcaption></figcaption></figure>

#### **Node 5: Get Test Analytics Data (Loop)**

1. Add **"HTTP Request"** node
2. Configure:
   * **Method**: GET
   * **URL**: `https://api.intelligems.io/v25-10-beta/analytics/resource/{{ $json.experienceId }}`
   * **Authentication**: None
   * Enable **Send Headers**
     * **Name**: `intelligems-access-token`
     * **Value**: `{{ $json.apiKey }}`
3. Click "Execute step" to verify it works

<figure><img src="https://docs.intelligems.io/~gitbook/image?url=https%3A%2F%2F2052204893-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F2SvefuMLsJyJPAcVXeWc%252Fuploads%252FumtLe94pm7cFpWZXsahT%252FScreenshot%25202026-01-30%2520at%252011.00.26%25E2%2580%25AFAM.png%3Falt%3Dmedia%26token%3D5770d8fb-e58a-45a0-abdc-09f759505b42&#x26;width=768&#x26;dpr=3&#x26;quality=100&#x26;sign=5ff81b64&#x26;sv=2" alt=""><figcaption></figcaption></figure>

#### Node 6: Check if below threshold

* Add **"Code"** node
* Select **"Code in JavaScript"**
* Select **"Run Once for All Items"**
* Update the below code so that the `minTotalVisitors`, `minOrdersPerVariant`, `srmPValueThreshold`, and `conversionDropThreshold` values match your threshold requirements. Then paste this code into the Code section in n8n.

```
const analytics = $input.item.json;
const filterData = $('Code in JavaScript1').item.json;
const testName = filterData.testName || "Unknown Test";
const clientName = filterData.clientName || "Unknown Client";
const experienceId = filterData.experienceId;

const THRESHOLDS = {
  minTotalVisitors: 1000,
  minOrdersPerVariant: 50,
  srmPValueThreshold: 0.01,
  conversionDropThreshold: 0.20
};

const overview = analytics.overview || analytics;
const variations = overview.variations || [];

if (!variations || variations.length === 0) {
  return [];
}

let totalVisitors = 0;
let controlVariation = null;
const variantData = [];

for (let i = 0; i < variations.length; i++) {
  const variant = variations[i];
  const sessions = variant.sessions || 0;
  const orders = variant.orders || 0;
  const cvr = sessions > 0 ? orders / sessions : 0;
  
  totalVisitors = totalVisitors + sessions;
  
  const isControl = variant.isControl || variant.name === "Control" || variant.label === "Control";
  
  const variantInfo = {
    name: variant.name || variant.label || "Variant " + i,
    sessions: sessions,
    orders: orders,
    cvr: cvr,
    isControl: isControl
  };
  
  variantData.push(variantInfo);
  
  if (isControl) {
    controlVariation = variantInfo;
  }
}

const failedThresholds = [];
const details = {};

if (totalVisitors < THRESHOLDS.minTotalVisitors) {
  failedThresholds.push("min_total_visitors");
  details.minTotalVisitors = {
    current: totalVisitors,
    required: THRESHOLDS.minTotalVisitors,
    message: "Only " + totalVisitors + " visitors (need " + THRESHOLDS.minTotalVisitors + ")"
  };
}

for (let i = 0; i < variantData.length; i++) {
  const variant = variantData[i];
  if (variant.orders < THRESHOLDS.minOrdersPerVariant) {
    if (!details.minOrdersPerVariant) {
      failedThresholds.push("min_orders_per_variant");
      details.minOrdersPerVariant = [];
    }
    details.minOrdersPerVariant.push({
      variant: variant.name,
      current: variant.orders,
      required: THRESHOLDS.minOrdersPerVariant,
      message: variant.name + ": Only " + variant.orders + " orders"
    });
  }
}

if (variantData.length >= 2 && totalVisitors > 0) {
  const expectedPerVariant = totalVisitors / variantData.length;
  let chiSquare = 0;
  
  for (let i = 0; i < variantData.length; i++) {
    const observed = variantData[i].sessions;
    const expected = expectedPerVariant;
    chiSquare = chiSquare + Math.pow(observed - expected, 2) / expected;
  }
  
  const df = variantData.length - 1;
  let isSRM = false;
  
  if (df === 1 && chiSquare > 6.635) {
    isSRM = true;
  } else if (df === 2 && chiSquare > 9.210) {
    isSRM = true;
  } else if (df >= 3 && chiSquare > 11.345) {
    isSRM = true;
  }
  
  if (isSRM) {
    failedThresholds.push("sample_ratio_mismatch");
    const distribution = [];
    for (let i = 0; i < variantData.length; i++) {
      const v = variantData[i];
      distribution.push({
        variant: v.name,
        sessions: v.sessions,
        percentage: ((v.sessions / totalVisitors) * 100).toFixed(1) + "%"
      });
    }
    details.sampleRatioMismatch = {
      chiSquare: chiSquare.toFixed(2),
      trafficDistribution: distribution,
      message: "Traffic split is uneven"
    };
  }
}

if (controlVariation && controlVariation.cvr > 0) {
  for (let i = 0; i < variantData.length; i++) {
    const variant = variantData[i];
    if (!variant.isControl) {
      const cvrChange = (variant.cvr - controlVariation.cvr) / controlVariation.cvr;
      
      if (cvrChange < -THRESHOLDS.conversionDropThreshold) {
        if (!details.conversionRateDrop) {
          failedThresholds.push("conversion_rate_drop");
          details.conversionRateDrop = [];
        }
        const dropPercent = Math.abs(cvrChange * 100).toFixed(1);
        details.conversionRateDrop.push({
          variant: variant.name,
          change: cvrChange,
          message: variant.name + ": CVR down " + dropPercent + "% vs control"
        });
      }
    }
  }
}

if (failedThresholds.length > 0) {
  return [{
    json: {
      hasIssues: true,
      testName: testName,
      clientName: clientName,
      experienceId: experienceId,
      failedThresholds: failedThresholds,
      details: details,
      totalVisitors: totalVisitors,
      variantCount: variantData.length
    }
  }];
}

return [];
```

<figure><img src="/files/YcWsXi5yNi1vvH0GfENE" alt=""><figcaption></figcaption></figure>

#### **Node 7: If hasIssues = True**

1. Add an **"If"** node
2. Set the condition as `{{ $json.hasIssues }}` is euqal to `true`

<figure><img src="/files/VWbNqEGi6dje82Krj8yI" alt=""><figcaption></figcaption></figure>

#### **Node 8: Send to Slack**

1. For the **"true"** response only, add a **"Slack"** node where the action is **"Send a message"**

<figure><img src="/files/0ujG0TJ2zbzLebUZjglA" alt=""><figcaption></figcaption></figure>

2. **Authentication**:

* Click "Create New Credential"
* Click "Connect my account" and follow the prompts

3. **Channel/Use**:

* Under "Send Message To" configure where you want this slack message to appear

4. **Message**:

* in "Message Text" input:

```
🔴 *TEST HEALTH ALERT*

*Client:* {{ $json.clientName }}
*Test:* <https://app.intelligems.io/experiment/{{ $json.experienceId }}|{{ $json.testName }}>
*Failed Threshold(s):* {{ $json.failedThresholds }}

*Action Required:* Review test setup and consider pausing if issues persist.
```

<figure><img src="/files/HN3UjxocC5xYDdAHM276" alt=""><figcaption></figcaption></figure>

### Step 3: Activate the Workflow <a href="#step-4-activate-the-workflow" id="step-4-activate-the-workflow"></a>

1. Once everything works, click **"publish"** at the top to make this workflow go live!
2. Your workflow will now run 3 times a day and notify you if an active test does not meet your threshold.

### Bonus: Automatically Pause Failing Tests <a href="#step-4-activate-the-workflow" id="step-4-activate-the-workflow"></a>

Follow [this guide](https://docs.intelligems.io/developer-resources/external-api/automations-and-guides/how-to-automatically-pause-failing-tests) to also automatically pause tests with issues.


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.intelligems.io/developer-resources/external-api/automations-and-guides/automated-test-alerting-for-slack.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.