Automated Test Alerting for Slack

Learn how to build an automated workflow using n8n to monitor running tests for specific health metrics and receive instant alerts in Slack.

Overview:

This guide walks you through creating a daily automation that checks running tests for traffic or data issues (low visitors, SRM failures, large conversion drops, minimum orders per variant) and posts a Slack alert when a test fails health thresholds.

Softwares Used

To build this automated reporting pipeline, you will need the following tools:

• n8n: The primary workflow automation platform used to connect APIs and schedule tasks.

• Slack: The final destination where the AI-generated health checks and reports will be posted.

How to Create Your Automated Test Alerts

Step 1: Get Your Intelligems API Keys

[How to get Intelligems API Key]

Step 2: Create the Workflow in n8n

Node 1: Schedule Trigger

1. Click the "+" button to add a node

2. Search for "Schedule Trigger"

3. Configure it:

   • Trigger Interval: "Hours"

   • Hours Between Triggers: 8 (runs 3x daily: morning, afternoon, evening)

   • Or set to 12 for 2x daily

Node 2: Create Organization List

• Add "Code" node

• Select "Code in JavaScript"

• Select "Run Once for All Items"

• Paste this code. Update it with a display name for your client & their API key:

// Define all your Intelligems organizations
const organizations = [
  {
    name: "Client 1",
    apiKey: "YOUR_API_KEY_1"
  },
  {
    name: "Client 2", 
    apiKey: "YOUR_API_KEY_2"
  },
  {
    name: "Client 3",
    apiKey: "YOUR_API_KEY_3"
  }
];

// Return each org as a separate item for n8n to process
return organizations.map(org => ({ json: org }));

Node 3: Get All Running Tests

1. Click "+" after the Schedule node

2. Search for "HTTP Request"

3. Configure:

   • Method: GET

   • URL: https://api.intelligems.io/v25-10-beta/experiences-list

   • Authentication: None

   • Enable Send Headers

     • Name: intelligems-access-token

     • Value: {{ $json.apiKey }}

4. Click "Execute step" to verify it works

Node 4: Filter Tests by Status = "started"

• Add "Code" node

• Select "Code in JavaScript"

• Select "Run Once for All Items"

• Update the below code so that InsertOrgId maps to match the Intelligems organization IDs of your clients (you can find these in Intelligems under Settings > General > Organization Settings), the name value is display name for your clients, and the apiKey value is the Intelligems API Key for those clients. Then paste this code into the Code section in n8n.

// Filter for running tests with status = started
const items = $input.all();
const today = new Date();
const filteredTests = [];

// Static mapping of organization IDs to client names AND API keys
const clientMapping = {
  "InsertOrgId": {
    name: "Client 1",
    apiKey: "ig_live_11111"
  },
  "InsertOrgId": {
    name: "Client 2",
    apiKey: "ig_live_22222"
  }
};

for (const item of items) {
  // Get organization ID from the API response
  const orgId = item.json.organizationId || item.json.organization?.id || (item.json.experiencesList && item.json.experiencesList[0]?.organizationId);
  const clientData = clientMapping[orgId] || { name: "Unknown Client", apiKey: "" };
  
  // Access the experiencesList array
  const experiencesList = item.json.experiencesList;
  
  // Iterate through each test in the experiencesList
  if (experiencesList) {
    for (const test of experiencesList) {
      // Only check if status is "started"
      if (test.status === "started") {
        // Calculate days running for reporting purposes
        const startDate = new Date(test.startedAtTs);
        const daysRunning = Math.floor((today - startDate) / (1000 * 60 * 60 * 24));
        
        filteredTests.push({
          json: {
            clientName: clientData.name,
            apiKey: clientData.apiKey,
            experienceId: test.id,
            testName: test.name,
            daysRunning: daysRunning,
            weekNumber: Math.ceil(daysRunning / 7),
            startDate: startDate.toISOString().split('T')[0],  
            startDateReadable: startDate.toLocaleDateString('en-US', { month: 'short', day: 'numeric', year: 'numeric' }),
            endDate: today.toISOString().split('T')[0],
            endDateReadable: today.toLocaleDateString('en-US', { month: 'short', day: 'numeric', year: 'numeric' })
          }
        });
      }
    }
  }
}

return filteredTests;

Node 5: Get Test Analytics Data (Loop)

1. Add "HTTP Request" node

2. Configure:

   • Method: GET

   • URL: https://api.intelligems.io/v25-10-beta/analytics/resource/{{ $json.experienceId }}

   • Authentication: None

   • Enable Send Headers

     • Name: intelligems-access-token

     • Value: {{ $json.apiKey }}

3. Click "Execute step" to verify it works

Node 6: Check if below threshold

• Add "Code" node

• Select "Code in JavaScript"

• Select "Run Once for All Items"

• Update the below code so that the minTotalVisitors, minOrdersPerVariant, srmPValueThreshold, and conversionDropThreshold values match your threshold requirements. Then paste this code into the Code section in n8n.

const analytics = $input.item.json;
const filterData = $('Code in JavaScript1').item.json;
const testName = filterData.testName || "Unknown Test";
const clientName = filterData.clientName || "Unknown Client";
const experienceId = filterData.experienceId;

const THRESHOLDS = {
  minTotalVisitors: 1000,
  minOrdersPerVariant: 50,
  srmPValueThreshold: 0.01,
  conversionDropThreshold: 0.20
};

const overview = analytics.overview || analytics;
const variations = overview.variations || [];

if (!variations || variations.length === 0) {
  return [];
}

let totalVisitors = 0;
let controlVariation = null;
const variantData = [];

for (let i = 0; i < variations.length; i++) {
  const variant = variations[i];
  const sessions = variant.sessions || 0;
  const orders = variant.orders || 0;
  const cvr = sessions > 0 ? orders / sessions : 0;
  
  totalVisitors = totalVisitors + sessions;
  
  const isControl = variant.isControl || variant.name === "Control" || variant.label === "Control";
  
  const variantInfo = {
    name: variant.name || variant.label || "Variant " + i,
    sessions: sessions,
    orders: orders,
    cvr: cvr,
    isControl: isControl
  };
  
  variantData.push(variantInfo);
  
  if (isControl) {
    controlVariation = variantInfo;
  }
}

const failedThresholds = [];
const details = {};

if (totalVisitors < THRESHOLDS.minTotalVisitors) {
  failedThresholds.push("min_total_visitors");
  details.minTotalVisitors = {
    current: totalVisitors,
    required: THRESHOLDS.minTotalVisitors,
    message: "Only " + totalVisitors + " visitors (need " + THRESHOLDS.minTotalVisitors + ")"
  };
}

for (let i = 0; i < variantData.length; i++) {
  const variant = variantData[i];
  if (variant.orders < THRESHOLDS.minOrdersPerVariant) {
    if (!details.minOrdersPerVariant) {
      failedThresholds.push("min_orders_per_variant");
      details.minOrdersPerVariant = [];
    }
    details.minOrdersPerVariant.push({
      variant: variant.name,
      current: variant.orders,
      required: THRESHOLDS.minOrdersPerVariant,
      message: variant.name + ": Only " + variant.orders + " orders"
    });
  }
}

if (variantData.length >= 2 && totalVisitors > 0) {
  const expectedPerVariant = totalVisitors / variantData.length;
  let chiSquare = 0;
  
  for (let i = 0; i < variantData.length; i++) {
    const observed = variantData[i].sessions;
    const expected = expectedPerVariant;
    chiSquare = chiSquare + Math.pow(observed - expected, 2) / expected;
  }
  
  const df = variantData.length - 1;
  let isSRM = false;
  
  if (df === 1 && chiSquare > 6.635) {
    isSRM = true;
  } else if (df === 2 && chiSquare > 9.210) {
    isSRM = true;
  } else if (df >= 3 && chiSquare > 11.345) {
    isSRM = true;
  }
  
  if (isSRM) {
    failedThresholds.push("sample_ratio_mismatch");
    const distribution = [];
    for (let i = 0; i < variantData.length; i++) {
      const v = variantData[i];
      distribution.push({
        variant: v.name,
        sessions: v.sessions,
        percentage: ((v.sessions / totalVisitors) * 100).toFixed(1) + "%"
      });
    }
    details.sampleRatioMismatch = {
      chiSquare: chiSquare.toFixed(2),
      trafficDistribution: distribution,
      message: "Traffic split is uneven"
    };
  }
}

if (controlVariation && controlVariation.cvr > 0) {
  for (let i = 0; i < variantData.length; i++) {
    const variant = variantData[i];
    if (!variant.isControl) {
      const cvrChange = (variant.cvr - controlVariation.cvr) / controlVariation.cvr;
      
      if (cvrChange < -THRESHOLDS.conversionDropThreshold) {
        if (!details.conversionRateDrop) {
          failedThresholds.push("conversion_rate_drop");
          details.conversionRateDrop = [];
        }
        const dropPercent = Math.abs(cvrChange * 100).toFixed(1);
        details.conversionRateDrop.push({
          variant: variant.name,
          change: cvrChange,
          message: variant.name + ": CVR down " + dropPercent + "% vs control"
        });
      }
    }
  }
}

if (failedThresholds.length > 0) {
  return [{
    json: {
      hasIssues: true,
      testName: testName,
      clientName: clientName,
      experienceId: experienceId,
      failedThresholds: failedThresholds,
      details: details,
      totalVisitors: totalVisitors,
      variantCount: variantData.length
    }
  }];
}

return [];

Node 7: If hasIssues = True

1. Add an "If" node

2. Set the condition as {{ $json.hasIssues }} is euqal to true

Node 8: Send to Slack

1. For the "true" response only, add a "Slack" node where the action is "Send a message"

1. Authentication:

• Click "Create New Credential"

• Click "Connect my account" and follow the prompts

1. Channel/Use:

• Under "Send Message To" configure where you want this slack message to appear

1. Message:

• in "Message Text" input:

🔴 *TEST HEALTH ALERT*

*Client:* {{ $json.clientName }}
*Test:* <https://app.intelligems.io/experiment/{{ $json.experienceId }}|{{ $json.testName }}>
*Failed Threshold(s):* {{ $json.failedThresholds }}

*Action Required:* Review test setup and consider pausing if issues persist.

Step 3: Activate the Workflow

1. Once everything works, click "publish" at the top to make this workflow go live!

2. Your workflow will now run 3 times a day and notify you if an active test does not meet your threshold.

Bonus: Automatically Pause Failing Tests

Follow this guide to also automatically pause tests with issues.