Scheduled Analysis: Building 7 & 14-Day Post-Launch Test Reports
Automate your reporting with AI-generated 7 and 14-day test summaries delivered directly to Slack threads.
Overview
This guide shows you how to build a fully automated reporting engine that keeps both your team and your clients informed without manual effort.
Every day, the workflow checks for tests that are either 7 or 14 days old, pulls fresh Intelligems results, leverages AI to generate an in-depth Internal CRO Strategist Report, and crafts a concise, client-ready summary—all delivered directly into a threaded Slack message for easy discussion.
AI can make mistakes. Always check it's work before sharing with a client.
Softwares Used
To build this automated reporting pipeline, you will need the following tools:
n8n: The primary workflow automation platform used to connect APIs and schedule tasks.
Groq: A high-speed AI inference engine used to process test data and generate natural language reports.
Note: You can swap this for OpenAI or Anthropic if preferred, but this guide uses Groq for its free API tier.
Slack: The final destination where the AI-generated health checks and reports will be posted.
Node 4: Filter Tests by Status = "started" AND Duration
Now we need to filter for both status = "started" AND tests at 7 or 14 days:
Add "Code" node
Select "Code in JavaScript"
Select "Run Once for All Items"
Update the below code so that InsertOrgId maps to match the Intelligems organization IDs of your clients (you can find these in Intelligems under Settings > General > Organization Settings), the name value is display name for your clients, and the apiKey value is the Intelligems API Key for those clients. Then paste this code into the Code section in n8n.
// Define all your Intelligems organizations
const organizations = [
{
name: "Client 1",
apiKey: "YOUR_API_KEY_1"
},
{
name: "Client 2",
apiKey: "YOUR_API_KEY_2"
},
{
name: "Client 3",
apiKey: "YOUR_API_KEY_3"
}
];
// Return each org as a separate item for n8n to process
return organizations.map(org => ({ json: org }));
// Filter for running tests that are at 7 or 14 days old
const items = $input.all();
const today = new Date();
const filteredTests = [];
// Static mapping of organization IDs to client names AND API keys
const clientMapping = {
"InsertOrgId": {
name: "Client 1",
apiKey: "ig_live_11111"
},
"InsertOrgId": {
name: "Client 2",
apiKey: "ig_live_22222"
}
};
for (const item of items) {
// Get organization ID from the API response
const orgId = item.json.organizationId || item.json.organization?.id || (item.json.experiencesList && item.json.experiencesList[0]?.organizationId);
const clientData = clientMapping[orgId] || { name: "Unknown Client", apiKey: "" };
// Access the experiencesList array
const experiencesList = item.json.experiencesList;
// Iterate through each test in the experiencesList
if (experiencesList) {
for (const test of experiencesList) {
// First check if status is "started"
if (test.status !== "started") {
continue;
}
// Then check if it's been 7 or 14 days
const startDate = new Date(test.startedAtTs);
const daysRunning = Math.floor((today - startDate) / (1000 * 60 * 60 * 24));
// Check if test is at 7 or 14 days
if (daysRunning === 7 || daysRunning === 14) {
filteredTests.push({
json: {
clientName: clientData.name,
apiKey: clientData.apiKey,
experienceId: test.id,
testName: test.name,
daysRunning: daysRunning,
weekNumber: daysRunning === 7 ? 1 : 2,
startDate: startDate.toISOString().split('T')[0],
startDateReadable: startDate.toLocaleDateString('en-US', { month: 'short', day: 'numeric', year: 'numeric' }),
endDate: today.toISOString().split('T')[0],
endDateReadable: today.toLocaleDateString('en-US', { month: 'short', day: 'numeric', year: 'numeric' })
}
});
}
}
}
}
return filteredTests;
Analyze the A/B test data below and create a Slack-formatted report that matches this EXACT structure:
*A/B TEST RESULTS: [Test Name]*
📊 *Test Overview*
• Run Period: {{ $('Code in JavaScript1').item.json.startDateReadable }} to {{ $('Code in JavaScript1').item.json.endDateReadable }} ({{ $('Code in JavaScript1').item.json.daysRunning }} days)
• Traffic Split: [X] sessions per variant
• Conversions: [Variant 1] ([X] orders) | [Variant 2] ([X] orders)
💰 *Performance Comparison*
For each variant, display results in this order:
*[Use actual variant name from data, e.g., $Off or %Off]:*
Revenue Per Visitor: [+/- X.XX%] (confidence: [XX%])
Conversion Rate: [+/- X.XX%] (confidence: [XX%])
Average Order Value: [+/- X.XX%] (confidence: [XX%])
Profit Per Visitor: [+/- X.XX%] (confidence: [XX%]) // include only if COGS data exists
Projected Monthly Impact: [+/- $X,XXX]
Sample: [X] orders from [X] sessions
🔍 *Key Insights*
*Signal Strength:* [Strong / Moderate / Weak / Insufficient]
- Explain what the data pattern shows
- Identify which metrics are aligned vs. conflicting
- Note any segments or patterns worth investigating
*DECISION FRAMEWORK*
*Sample Adequacy:* [Met / Not Met]
✓ Minimum 7-day runtime including weekend
✓ Minimum 350 conversions per variant
✓ Minimum 8,000 sessions per variant
✅ *Decision:* [IMPLEMENT / ITERATE / ABANDON / EXTEND]
IMPLEMENT if:
- Sample adequacy met
- Primary metric shows ≥4% improvement
- Confidence level ≥85%
- Downside risk is acceptable (lower bound of credible interval ≥-2%)
ABANDON if:
- Sample adequacy met
- Primary metric shows ≤-4% decline
- Confidence that variant is worse ≥85%
- Upside potential is negligible (upper bound ≤+2%)
ITERATE if:
- Sample adequacy met
- Mixed signals (one key metric up, another down)
- OR improvement exists but confidence <85% and test passed planned end date
- Recommendation: [specific iteration to test next]
EXTEND if:
- Sample adequacy not met
- OR confidence between 70-85% with meaningful trend (≥3% change)
- Estimated runway: [X] additional days needed
*Rationale:* [1-2 sentences explaining the decision]
🔗 *View Full Results*
https://app.intelligems.io/experiment/{{ $('Code in JavaScript1').item.json.experienceId }}
Formatting Rules
- Use one asterisks * for bold text (e.g., *Variant A*)
- Round percentages to 2 decimal places (e.g., +3.47%, -2.03%)
- Round confidence scores to whole numbers (e.g., 87%, 92%)
- Format revenue with commas, no decimals (e.g., +$8,450, -$3,200)
- Use proper em dashes (—) for breaks in text
- Keep spacing consistent
- For confidence, use the p2bc value: if p2bc = 0.87, display as "87%"
- Include emojis exactly as shown: 📊 💰 🔍 📋 ✅ 🔗
- Use bullet points with - symbol
- DO NOT include the word "text:" or any JSON formatting in your output
- Output only the formatted message content, no JSON wrapper
Decision Calculation Guidelines
Determining Confidence: Use p2bc (probability to beat control) directly as the confidence percentage
Estimating Extension Time:
- Calculate average daily session volume: total sessions ÷ days run
- Determine sessions needed for 8,000 minimum per variant
- For confidence building: if confidence is 70-75%, assume need 80% more data; if 75-85%, assume need 40% more data
- Convert to days and cap at 14 additional days maximum
- If >14 days needed, classify as ITERATE instead
Signal Strength Classification:
- Strong: confidence ≥85% and absolute change ≥4%
- Moderate: confidence 70-85% OR absolute change 3-4%
- Weak: confidence 50-70% OR absolute change 1-3%
- Insufficient: confidence <50% OR duration <7 days
Handling Multiple Variants:
- Report all variants separately
- In decision section, identify best-performing variant
- If multiple variants show promise, note this in rationale
- IMPORTANT: Use the actual variant names from the data (like "$Off", "%Off", "Variant A", etc.), do NOT use placeholder text like "[Variant Name]"
Test Name: {{ $('Code in JavaScript1').item.json.testName }}
Days Running: {{ $('Code in JavaScript1').item.json.daysRunning }}
Input Data
{{ JSON.stringify($json, null, 2) }}
Transform this in-depth test analysis into a concise client update following the format below.
Use straightforward, conversational language—skip the industry lingo and statistical terminology.
The goal is for clients to instantly grasp the results without having to figure out what anything means.
# Required Structure
Hi CLIENT NAME, quick update on the insert test name test:
*Current Results*: The test has been running for [X days/weeks] with [X,XXX] visitors per variant. [Variant A/B/C/D] is currently leading with a [X%] lift in [primary metric]. We're at [XX%] statistical confidence, so [we're close to calling a winner / we need another week or two of data / we can confidently move forward].
*Key Metrics:*
• Conversion Rate: [Variant A: X%] vs [Variant B: X%]
• Revenue per Visitor: [Variant A: X%] vs [Variant B: X%]
• Average Order Value: [Variant A: X%] vs [Variant B: X%]
• Est. Monthly Rev Change for winning variant: *+/-$[number]*
*Intelligems Link:* <https://app.intelligems.io/experiment/{{ $('Code in JavaScript1').item.json.experienceId }}|HERE>
*What's next:* [e.g., "Planning to let this run until end of week to hit 95% confidence, then we'll implement the winner" / "Results are clear - I'll prepare a rollout plan for the winning variant" / "Considering a follow-up test based on what we're seeing"]
Happy to discuss in more detail if you have questions!
# Rules
Keep it more concise and easier to digest than internal reporting.
Focus on the most important numbers—skip the rest.
Format percentages with one decimal place (+2.1%, -15.7%), except drop the decimal when it's zero (+2% not +2.0%).
Format estimated monthly revenue changes as whole numbers with comma separators (+$12,395 or -$7,123).
Write in a relaxed, assured tone that sounds natural.
Stick to only the specified section headers.
Bold text by wrapping it in single asterisks (*like this*).
Begin each line in the Key Metrics section with a bullet point (•). DO NOT use dashes, use the bullet character.
# Internal CRO test report:
{{ $json.output }}
Client: {{ $('Code in JavaScript1').item.json.clientName }}
Test: <https://app.intelligems.io/experiment/{{ $('Code in JavaScript1').item.json.experienceId }}|{{ $('Code in JavaScript1').item.json.testName }}> - reached {{ $('Code in JavaScript1').item.json.daysRunning === 7 ? '7 days' : '14 days' }}
