Testing Checklist

Use this checklist to validate your ai12z deployment at each stage — from initial setup through production launch. Work through each section top to bottom before advancing to the next phase.

1. Account & Agent Setup

Account created and email verified
Agent created with correct Organization name (case-sensitive — appears as the bot header)
URL of Organization is accurate and publicly accessible
Purpose field clearly describes the agent's scope
Automate full agent creation was selected (or manual setup is complete)
Website ingestion confirmation email received

2. Content Ingestion

Website crawl completed — review Knowledge Base for expected pages
Any missing pages added manually via URL ingestion or file upload
PDFs and documents uploaded where needed
No broken or 404 source URLs in the connector list
Content re-ingested after major website updates

Connectors (if used)

CMS connector configured (Contentstack, Drupal, WordPress, Optimizely, Sitecore, etc.) and sync completed
AWS S3 connector configured — bucket name, region, and credentials verified
Connector sync completed and ingested content visible in the Knowledge Base
Re-sync triggered after content updates in the source system

Guides: Content Ingestion, Knowledge Base, Connectors, AWS S3

3. AI Settings

Answer AI

System prompt reviewed — tone, purpose, and organization details are accurate
"I don't know" behavior confirmed — AI declines out-of-scope questions gracefully
Temperature set appropriately for your use case (default 0.0 for factual content)
Security & Role Integrity section included in system prompt
Bad Actor Detection ([directive=badActor]) enabled
If using {attributes}: security wrapper block present in system prompt

ReAct (if using integrations)

Integrations are connected and responding correctly
Fallback to Answer AI confirmed when no integration matches
Security & Role Integrity section included in ReAct system prompt
Bad Actor Detection enabled in ReAct system prompt
If using {attributes}: security wrapper block present in ReAct system prompt
Max Agent Calls per Question configured to prevent runaway loops

Guides: Answer AI, ReAct, Security Checklist

4. Bot Control (`<ai12z-bot>`)

Guide: Bot Configuration

5. CTA Search (`<ai12z-cta>`)

CTA search modal opens correctly via button/trigger
Quick Answer buttons on the welcome screen work and return relevant results
Search returns accurate, relevant results for representative queries
Modal branding (logo, theme, colors) matches your site
reCAPTCHA v3 enabled with Site Key, Secret Key, and Failed Message configured
Liveagent tab configured (if using Salesforce live agent escalation)
Tested on mobile
Embed script loads without console errors

Guide: CTA Configuration

6. Forms (if used)

All form questions are accurate and in the correct order
Required fields enforce submission correctly
Form submission delivers data to the expected destination
Success/thank-you message displays after submission
Form renders correctly inside the bot conversation
Tested via Test Drive end-to-end (open bot → trigger form → complete → submit)

Guide: Forms

7. Integrations & Custom Integrations (if used)

Out-of-the-Box Integrations

Required integrations added to the agent (Date/Time, Google Maps, Google Places, Email, SMS, Weather, Stock, Salesforce, etc.)
Each integration tested in Test Drive — trigger a query that invokes the integration and verify the response
API keys / credentials configured where required (Google Maps API key, SMTP credentials, Salesforce credentials, etc.)
Integrations respond within an acceptable time window
Fallback to Answer AI confirmed when no integration matches the query

Custom Integrations (JSON REST / GraphQL / MCP)

Endpoint URL, method, and headers configured correctly
Authentication (API key, Bearer token, OAuth) working
Request parameters and data mappings verified
Response handling / JSONata transformation returns expected output
Integration tested using the built-in Test Agent tool in the custom integration editor
Integration response renders correctly in the bot (plain text, HTML widget, carousel, etc.)
Edge cases tested: empty response, error response, timeout
ReAct correctly selects this integration for relevant user queries

Guides: Integrations, Custom Integrations

8. Calls to Action (CTAs)

CTAs drive users toward goals — booking, sign-ups, contact forms, recommendations — directly inside the conversation. Test both button-based and HTML template CTAs before going live.

CTA Buttons

CTA buttons defined in the system prompt for key scenarios (activity listings, sign-ups, product options, etc.)
Button text is clear and action-oriented
sendMessage() trigger on each button invokes the correct follow-up response from the agent
Buttons render correctly inside the bot conversation (not broken HTML)
Buttons tested in Test Drive end-to-end — click → agent responds as expected
Buttons display correctly on mobile

HTML Template CTAs (Advanced)

Custom Integration created with Handle Response: Template (HTML)
Integration Description text clearly describes the trigger scenario so ReAct calls it at the right time
HTML template renders correctly inside the chat window
Any form inputs, radio buttons, or interactive elements within the template work correctly
ai12zBot.sendMessage() or ai12zBot.sendJSON() calls in the template trigger the expected follow-up agent behavior
Multi-step wizard or guided flow tested end-to-end
Template tested on mobile (layout does not overflow or break)
Edge case tested: user skips or abandons the CTA mid-flow

Guides: CTA Buttons, HTML Templates as CTAs

9. Directives & Persona Personalization (if used)

Directives

Directives allow the AI to control the browser in real-time — navigating URLs, switching UI panels, showing forms, triggering animations, and more — by embedding invisible tags in its responses.

Directives defined in the system prompt using the correct format: [directive=directiveName] or [directive=directiveName data="value"]
messageReceived event listener added to the page to receive and process directive data
Each directive tested end-to-end: trigger the scenario in Test Drive → confirm the expected browser action fires
[directive=urlChange data="..."] tested — browser navigates to the correct URL
[directive=badActor] fires when abuse patterns are detected (see Security Checklist)
Custom directives (if any) tested for correct data extraction and client-side handling
Directive tags are stripped from the visible response text — user never sees raw [directive=...] strings

Intent + Persona Personalization

Feature flag enabled: dataAttributes.personaEnable = true set on the <ai12z-bot> element
personaCatalog defined — all user personas documented with descriptions and next actions
AI Settings → Intent + Persona Personalization configured (prompt, model, confidence threshold)
profile (customer profile data) passed in dataAttributes where available
Persona detection tested in conversation — verify correct persona is matched for representative queries
next actions for each persona tested end-to-end (navigate to URL, show form, display CTA, switch panel, etc.)
activePersonaId updated on the client after a persona match to maintain state across the session
Confidence threshold validated — low-confidence matches do not trigger incorrect persona actions
Tested with no profile data (anonymous user) — fallback behavior is acceptable

Guides: Directives, Intent + Persona Personalizer, Persona Detection API, Portal Admin — Persona

10. Test Drive

Test Drive accessed from the Agent left-hand menu
Bot launches correctly in Test Drive
Representative questions tested and answers are accurate
Any inaccurate answers traced to source content and corrected
Welcome screen and Quick Answers display as expected
Background image uploaded for stakeholder share link (recommended)
Share link generated and reviewed by at least one team member

Guide: Test Drive

11. QA Copilot Tests

Run automated conversation tests to validate agent behavior at scale.

At least one QA test created per primary use case or FAQ category
Multi-step workflow tests written for any form or escalation flow
All tests pass individually (Run Test)
Full test suite passes (Run All Tests)
Failed tests investigated — root cause identified (system prompt, content, or form config)
Tests re-run after any system prompt or content changes

Guide: QA Copilot Tests

12. Deployment of Web Controls

Deploy the <ai12z-bot>, <ai12z-cta>, and/or Search Result controls to your website using one of the two methods below.

Option A — Direct Template Integration

CDN <script> and <link> tags added to the <head> of your site template (all pages that should show the control)
<ai12z-bot data-key="YOUR_API_KEY"> (or <ai12z-cta> / search result tag) placed in the <body> of the template
API key confirmed correct — copied from the Web Controls configuration page
Control renders correctly on the live template (not just Test Drive)
No JavaScript console errors on page load
Control tested on at least one staging/preview URL before pushing to production

Option B — Google Tag Manager

New Custom HTML tag created in GTM
GTM script snippet added (loads CDN library + calls window.Bot.init())
<ai12z-bot> (or other control tag) injected via a separate Custom HTML tag or present in the page template
Trigger set to All Pages (or scoped trigger if the control should only appear on specific pages)
GTM container Published after adding the tags
Control verified loading correctly on the live site after GTM publish
No conflicts with other GTM tags (ad scripts, analytics, etc.)

Both Methods

Control appears and functions correctly across major browsers (Chrome, Safari, Firefox, Edge)
Control tested on mobile devices
Page performance acceptable — CDN script loads asynchronously and does not block rendering

Guides: Bot Integration, Google Tag Manager

13. Analytics & Logs

Analytics dashboard shows sessions after test conversations
Logs reviewed for any unexpected errors or failed responses
Data tab reviewed for submitted form entries (if applicable)

UTM Parameter Attribution (if tracking link clicks)

UTM string defined and agreed with analytics/marketing team (e.g., utm_source=chatbot&utm_medium=cta&utm_campaign=launch)
Custom JS script added to the CTA/bot config under Script → Custom JS
utmParams constant in the script updated to match your campaign taxonomy
UTM parameters append correctly to links in bot/CTA responses — verify in browser DevTools that <a href> values include the UTM string
Links that already contain utm_ are not double-tagged
UTM parameters flow through correctly to your analytics platform (GA4, Adobe Analytics, etc.) — confirm campaign appears in reports after a test click
Script tested after a follow-up/second message to confirm MutationObserver re-applies UTM on incremental renders
Any domains that should be excluded from UTM tagging added to the blocklist in the script

Guide: Append UTM Parameters

Guides: Analytics, Logs, Data Reports

14. Billing & Subscription

Subscription plan selected appropriate for expected usage volume
AI model selected (GPT-4.1-mini recommended for production)
Plan Calculator used to estimate monthly cost
Payment method configured
For partner/sub-orgs: subscription funded correctly from partner credit pool or own billing

Guide: Subscription & Plans

15. Pre-Launch Security

All items in the Security Checklist completed
reCAPTCHA v3 active on all public-facing controls
Max Questions per Session set in the Settings tab
System prompts include Security & Role Integrity, Bad Actor Detection, and (if applicable) Attributes Security

16. Go-Live

Embed code added to the production website
Bot/CTA loads correctly on live pages (not just localhost or staging)
One final Test Drive on the production URL
QA Copilot Tests run one final time after going live
At least one team member monitoring Analytics and Logs in the first 48 hours

tip

After going live, re-run your QA Copilot test suite any time you update the system prompt, re-ingest content, or add/remove integrations.

1. Account & Agent Setup​

2. Content Ingestion​

Connectors (if used)​

3. AI Settings​

Answer AI​

ReAct (if using integrations)​

4. Bot Control (<ai12z-bot>)​

5. CTA Search (<ai12z-cta>)​

6. Forms (if used)​

7. Integrations & Custom Integrations (if used)​

Out-of-the-Box Integrations​

Custom Integrations (JSON REST / GraphQL / MCP)​

8. Calls to Action (CTAs)​

CTA Buttons​

HTML Template CTAs (Advanced)​

9. Directives & Persona Personalization (if used)​

Directives​

Intent + Persona Personalization​

10. Test Drive​

11. QA Copilot Tests​

12. Deployment of Web Controls​

Option A — Direct Template Integration​

Option B — Google Tag Manager​

Both Methods​

13. Analytics & Logs​

UTM Parameter Attribution (if tracking link clicks)​

14. Billing & Subscription​

15. Pre-Launch Security​

16. Go-Live​