Skip to main content

Testing Checklist

Use this checklist to validate your ai12z deployment at each stage — from initial setup through production launch. Work through each section top to bottom before advancing to the next phase.

1. Account & Agent Setup​

  • Account created and email verified
  • Agent created with correct Organization name (case-sensitive — appears as the bot header)
  • URL of Organization is accurate and publicly accessible
  • Purpose field clearly describes the agent's scope
  • Automate full agent creation was selected (or manual setup is complete)
  • Website ingestion confirmation email received

2. Content Ingestion​

  • Website crawl completed — review Knowledge Base for expected pages
  • Any missing pages added manually via URL ingestion or file upload
  • PDFs and documents uploaded where needed
  • No broken or 404 source URLs in the connector list
  • Content re-ingested after major website updates

Connectors (if used)​

  • CMS connector configured (Contentstack, Drupal, WordPress, Optimizely, Sitecore, etc.) and sync completed
  • AWS S3 connector configured — bucket name, region, and credentials verified
  • Connector sync completed and ingested content visible in the Knowledge Base
  • Re-sync triggered after content updates in the source system

If a CMS connector is configured, it reads content directly from the CMS, not the live site, so the checks below don't apply.

Web Scraping (if used, no CMS connector)​

  • Ingested page content spot-checked in the Vector tab: real page text, not a challenge/CAPTCHA page or empty content
  • If content looks blocked, garbled, or empty: site's CDN/bot protection likely needs a crawler exception, see Whitelisting the ai12z Crawler
  • ai12z user-agent (ai12zCopilot/1.0) enabled in ingestion settings if the site requires identifying the crawler explicitly

Guides: Content Ingestion, Knowledge Base, Connectors, AWS S3, Whitelisting the ai12z Crawler

3. AI Settings​

Answer AI​

  • System prompt reviewed — tone, purpose, and organization details are accurate
  • "I don't know" behavior confirmed — AI declines out-of-scope questions gracefully
  • Temperature set appropriately for your use case (default 0.0 for factual content)
  • Security & Role Integrity section included in system prompt
  • Bad Actor Detection ([directive=badActor]) enabled
  • If using {attributes}: security wrapper block present in system prompt

ReAct (if using integrations)​

  • Integrations are connected and responding correctly
  • Fallback to Answer AI confirmed when no integration matches
  • Security & Role Integrity section included in ReAct system prompt
  • Bad Actor Detection enabled in ReAct system prompt
  • If using {attributes}: security wrapper block present in ReAct system prompt
  • Max Agent Calls per Question configured to prevent runaway loops

Guides: Answer AI, ReAct, Security Checklist

4. Bot Control (<ai12z-bot>)​

  • Correct agent selected in the bot configuration
  • Welcome screen message and Quick Answer buttons reviewed and accurate
  • Theme / brand colors match your organization's style
  • Bot opens and closes correctly on your page
  • Bot logo and avatar display correctly
  • Privacy Statement link (if required) is present and functional
  • Tested on mobile as well as desktop
  • reCAPTCHA v3 enabled (required for public-facing deployments)
  • reCAPTCHA score threshold set (recommended: 0.5)
  • Embed script loads without console errors on the target page

Guide: Bot Configuration

5. CTA Search (<ai12z-cta>)​

  • CTA search modal opens correctly via button/trigger
  • Quick Answer buttons on the welcome screen work and return relevant results
  • Search returns accurate, relevant results for representative queries
  • Modal branding (logo, theme, colors) matches your site
  • reCAPTCHA v3 enabled with Site Key, Secret Key, and Failed Message configured
  • Liveagent tab configured (if using Salesforce live agent escalation)
  • Tested on mobile
  • Embed script loads without console errors

Guide: CTA Configuration

6. Forms (if used)​

  • All form questions are accurate and in the correct order
  • Required fields enforce submission correctly
  • Form submission delivers data to the expected destination
  • Success/thank-you message displays after submission
  • Form renders correctly inside the bot conversation
  • Tested via Test Drive end-to-end (open bot → trigger form → complete → submit)

Guide: Forms

7. Integrations & Custom Integrations (if used)​

Out-of-the-Box Integrations​

  • Required integrations added to the agent (Date/Time, Google Maps, Google Places, Email, SMS, Weather, Stock, Salesforce, etc.)
  • Each integration tested in Test Drive — trigger a query that invokes the integration and verify the response
  • API keys / credentials configured where required (SMTP credentials, Salesforce credentials, etc.)
  • Integrations respond within an acceptable time window
  • Fallback to Answer AI confirmed when no integration matches the query

Custom Integrations (JSON REST / GraphQL / MCP)​

  • Endpoint URL, method, and headers configured correctly
  • Authentication (API key, Bearer token, OAuth) working
  • Request parameters and data mappings verified
  • Response handling / JSONata transformation returns expected output
  • Integration tested using the built-in Test Agent tool in the custom integration editor
  • Integration response renders correctly in the bot (plain text, HTML widget, carousel, etc.)
  • Edge cases tested: empty response, error response, timeout
  • ReAct correctly selects this integration for relevant user queries

Guides: Integrations, Custom Integrations

8. Calls to Action (CTAs)​

CTAs drive users toward goals — booking, sign-ups, contact forms, recommendations — directly inside the conversation. Test both button-based and HTML template CTAs before going live.

CTA Buttons​

  • CTA buttons defined in the system prompt for key scenarios (activity listings, sign-ups, product options, etc.)
  • Button text is clear and action-oriented
  • sendMessage() onclick, trigger on each button invokes the correct follow-up response from the agent
  • class ai12zBtn
  • Example in answerAI system prompt
  • Buttons render correctly inside the bot conversation (not broken HTML)
  • Buttons tested in Test Drive end-to-end — click → agent responds as expected
  • Buttons display correctly on mobile

HTML Template CTAs (Advanced)​

  • Custom Integration created with Handle Response: Template (HTML)
  • Integration Description text clearly describes the trigger scenario so ReAct calls it at the right time
  • HTML template renders correctly inside the chat window
  • Any form inputs, radio buttons, or interactive elements within the template work correctly
  • ai12zBot.sendMessage() or ai12zBot.sendJSON() calls in the template trigger the expected follow-up agent behavior
  • Multi-step wizard or guided flow tested end-to-end
  • Template tested on mobile (layout does not overflow or break)
  • Edge case tested: user skips or abandons the CTA mid-flow

Guides: CTA Buttons, HTML Templates as CTAs

9. Directives & Persona Personalization (if used)​

Directives​

Directives allow the AI to control the browser in real-time — navigating URLs, switching UI panels, showing forms, triggering animations, and more — by embedding invisible tags in its responses.

  • Directives defined in the system prompt using the correct format: [directive=directiveName] or [directive=directiveName data="value"]
  • messageReceived event listener added to the page to receive and process directive data
  • Each directive tested end-to-end: trigger the scenario in Test Drive → confirm the expected browser action fires
  • [directive=urlChange data="..."] tested — browser navigates to the correct URL
  • [directive=badActor] fires when abuse patterns are detected (see Security Checklist)
  • Custom directives (if any) tested for correct data extraction and client-side handling
  • Directive tags are stripped from the visible response text — user never sees raw [directive=...] strings

Intent + Persona Personalization​

  • Feature flag enabled: dataAttributes.personaEnable = true set on the <ai12z-bot> element
  • personaCatalog defined — all user personas documented with descriptions and next actions
  • AI Settings → Intent + Persona Personalization configured (prompt, model, confidence threshold)
  • profile (customer profile data) passed in dataAttributes where available
  • Persona detection tested in conversation — verify correct persona is matched for representative queries
  • next actions for each persona tested end-to-end (navigate to URL, show form, display CTA, switch panel, etc.)
  • activePersonaId updated on the client after a persona match to maintain state across the session
  • Confidence threshold validated — low-confidence matches do not trigger incorrect persona actions
  • Tested with no profile data (anonymous user) — fallback behavior is acceptable

Guides: Directives, Intent + Persona Personalizer, Persona Detection API, Portal Admin — Persona

10. Test Drive​

  • Test Drive accessed from the Agent left-hand menu
  • Bot launches correctly in Test Drive
  • Representative questions tested and answers are accurate
  • Any inaccurate answers traced to source content and corrected
  • Welcome screen and Quick Answers display as expected
  • Background image uploaded for stakeholder share link (recommended)
  • Share link generated and reviewed by at least one team member

Guide: Test Drive

11. QA Copilot Tests​

Run automated conversation tests to validate agent behavior at scale.

  • At least one QA test created per primary use case or FAQ category
  • Multi-step workflow tests written for any form or escalation flow
  • All tests pass individually (Run Test)
  • Full test suite passes (Run All Tests)
  • Failed tests investigated — root cause identified (system prompt, content, or form config)
  • Tests re-run after any system prompt or content changes

Guide: QA Copilot Tests

12. Deployment of Web Controls​

Deploy the <ai12z-bot>, <ai12z-cta>, and/or Search Result controls to your website using one of the two methods below.

Option A — Direct Template Integration​

  • CDN <script> and <link> tags added to the <head> of your site template (all pages that should show the control)
  • <ai12z-bot data-key="YOUR_API_KEY"> (or <ai12z-cta> / search result tag) placed in the <body> of the template
  • API key confirmed correct — copied from the Web Controls configuration page
  • Control renders correctly on the live template (not just Test Drive)
  • No JavaScript console errors on page load
  • Control tested on at least one staging/preview URL before pushing to production

Option B — Google Tag Manager​

  • New Custom HTML tag created in GTM
  • GTM script snippet added (loads CDN library + calls window.Bot.init())
  • <ai12z-bot> (or other control tag) injected via a separate Custom HTML tag or present in the page template
  • Trigger set to All Pages (or scoped trigger if the control should only appear on specific pages)
  • GTM container Published after adding the tags
  • Control verified loading correctly on the live site after GTM publish
  • No conflicts with other GTM tags (ad scripts, analytics, etc.)

Both Methods​

  • Control appears and functions correctly across major browsers (Chrome, Safari, Firefox, Edge)
  • Control tested on mobile devices
  • Page performance acceptable — CDN script loads asynchronously and does not block rendering

Guides: Bot Integration, Google Tag Manager

13. Analytics & Logs​

  • Analytics dashboard shows sessions after test conversations
  • Logs reviewed for any unexpected errors or failed responses
  • Data tab reviewed for submitted form entries (if applicable)
  • UTM string defined and agreed with analytics/marketing team (e.g., utm_source=chatbot&utm_medium=cta&utm_campaign=launch)
  • Built-in UTM Parameters configuration enabled in the bot or CTA config
  • utm_source, utm_medium, and utm_campaign values match your campaign taxonomy
  • UTM parameters append correctly to links in bot/CTA responses — verify in browser DevTools that <a href> values include the UTM string
  • Existing utm_ parameters are skipped, merged, or overwritten according to the selected behavior
  • UTM parameters flow through correctly to your analytics platform (GA4, Adobe Analytics, etc.) — confirm campaign appears in reports after a test click
  • Preview/Test URL checked to confirm the final rewritten URL is correct

Guides: Analytics, Logs

14. Billing & Subscription​

  • Subscription plan selected appropriate for expected usage volume
  • AI model selected (GPT-4.1-mini recommended for production)
  • Plan Calculator used to estimate monthly cost
  • Payment method configured
  • For partner/sub-orgs: subscription funded correctly from partner credit pool or own billing

Guide: Subscription & Plans

15. Pre-Launch Security​

  • All items in the Security Checklist completed
  • reCAPTCHA v3 active on all public-facing controls
  • Max Questions per Session set in the Settings tab
  • System prompts include Security & Role Integrity, Bad Actor Detection, and (if applicable) Attributes Security

Filtered Content Responses (Organization Settings)​

  • Custom response messages reviewed for all four categories — Self-Harm/Crisis, Jailbreak/Prompt Injection, Active Threat/Violence Emergency, Default/Other Policy Violation (or platform defaults confirmed acceptable)
  • If your organization has specific crisis-line or emergency guidance, the Self-Harm/Crisis and Active Threat/Violence Emergency messages reflect it
  • Content Filter Reporting enabled if your security or compliance team needs alerts when a filter blocks a message
  • Reporting email address points to a monitored internal mailbox — alerts include the user's IP address and full conversation transcript, not just a notification
  • Email subject customized if needed (default: ai12z Content Filter Alert: {{contentFilterType}})

Guide: Filtered Content Responses

16. Go-Live​

  • Embed code added to the production website
  • Bot/CTA loads correctly on live pages (not just localhost or staging)
  • One final Test Drive on the production URL
  • QA Copilot Tests run one final time after going live
  • At least one team member monitoring Analytics and Logs in the first 48 hours
tip

After going live, re-run your QA Copilot test suite any time you update the system prompt, re-ingest content, or add/remove integrations.

17. AI Discoverability / GEO-AEO Suite (if used)​

This checklist covers core agent setup. If your organization is also measuring and improving how AI assistants like ChatGPT, Gemini, and Perplexity discover and cite your brand, that's a separate, ongoing capability — acknowledged here, not duplicated. It's typically set up after go-live, once real chatbot conversations and traffic exist to measure.

Guides: GEO Suite Overview, Get Found in AI Search — Customer Program, Get Found in AI Search — Agency Program

Last updated on