Skip to main content

Whitelisting the ai12z Crawler

Some websites run bot-protection services (Cloudflare, Akamai, Imperva, AWS WAF, and similar) that block or challenge automated crawlers by default. If a site you're trying to ingest is protected this way, the ai12z crawler can be silently blocked, and the ingested content will be empty, garbled, or a challenge/CAPTCHA page instead of the real content.

This only matters for web scraping (Add Website / Add URL). If you're using a CMS connector, the connector reads content directly from the CMS's API or database, never crawls the live site, and none of this applies. See Introduction to Connectors.

How to Tell This Is Happening​

After ingesting a page, open the document's Vector tab (or the histogram preview) and check the extracted content. Signs of a bot block instead of real content:

  • The page content is a Cloudflare (or similar) challenge screen: text like "Just a moment...", "Checking your browser," or "Enable JavaScript and cookies to continue."
  • The content is mostly script/style boilerplate with little or no readable page text.
  • The document's status shows a high rate of 403s or timeouts, but the same URL loads fine in a normal browser.

If you see this, the fix is whitelisting the crawler with the site's CDN/security provider, not adjusting your ingestion settings.

Crawler Identification​

Give this information to the client's IT or CDN/security team so they can add an exception:

FieldValue
User-Agentai12zCopilot/1.0
Gateway IP address52.0.5.12
HeaderRequests include Cf-Connecting-Ip: 52.0.5.12
HostingAll crawl traffic originates from Amazon AWS, from the IP address above

In the ingestion wizard, check ai12z user-agent (Website and Add URL, Step 3/2 Advanced) so requests identify as ai12zCopilot/1.0 rather than a generic browser user-agent. This makes it possible for a site to allow the crawler specifically, instead of opening access to all bots. See Website Ingestion and Add URL.

What to Ask the Client's Team to Do​

Most bot protection is configured at the CDN layer (Cloudflare is the most common). Ask the client's technical contact to:

  1. Add 52.0.5.12 to the IP allowlist (Cloudflare: IP Access Rules or Firewall Tools; other providers: the equivalent allowlist/bypass rule).
  2. Check WAF, rate-limiting, and bot-management rules (including "Bot Fight Mode") for anything that would block or challenge requests from that IP or from the ai12zCopilot/1.0 user-agent, and add an exception.
  3. Confirm the Cf-Connecting-Ip: 52.0.5.12 header is recognized if the site has any header-based restrictions.
  4. Check country-blocking rules. The crawler's traffic originates from the US; confirm US traffic from this IP isn't blocked even if other US traffic is allowed.
  5. Add an exception for CAPTCHAs or JavaScript challenges if the site presents them to new visitors, so the crawler isn't served a challenge page instead of content.

A ready-to-forward version of this request:

Subject: Whitelist Request: ai12z Copilot Crawler

To enable the ai12z Copilot to crawl and index our website for AI-powered search and assistance, please whitelist our crawler:

- User-Agent: ai12zCopilot/1.0
- Gateway IP address: 52.0.5.12
- Header: requests include Cf-Connecting-Ip: 52.0.5.12
- Hosting: all crawl traffic originates from Amazon AWS at the IP above

Please:
1. Add 52.0.5.12 to the IP allowlist (Cloudflare: IP Access Rules / Firewall Tools).
2. Check WAF, rate-limiting, and bot-management rules (including Bot Fight Mode) for anything
   that would block or challenge this IP or user-agent, and add an exception.
3. Confirm the Cf-Connecting-Ip header is recognized by any header-based restrictions.
4. Confirm US-based traffic from this IP is allowed if country-blocking is enabled.
5. Add an exception so this crawler isn't served a CAPTCHA or JavaScript challenge.

Let us know once this is whitelisted, or if you need any further details.

If Whitelisting Isn't Possible​

Not every client can get their IT or CDN team to make an exception quickly, or at all. A few fallbacks, roughly in order of how fast they unblock you:

  • Run the ai12z local crawler. ai12z has a crawler that runs locally, on your own machine, instead of from ai12z's cloud IP. Because it's not coming from a datacenter IP that bot-protection services already flag, it essentially always gets through, even against sites that are actively blocking the hosted crawler. This is the fastest fix while you're waiting on a whitelist approval, and it's especially useful for running a demo before a client has granted CMS connector access. Ask your ai12z contact for a copy, or ask the ai12z team to run the crawl for you.
  • Switch to a CMS connector, if one exists for the platform. This is the preferred long-term fix. It also removes the ongoing dependency on the site's bot protection staying configured correctly. See Introduction to Connectors.
  • Export the content and send it in as a file. If the client (or their CMS vendor) can export the pages as structured data, it can be ingested directly, without crawling the live site at all. See Custom Sitemap JSON and Custom Sitemap CSV.

Before You Escalate to the Client​

A few things to try first that don't require anyone else's help:

  • Request Type: Asynchronous or Advanced Mode (Website/URL ingestion, Step 3/2 Advanced) can get past some lighter JavaScript challenges on their own. See the Troubleshooting section of Website Ingestion.
  • Confirm the page actually loads for a normal, logged-out visitor in an incognito browser window. If it doesn't, the problem isn't the crawler.
  • Check robots.txt for an explicit Disallow rule. That's a separate, simpler fix (the crawler already honors robots.txt, so a blanket disallow needs to be addressed there directly, not through bot-protection whitelisting).
Last updated on