Skip to main content

Add URL

Overview​

Add a single web resource by pasting its URL. The platform fetches the content and adds it to your copilot’s knowledge base.

Supported URL types include:

  • Web pages (HTML)
  • Videos (YouTube links)
  • Documents hosted online: .pdf, .docx, .pptx, .xlsx, .csv, .txt, .json, .md/markdown

Adding content from a URL​

Adding a URL select Click the Add Document, select Add URL

Basic URL settings

The wizard has 2 steps: Basic Info and Advanced.

Step 1 — Basic Info​

Basic URL settings

  • Name: A recognizable label for this item.
  • URL: Paste the full URL to ingest.
  • Tags: (Optional) Apply one or more tags.
  • Description: (Optional) Short note about the content.

Step 2 — Advanced​

Advanced URL settings

  • Authentication Type:

    • None for public resources (default)
    • Basic for username/password
    • Token for token-based auth (beta)

  • Request Type:

    • Auto (default): ai12z chooses the best mode
    • Synchronous: Fastest for server-rendered pages
    • Asynchronous: Uses a headless browser for JS-rendered pages
    • Advanced Mode: For sites with extra protections — additional $0.0025 per page

  • Headers: Add custom HTTP headers (e.g., Authorization, Content-Type) when required by the site.

  • ai12z user-agent: Identify as ai12zCopilot/1.0. If unchecked, a common browser user-agent is used. If the site is behind Cloudflare or similar bot protection, see Whitelisting the ai12z Crawler.

Advanced settings

These options are for specific needs. Non‑optimal values may cause blocked/partial fetches. Prefer a CMS connector when available. Only use Advanced Mode if Auto/Sync/Async is blocked.

Tips​

  • For JS-heavy pages that render content after load, choose Asynchronous.
  • If you see many 403/robots blocks, validate headers and user‑agent, or ingest via CMS/API.
  • If the fetched content looks garbled, empty, or like a challenge page, the site's CDN or bot protection is likely blocking the crawler; see Whitelisting the ai12z Crawler.
  • Use tags to organize content for filtering and analytics later.

Meta Data Tags​

If ai12z detects a <meta> tag with the name tags, it will automatically apply these tags to the vector documents associated with the URL of the page.

Example:

<meta name="tags" content="blog, security, web" />

JSON-LD Ingestion​

If JSON-LD is found on a webpage, it will be included during the ingestion process.
By default, both the JSON-LD and the page content are ingested together.
You can configure this behavior in the Agent settings: if desired, you can disable page content ingestion and ingest only the JSON-LD.

Last updated on