Skip to main content

Add URL

Overview​

Add a single web resource by pasting its URL. The platform fetches the content and adds it to your copilot’s knowledge base.

Supported URL types include:

  • Web pages (HTML)
  • Videos (YouTube links)
  • Documents hosted online: .pdf, .docx, .pptx, .xlsx, .csv, .txt, .json, .md/markdown

Adding content from a URL​

Adding a URL select Click the Add Document, select Add URL

Basic URL settings

The wizard has 2 steps: Basic Info and Advanced.

Step 1 — Basic Info​

Basic URL settings

  • Name: A recognizable label for this item.
  • URL: Paste the full URL to ingest.
  • Tags: (Optional) Apply one or more tags.
  • Description: (Optional) Short note about the content.

Step 2 — Advanced​

Advanced URL settings

  • Authentication Type:

    • None for public resources (default)
    • Basic for username/password
    • Token for token-based auth (beta)

  • Request Type:

    • Auto (default): ai12z chooses the best mode
    • Synchronous: Fastest for server-rendered pages
    • Asynchronous: Uses a headless browser for JS-rendered pages
    • Advanced Mode: For sites with extra protections — additional $0.0025 per page

  • Headers: Add custom HTTP headers (e.g., Authorization, Content-Type) when required by the site.

  • ai12z user-agent: Identify as ai12zCopilot/1.0. If unchecked, a common browser user-agent is used.

Advanced settings

These options are for specific needs. Non‑optimal values may cause blocked/partial fetches. Prefer a CMS connector when available. Only use Advanced Mode if Auto/Sync/Async is blocked.

Tips​

  • For JS-heavy pages that render content after load, choose Asynchronous.
  • If you see many 403/robots blocks, validate headers and user‑agent, or ingest via CMS/API.
  • Use tags to organize content for filtering and analytics later.

Meta Data Tags​

If ai12z detects a <meta> tag with the name tags, it will automatically apply these tags to the vector documents associated with the URL of the page.

Example:

<meta name="tags" content="blog, security, web" />

JSON-LD Ingestion​

If JSON-LD is found on a webpage, it will be included during the ingestion process.
By default, both the JSON-LD and the page content are ingested together.
You can configure this behavior in the Agent settings: if desired, you can disable page content ingestion and ingest only the JSON-LD.