Agent-Readiness Audit
Is your site visible to AI agents before they even read a word?
AI assistants and coding agents (Claude, ChatGPT, Perplexity, Cursor, GitHub Copilot) check five signals before they look at your content: can they crawl you, do you have a content map, do you have a sitemap with freshness dates, are your pages small enough to fit in their context window, and can they get clean text without parsing heavy HTML? This report answers all five in one pass.
What It Does
- Fetches and parses your
robots.txtto identify which AI crawlers are explicitly blocked - Checks for
llms.txtat your site root — the agent-readable content map that tells AI what your site covers - Checks for
sitemap.xmland whether entries carry<lastmod>dates — AI crawlers use these to prioritize recently updated content - Estimates token counts for your homepage and key pages — pages over 25,000 tokens may be skipped or truncated by agents
- Tests whether key pages expose clean Markdown endpoints (
.mdURL or?format=markdown), which reduces agent token overhead - Generates a ready-to-publish
llms.txtstarter file if your site doesn't have one — copy it to your server root to close the most common agent-readiness gap immediately
No API keys required — all probes are plain HTTP requests against your public site.
How to Generate a Report
🛠️ Step 1: Navigate to Agent-Readiness Audit
From the ai12z GEO portal, select Agent-Readiness from the navigation menu.
🛠️ Step 2: Click + Generate Report
Click the + Generate Report button in the top-right corner.
🛠️ Step 3: Enter Your Website URL
| Field | Description |
|---|---|
| Website URL | The fully-qualified root URL of your site (e.g. https://example.com) |
🛠️ Step 4: Submit and Wait
Click Submit Job. The analysis runs asynchronously. Click Refresh to check for completion.
🛠️ Step 5: View Your Report
Once complete, click View PDF for the full report. If llms.txt is missing, the generated starter file is included in the report.
Probe Coverage
| Probe | What It Checks | Why It Matters |
|---|---|---|
| robots.txt | Which AI crawlers are blocked (GPTBot, ClaudeBot, Google-Extended, PerplexityBot, etc.) | A single Disallow: / under the wrong user-agent makes you invisible to that AI platform |
| llms.txt | Presence, structure, token count, descriptions, and section headers | The emerging standard for telling AI agents what your site covers and where to look |
| Sitemap | Presence of sitemap.xml, total URL count, and <lastmod> date coverage | AI crawlers use <lastmod> to prioritize recently updated pages |
| Token counts | Estimated token size of homepage + key pages | Pages over 25k tokens exceed most agent context limits |
| Markdown availability | Whether pages respond to .md or ?format=markdown requests | Markdown responses have 30–60% lower token overhead than parsed HTML |
Scoring Dimensions
| Dimension | Weight | Description |
|---|---|---|
| robots.txt | 30% | No AI crawlers are blocked |
| llms.txt | 35% | Present, well-formed, under 5,000 tokens |
| Sitemap | 20% | Present with <lastmod> dates on all entries |
| Token counts | 10% | Key pages fit within agent context budget |
| Markdown availability | 5% | Pages expose clean Markdown endpoints |
Verdict
| Verdict | Meaning |
|---|---|
| Ready | All five probes pass — excellent agent accessibility |
| Mostly Ready | Minor issues; no crawlers blocked; llms.txt present |
| Needs Work | One or more significant gaps (missing llms.txt, no sitemap, oversized pages) |
| Blocked | AI crawlers explicitly blocked by robots.txt |
Key Outputs
- Agent Readiness Score (0–100) — Overall score with breakdown across all five probes
- robots.txt Crawler Report — Pass/warn/fail per AI crawler with the exact blocking rules quoted
- llms.txt Quality Assessment — Found/missing, token count, structure issues, and improvement recommendations
- Generated llms.txt Starter File — Ready-to-publish when your site doesn't have one
- Sitemap Report — Found/missing, total URL count,
<lastmod>coverage %, freshest and stalest modification dates - Token Count Table — Per-page status (ok / warn / too large) with chunking recommendations
- Markdown Availability Map — Which pages have Markdown endpoints and which don't
- Prioritized Action Plan — Ranked by priority (critical → low) with effort estimates
- PDF Report — Complete audit document
Perfect For
- Developer-tool companies and API documentation sites
- Any business that has run a GEO or URL Analysis audit and wants to close the technical layer
- Marketing teams preparing for AI-powered discovery
- SEO specialists adding agent-readiness to their technical SEO checklist
- Developers implementing
llms.txtfor the first time
Example Use Case
A SaaS company runs the Agent-Readiness Audit and discovers: (1) their robots.txt blocks PerplexityBot with a broad Disallow: / left over from a bot-blocking campaign, (2) they have no llms.txt, (3) their sitemap.xml exists but has no <lastmod> dates, (4) their API reference page is 38,000 tokens — too large for most agent context windows.
The report generates a starter llms.txt, quotes the exact robots.txt line to remove, flags the sitemap freshness gap, and recommends splitting the API reference into per-endpoint pages. Within a week all four gaps are closed.