Agent-Readiness Audit
Is your site visible to AI agents before they even read a word?
AI assistants and coding agents (Claude, ChatGPT, Perplexity, Cursor, GitHub Copilot) check five signals before they look at your content: can they crawl you, do you have a content map, do you have a sitemap with freshness dates, are your pages small enough to fit in their context window, and can they get clean text without parsing heavy HTML? This report answers all five in one pass, and generates the missing pieces for you rather than just flagging them.
What It Does
- Fetches and parses your
robots.txtto identify which AI crawlers are explicitly blocked - Checks for
llms.txtat your site root: the agent-readable content map that tells AI what your site covers - Checks for
sitemap.xmland whether entries carry<lastmod>dates: AI crawlers use these to prioritize recently updated content, and the audit flags suspicious values (e.g. future-dated timestamps that signal a misconfigured generator) - Estimates token counts for your homepage and key pages: pages over 25,000 tokens may be skipped or truncated by agents
- Tests whether key pages expose clean Markdown endpoints (
.mdURL or?format=markdown), which reduces agent token overhead - Generates a ready-to-publish
llms.txtstarter file, pre-filled with your actual page titles, URLs, and descriptions: copy it straight to your server root to close the most common agent-readiness gap immediately
No API keys required: all probes are plain HTTP requests against your public site.
How to Generate a Report
🛠️ Step 1: Navigate to Agent-Readiness Audit
From the ai12z GEO portal, select Agent-Readiness from the navigation menu.
🛠️ Step 2: Click + Generate Report
Click the + Generate Report button in the top-right corner.
🛠️ Step 3: Enter Your Website URL
| Field | Description |
|---|---|
| Website URL* | Required. The fully-qualified root URL of your site (e.g. https://ai12z.com) |
🛠️ Step 4: Submit and Wait
Click Submit Job. The analysis runs asynchronously. Click Refresh to check for completion.
🛠️ Step 5: View Your Report
Once complete, click View PDF for the full report. If llms.txt is missing, the generated starter file is included directly in the report, ready to copy and publish.
What's Inside the PDF
| Section | Covers |
|---|---|
| Executive Summary | Overall Agent Readiness Score and verdict badge, the five probe scores as metric cards, and a highlighted Critical Actions callout |
| 01 · robots.txt Analysis | Pass/fail status, the actual relevant rules found, which (if any) AI crawlers are blocked, and specific fixes |
| 02 · llms.txt Analysis | Presence/absence, issues found, and (if missing) a full Generated Starter llms.txt file ready to copy and publish |
| 03 · Token Count Analysis | Estimated token size for the homepage and key pages, each flagged OK or oversized against the ~25,000-token agent context ceiling |
| 04 · Markdown Availability | Whether each key page responds to a .md URL, with the exact URL tried and a pass/fail per page |
| 05 · Sitemap | Total URL count, <lastmod> coverage %, and the freshest/stalest modification dates found, including sanity checks like future-dated timestamps |
| 06 · Prioritized Action Plan | Every probe's recommendation, ranked Critical → High → Medium → Low, each with expected impact and effort |
Probe Coverage
| Probe | What It Checks | Why It Matters |
|---|---|---|
| robots.txt | Which AI crawlers are blocked (GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot, etc.) | A single Disallow: / under the wrong user-agent makes you invisible to that AI platform |
| llms.txt | Presence, structure, token count, descriptions, and section headers | The emerging standard for telling AI agents what your site covers and where to look |
| Sitemap | Presence of sitemap.xml, total URL count, and <lastmod> date coverage, plus sanity-checking those dates | AI crawlers use <lastmod> to prioritize recently updated pages; a future-dated or clearly wrong timestamp undermines that signal |
| Token counts | Estimated token size of homepage + key pages | Pages over ~25k tokens exceed most agent context limits and risk partial ingestion |
| Markdown availability | Whether pages respond to .md or ?format=markdown requests | Markdown responses have roughly 30–50% lower token overhead than parsed HTML |
Scoring & Verdict
The Agent Readiness Score (0–100) reflects all five probes above. llms.txt and robots.txt carry the most weight in practice (a bad result on either can make your entire site invisible to a whole class of AI crawlers), but the exact formula isn't a simple flat average across the five probe scores, so don't try to hand-compute it from the individual numbers.
The score is paired with a short verdict badge and, directly beneath it, the four reference bands the report uses:
| Band | Score Range |
|---|---|
| Strong | 85–100 |
| Moderate | 70–84 |
| Needs Improvement | 45–69 |
| Needs Immediate Attention | 0–44 |
The badge text (e.g. "Mostly Ready" for a score of 68 in the example below) is a short, human-readable summary shown alongside these bands: treat the numeric band as the authoritative range and the badge as a friendly restatement of it, rather than assuming the two always use identical wording.
Key Outputs
- Agent Readiness Score (0–100): Overall score with a breakdown across all five probes
- Verdict Badge: Short human-readable summary (e.g. "Mostly Ready") paired with the four numeric reference bands
- Critical Actions Callout: The single highest-priority fix, surfaced directly in the Executive Summary as well as in the full action plan
- robots.txt Crawler Report: Pass/fail status, the actual rules found, and specific recommended additions
- llms.txt Quality Assessment: Found/missing, issues, and recommendations
- Generated llms.txt Starter File: A complete, ready-to-publish file pre-filled with your real page titles, URLs, and descriptions
- Token Count Table: Per-page estimated tokens and OK/oversized status
- Markdown Availability Table: The exact
.mdURL tried per page and whether it resolved - Sitemap Report: Total URL count,
<lastmod>coverage %, freshest and stalest modification dates, and date-sanity issues - Prioritized Action Plan: Every probe's recommendation ranked Critical → High → Medium → Low with effort and expected impact, note that even a probe scoring 95/100 typically still gets a lower-priority refinement, not just the failing ones
- PDF Report: Complete audit document
Perfect For
- Developer-tool companies and API documentation sites
- Any business that has run a GEO or URL Analysis audit and wants to close the technical layer
- Marketing teams preparing for AI-powered discovery
- SEO specialists adding agent-readiness to their technical SEO checklist
- Developers implementing
llms.txtfor the first time
Example Report Walkthrough
A real audit of https://ai12z.com/ produced an Agent Readiness Score of 68: "Mostly Ready."
Executive Summary cards read: robots.txt 95, llms.txt 0, Token Size 95, Markdown 20, Sitemap 95: a site that's broadly crawlable but missing the two assets that help agents work efficiently once they arrive.
01 · robots.txt passed at 95/100 with no AI crawlers blocked. The actual rules found were a wildcard User-agent: *, a Disallow: /wp-content/uploads/wpforms/ (correctly protecting form uploads), and a Sitemap: directive. The only note: there's no explicit allow-listing for individual bots like GPTBot, ClaudeBot, or PerplexityBot, acceptable today via the wildcard, but not future-proof as crawler behavior evolves.
02 · llms.txt scored 0/100: missing entirely. Rather than just flagging it, the report generated a complete starter file specific to the site's real pages:
# ai12z
> ai12z is an AI experience platform that delivers generative, adaptive, and
personalized web experiences powered by Generative Engine Optimization (GEO).
## Docs
- [Platform Overview](https://ai12z.com/platform): Overview of the ai12z
platform, its core capabilities, and how it powers AI-driven customer
experiences.
- [AI Experience Platform](https://ai12z.com/ai-experience-platform): Details
on the AI Experience Platform, including generative search, conversational AI,
and unified content orchestration.
## Home
- [ai12z Home](https://ai12z.com/): Landing page introducing the ai12z platform
and its value proposition.
That's copy-paste-ready for https://ai12z.com/llms.txt, no manual drafting required.
03 · Token Count scored 95/100. All five checked pages came back well under the ~25,000-token ceiling, ranging from 999 tokens (Adaptive Personalized Web Experiences) to 2,710 tokens (AI Experience Platform), comfortably ingestible in a single pass, with a suggestion to add TL;DR summaries to the longer ones anyway.
04 · Markdown Availability scored 20/100: 0 of 5 pages resolved a .md URL (e.g. https://ai12z.com/platform.md returned nothing). The report recommends publishing Markdown versions or supporting Accept: text/markdown content negotiation, prioritizing the homepage, /platform, and /ai-experience-platform first.
05 · Sitemap scored 95/100 with 535 total URLs, 100% <lastmod> coverage, but caught a real bug: the freshest <lastmod> was dated after the report's own generation date, a future-dated value the report flags as a likely misconfigured generator rather than real freshness.
06 · Prioritized Action Plan turned all five findings into ranked actions: Critical: publish the generated llms.txt (low effort); High: serve Markdown for the top three pages (medium effort); Medium: fix the future-dated sitemap timestamps, and separately, add explicit per-bot Allow rules to robots.txt (both low effort); Low: add TL;DR summaries to longer pages. Notice that even the two probes already scoring 95/100 (sitemap, robots.txt) still produced a Medium-priority action: a passing score doesn't mean there's nothing left to refine.
Related Documentation
- Q&A Analysis: Once agents can reach your site, find out what your customers are actually asking
- URL Analysis: Deep-dive the content quality of the specific pages this audit flags as high-value
- Site-Wide Sweep: Check token budget and structural signals across every indexed page, not just the homepage
- AI Visibility Suite Overview: How Agent-Readiness fits alongside the rest of the AI Visibility Suite