Every scan follows the same steps and the same fixed rules, so the same site gives the same result. Here is everything we fetch, every check we run, and how the score adds up.
Steps
A scan takes about a minute and makes about 30 to 40 requests to your site, never more than 50.
1. Check the addressWe only scan http and https addresses on ports 80 and 443, without a username or password. We look the name up in DNS once and check that every address it points to is public: never a private, loopback, link-local or otherwise reserved network. Then we connect to exactly that address, so DNS can't change in the middle of a scan.
2. Read robots.txtWe fetch /robots.txt as CanAgentsReadBot. If it disallows CanAgentsReadBot from the whole site, we stop there: no scan and no report.
3. Fetch as a normal browserWe fetch the address you submitted with a normal Chrome user agent. This is the baseline the agent fetches are compared with. We fetch it even if robots.txt disallows it, because a person asked for it.
4. Fetch as AI agentsWe fetch the same address as ChatGPT-User, OAI-SearchBot, Claude-User and PerplexityBot, each with its own user agent and accepting any format, and skip any agent robots.txt keeps out. Then once more as CanAgentsReadBot, asking for text/markdown.
5. Follow the Markdown linkIf the homepage's head links to a Markdown version on your site, we fetch it.
6. Read the sitemapWe look for Sitemap lines in robots.txt, then /sitemap.xml and /sitemap_index.xml, follow one level of sitemap index, and open up to five of the pages it lists.
7. Read llms.txtWe fetch /llms.txt and open up to five of its links that point to your site.
8. Sample pagesWe pick up to 7 more pages: About, Contact, Privacy and Pricing when the homepage links to them, then one page per top-level section from the homepage's links and the sitemap. Only pages on the same host, never files such as PDFs or images.
9. Probe for Markdown, feeds and agent filesWe request a random address that can't exist, to see how missing pages answer, then /index.md and the .md copies of three sampled pages. We also fetch the feed the homepage links to, /.well-known/api-catalog and /.well-known/agent-card.json.
10. Run 41 checksEvery check reads only the responses we already have. Nothing is fetched while the checks run, and no AI model is involved: each check is a fixed rule, so the same responses always give the same result.
11. Score and save the reportWe add up the score, save the report at canagentsread.com/<host>, and draw its share image.
Our bot and robots.txt
How we identify ourselves
Everything we fetch for ourselves carries this user agent:
The four agent fetches use each agent's own user agent instead, so you see exactly what those agents get. The baseline fetch uses a normal Chrome user agent.
What robots.txt controls
We follow robots.txt rules for CanAgentsReadBot, or the * rules when CanAgentsReadBot isn't named, for every page and probe. Three things are fetched regardless: the address you submitted, because a person asked for it; robots.txt itself; and your sitemap and /llms.txt, which exist for bots.
A page or probe robots.txt disallows is never fetched. The checks that need it say so instead of guessing.
How to opt out
Add this to your robots.txt:
User-agent: CanAgentsReadBot
Disallow: /
We then won't scan the site or show a report for it, and its older reports disappear too.
Limits
Every address someone submits is treated as untrusted, and every scan stays within these limits.
50 requests per scan, at most
Scans usually need 30 to 40.
10 seconds per request
And 5 seconds to connect.
2 MB per response
We stop reading a larger response and record it as too large.
5 redirects per request
Every hop is checked against the same safety rules as the first address.
One scan per site every 10 minutes
A rescan of the same site waits for the cooldown.
10 scans per visitor per hour
So the scanner stays quick for everyone.
Your data
What we keep
Every response we fetch (its status, headers and body) is kept with the scan, so reports can show the evidence and old scans can be rescored when a rule changes.
Reports are public at canagentsread.com/<host>. They only show what any visitor to your site could see.
What we never do
Run JavaScript. We read the HTML your server sends, as most agents do.
Sign in, fill in forms or click buttons.
Follow links to other sites.
Use an AI model to judge your site. Every check is a fixed rule.
How the score adds up
GroupPointsChecks
Getting in
Can an agent reach your pages and find the words?
75 points
5 checks
Making sense
Can it tell who you are and what each page is about?
15 points
11 checks
Extras
Newer habits some agents already look for.
10 points
25 checks
Within each group, a check's points are its share of the group. When a check doesn't apply, its points are shared out among the others, so a site isn't marked down for, say, not selling anything.
Partial results earn part of a check's points, as each check below describes.
Extras count like the other groups, so a score of 100 means every check that applies passes.
If AI agents are kept out, or the homepage's words aren't in its HTML, the score stays at 39 however well the rest does.
If 2 or more Getting in checks can't run, usually because a firewall turned our scanner away, the scan gets no score rather than a wrong one.
Every report shows its scoring version. When we change a rule or a weight, the version goes up, and the timeline marks the change.
How to read the score
0–39NoAgents can't reliably read the site.
40–64PartlyAgents miss important content.
65–84MostlyA few things to fix.
85–100YesAgents can read this site.
Every check
What each check looks for, why it matters to agents, and what earns all, part or none of its points.
Getting in
75 points · 5 checks
AI agents are let in
agent_access · 21 points
If an AI agent can't fetch your pages, nothing else matters: it can't read, quote or recommend you.
We fetch your homepage as ChatGPT-User, OAI-SearchBot, Claude-User and PerplexityBot, each with its own user agent, and compare what each gets with what a normal browser gets. An agent robots.txt disallows isn't fetched; it counts as kept out.
Passed. All four agents are allowed by robots.txt and get the page, with at least half the text a normal browser gets.
Partial. Some agents get in. Credit is the share of agents that do.
Failed. No agent gets the page: robots.txt keeps them out, or your server or firewall turns them away. A fail holds the whole score at 39.
When it can't count. Couldn't check when a normal browser is turned away too, because then we can't tell a block on agents from a block on our scanner.
Words are in the HTML
no_js_content · 21 points
Most AI agents don't run JavaScript. They read the HTML your server sends, so words added later by scripts are invisible to them.
Passed. At least 1,500 characters of visible text in the homepage's server HTML, exactly one H1 and at least one H2.
Partial. 300 to 1,499 characters, or the headings are missing: half the points.
Failed. Under 300 characters of text, so agents see an almost empty page. A fail holds the whole score at 39.
When it can't count. Couldn't check when the homepage doesn't load.
Pages are public, with real text
public_content · 13 points
Agents can't sign in. A page behind a login is a page they can't read or cite.
We look at the homepage and every page we sample.
Passed. Every page loads with at least 300 characters of text, and none redirects to a login page or answers 401 or 403.
Partial. Some pages are walled off or nearly empty. Credit is the share that are public.
Failed. The homepage itself asks visitors to sign in, or no page we read is public with real text.
When it can't count. Couldn't check when our scanner is turned away from the homepage (an error, 403, 429 or 503).
Redirects don't detour
clean_redirects · 11 points
Every extra hop is a chance for an agent to give up, and meta-refresh or JavaScript redirects don't work for agents that only read HTML.
Passed. Every page arrives in at most two redirect hops, with no meta-refresh or JavaScript redirect pages, and stays on your domain. Moving between www and the bare domain, or from http to https, is fine.
Partial. Some pages take a detour. Credit is the share of pages that don't.
Failed. The homepage itself takes a detour, or redirects in a loop.
When it can't count. Couldn't check when the homepage doesn't load.
Dead links say so
agent_404 · 9 points
When a missing page returns your homepage with 200, agents treat dead links as real pages and may quote the wrong content.
We request a random address that can't exist, such as /canagentsread-404-k3x9q2mf.
Passed. It answers 404 or 410 with links back into the site, or with a Markdown page.
Partial. It answers 404 with a bare page, or another error such as 500: half the points.
Failed. It answers 200 (a soft 404), or redirects to the homepage.
When it can't count. Couldn't check when the request fails, or robots.txt disallows the address.
Making sense
15 points · 11 checks
Speaks Markdown when asked
md_negotiation · 3 points
Agents that ask for Markdown get your words without the layout, which is cheaper and more accurate for them.
We request your homepage with Accept: text/markdown.
Passed. Markdown comes back, sent as text/markdown, starting with a heading, with a Vary: Accept header.
Partial. Markdown comes back, but without the text/markdown type, the heading or Vary: Accept: half the points.
Failed. HTML comes back.
Visible to search
indexable · 1.5 points
AI search builds on search indexes. A page that tells search engines to stay away is hard for agents to find.
Passed. No noindex (or none) in the homepage's robots, googlebot or bingbot meta tags, or in an X-Robots-Tag header.
Failed. The homepage tells search engines not to index it.
Has a map of its pages
sitemap · 1.5 points
A sitemap tells agents which pages exist, so they don't have to guess from links.
We look for Sitemap lines in robots.txt, then /sitemap.xml and /sitemap_index.xml, follow one level of sitemap index, and open up to five of the pages it lists.
Passed. Valid sitemap XML that lists pages on your site, and every listed page we open loads.
Partial. The sitemap isn't valid XML, lists no pages on your site, or some listed pages don't load.
Failed. No sitemap found.
Prices are findable
pricing_page · 1.5 points
"How much does it cost?" is one of the questions people ask agents most about a business.
Passed. Prices appear as text in the server HTML of a page we read, or in Offer schema.
Partial. There is a pricing page, but no prices in its HTML, so they probably load with JavaScript: half the points.
Failed. The site sells something, but we found no prices.
When it can't count. Doesn't apply when we find nothing for sale: no pricing or plans page, no shop or cart links, and no Product schema.
Describes itself in schema.org
json_ld · 1.5 points
Schema.org JSON-LD tells agents in plain data what the site is, instead of leaving them to guess.
Passed. The homepage's JSON-LD parses and includes an Organization (or one of its subtypes) or a WebSite with a name and a url.
Partial. There is JSON-LD, but a block doesn't parse or nothing in it names the site: half the points.
Failed. No JSON-LD on the homepage.
Page basics filled in
metadata · 1.5 points
Agents use these basics to name, link and preview your page correctly.
Passed. All six are filled in on the homepage: canonical URL, html lang, meta description, og:title, og:image and og:type.
Partial. Some are missing. Credit is the share filled in.
Failed. None of the six.
Each page names one address
canonical · 0.5 points
Agents cite the address a page calls canonical. When it names another page, their answer links to the wrong place.
We read the canonical link on every page we read that has one, and compare it with the page's own address after redirects.
Passed. Every canonical names its own page. Pointing to the www or bare version of the same address, or from http to https, is fine.
Partial. Some canonicals name another page. Credit is the share that name their own.
Failed. None does, such as every page pointing to the homepage.
When it can't count. Doesn't apply when no page we read has a canonical link.
Images are described
image_alt · 0.5 points
Agents can't see images. Alt text is how they learn what a photo, chart or product shot shows.
We look at every image on the pages we read, once each. Images marked decorative with an empty alt or hidden from assistive technology don't count, and neither do tracking pixels.
Passed. At least 90% of the other images have alt text that describes them, not a file name or "image".
Partial. 50% to 89%. Credit is the share described.
Failed. Under half.
When it can't count. Doesn't apply when the pages we read have no content images.
Pages fit in one read
token_budget · 1.5 points
Agents read within a limited context. A page too long to read in one go gets cut off or skipped.
We estimate what each page costs an agent that turns it into Markdown: its visible text plus every link address, at about four characters a token.
Passed. Every page we read is at most 20,000 tokens.
Partial. Some pages are over. Credit is the share under the limit.
Failed. The homepage itself is over.
Says who runs it
org_schema · 0.5 points
Agents want to know who is behind a site before they recommend it.
We take the first Organization schema on the pages we read, starting with the homepage.
Passed. It has at least five of name, url, logo, sameAs, contactPoint and address.
Partial. It has two to four of them. Credit is the share present.
Failed. It has one or none, or there is no Organization schema.
About, Contact, Privacy linked
trust_pages · 1.5 points
About, Contact and Privacy pages are how agents, and people, confirm a site is real and reachable.
Passed. The homepage links to all three, and each one loads.
Partial. One or two of them. Credit is the share found.
Failed. None of them.
Extras
10 points · 25 checks
Leaves an llms.txt
llms_txt · 0.4 points
llms.txt is a short guide to your site written for language models.
Passed. /llms.txt is a text file with something in it, not an HTML page.
Failed. It's missing, empty or an HTML page.
llms.txt reads like a contents page
llms_format · 0.4 points
An llms.txt laid out like a contents page lets an agent find the right page in one read.
Passed. It starts with a # heading, links to pages, and is at most 30,000 characters.
Partial. One of those three is missing: half the points.
Failed. Two or more are missing, or there is no llms.txt.
llms.txt links work
llms_links · 0.4 points
Broken links in llms.txt send agents to dead ends.
We open up to five links in llms.txt that point to your site.
Passed. Every link we open loads.
Partial. Some do. Credit is the share that load.
Failed. None load, there are no links, or there is no llms.txt.
When it can't count. Doesn't apply when every link points to another site.
Says when to use it
llms_guidance · 0.4 points
Telling agents when your site is the right answer helps them recommend it for the right questions.
Passed. llms.txt has a heading such as "When to use", "Best for" or "Use cases".
Partial. The guidance is there, but not under its own heading: half the points.
Failed. No guidance, or no llms.txt.
.md copies of pages
md_twins · 0.4 points
Markdown copies at predictable addresses let agents fetch clean text directly.
We request /index.md and the .md version of three pages we sample, such as /pricing.md.
Passed. /index.md and every .md copy we try return Markdown.
Partial. /index.md alone earns half the points; each working .md copy adds to the other half.
Failed. No Markdown copies.
Points to its Markdown
md_alternate · 0.4 points
A link in the page's head tells agents exactly where its Markdown version is.
Passed. The homepage has <link rel="alternate" type="text/markdown"> pointing to an address that returns Markdown.
Partial. The link is there, but the address doesn't return Markdown or is on another site: half the points.
Failed. No such link.
Gives AI agents Markdown without being asked
bot_markdown · 0.4 points
Most agents never ask for Markdown, so serving it to known AI user agents gives them clean text anyway.
Passed. Every AI agent that gets the homepage without asking for Markdown receives it anyway.
Partial. Some do. Credit is the share that do.
Failed. They all get HTML.
When it can't count. Couldn't check when no agent got the homepage.
Points to agent files in HTTP headers
link_headers · 0.4 points
An HTTP Link header points agents to your agent files before they parse any HTML.
Passed. The homepage response has a Link header that names llms.txt, a text/markdown version or an API catalog.
Partial. There is a Link header, but it names none of those: half the points.
Failed. No Link header.
Sitemap says what changed
sitemap_fresh · 0.4 points
Dates in the sitemap let agents re-read only what changed.
Passed. At least 90% of sitemap entries have a <lastmod> date.
Partial. Some do. Credit is the share that do.
Failed. None do, or there is no sitemap.
Links to its other profiles
same_as · 0.4 points
Links to your official profiles help agents confirm who you are and tell you apart from look-alikes.
Passed. Your schema lists at least two profile URLs in sameAs.
Partial. One profile: half the points.
Failed. None.
Clear rules for AI bots
robots_ai_policy · 0.4 points
Naming AI agents in robots.txt shows you decided to let them in, rather than leaving it to a catch-all rule.
Blocking training crawlers such as GPTBot, ClaudeBot or CCBot never costs points. This check is about the agents that fetch pages to answer people.
Passed. robots.txt names OAI-SearchBot, Claude-SearchBot, PerplexityBot, ChatGPT-User, Claude-User and Perplexity-User, and allows them.
Partial. They are allowed, but only through User-agent: *, or there is no robots.txt: half the points.
Failed. At least one of them is blocked.
When it can't count. Couldn't check when robots.txt can't be read: an error, 401, 403, 429 or a server error.
Clear page outline
doc_structure · 0.4 points
Landmarks and a clean heading outline let agents find the main content and skip menus and footers.
Passed. The homepage has main, header, nav and footer landmarks (or their ARIA roles), exactly one H1, and no skipped heading levels.
Partial. Each of those six parts earns a sixth of the points.
Failed. None of them.
Real buttons and links
native_controls · 0.4 points
Agents that act on pages look for real links and buttons. Divs made clickable with script are invisible to them.
We count real controls (links with an href, buttons, inputs, selects, text areas and summaries) against imitations: other elements with click handlers, or with button, link, checkbox, tab or menu item roles.
Passed. At least 95% of controls in the server HTML are real.
Partial. 80% to 94%: half the points.
Failed. Under 80%.
When it can't count. Doesn't apply when the pages we read have no controls.
Buttons say what they do
control_names · 0.4 points
An icon-only button without a name tells an agent nothing about what it does.
Passed. At least 95% of buttons and links have a name: their text, an aria-label, aria-labelledby, a title, or alt text on an image inside.
Partial. 80% to 94%: half the points.
Failed. Under 80%.
When it can't count. Doesn't apply when the pages we read have no buttons or links.
Every field has a label
form_labels · 0.4 points
An agent filling in a form needs labels to know what goes where. Placeholders don't count.
Passed. Every field has a label: a <label>, an aria-label, aria-labelledby or a title.
Partial. At least half do. Credit is the share labelled.
Failed. Fewer than half do.
When it can't count. Doesn't apply when the pages we read have no form fields.
No hidden instructions
no_hidden_prompts · 0.4 points
Text that tries to instruct AI agents from hidden places is a prompt-injection pattern. Agents that spot it may distrust the whole site.
We read hidden and off-screen elements, aria-label, title and alt attributes, and HTML comments, looking for phrases only an AI would be told, such as "ignore previous instructions" or "note to AI agents". Ordinary hidden text, like collapsed menus, is fine.
Passed. No instructions for AI agents hidden on the pages we read.
Failed. Instruction-like text is hidden somewhere on a page we read.
Says when pages were updated
dates · 0.4 points
Agents prefer recent sources and often say how old something is. A page without a date can't tell them.
We look on each page we read for a publish or update date: dateModified or datePublished in JSON-LD, an article:modified_time or article:published_time meta tag, or a <time datetime> element.
Passed. At least half the pages we read give a date.
Partial. Some do, but fewer than half: half the points.
Failed. None does.
Answers quickly
speed · 0.4 points
Agents fetch pages while someone waits for an answer, and many give up after a few seconds.
We time the homepage as a normal browser and as each AI agent, from our server, including redirects and the whole response, and take the middle time.
Passed. The middle time is under 2 seconds.
Partial. 2 to 5 seconds: half the points.
Failed. Over 5 seconds.
When it can't count. Couldn't check when none of those fetches loaded.
States its AI preferences
content_signals · 0.4 points
A Content-Signal line in robots.txt tells AI companies how you want your content used: for search, for live answers, for training. We only check that you say it, never what you choose.
We read robots.txt for Content-Signal lines with search, ai-input or ai-train set to yes or no.
Passed. robots.txt has a Content-Signal line that sets at least one of them.
Partial. There is a Content-Signal line, but it doesn't set any of the three to yes or no: half the points.
Failed. No Content-Signal line, or no robots.txt.
When it can't count. Couldn't check when robots.txt can't be read: an error, 401, 403, 429 or a server error.
Can be searched by URL
site_search · 0.4 points
When an agent knows how your search works, it can look things up on your site directly instead of guessing addresses.
We look on the homepage for a search box, an OpenSearch description link and a schema.org SearchAction.
Passed. The homepage links an OpenSearch description, or its schema has a SearchAction with a URL template.
Failed. The site has a search box, but nothing tells agents how to use it.
When it can't count. Doesn't apply when the homepage has no search box.
Has an update feed
feed · 0.4 points
A feed lets agents, and the tools around them, notice new pages and changes without crawling the whole site.
We look for an RSS, Atom or JSON Feed link in the homepage's head, fetch it when it is on your site, and check that it has entries.
Passed. The homepage links a feed that loads and has at least one entry, or one on another host.
Partial. A feed is linked, but it doesn't load, isn't a feed, or is empty: half the points.
Failed. No feed linked from the homepage.
Lists its APIs
api_catalog · 0.4 points
An API catalog at a fixed address tells agents which APIs you offer and where they are described, so they can call them instead of reading pages.
We request /.well-known/api-catalog, the address RFC 9727 sets, and read it as a linkset.
Passed. It is a linkset with at least one API in it.
Partial. It is JSON, but not a linkset that lists an API: half the points.
Failed. There is a file at that address, but it isn't JSON.
When it can't count. Doesn't apply when there is no API catalog, so sites without an API aren't marked down.
Publishes an agent card
agent_card · 0.4 points
If your site offers an AI agent, an A2A agent card tells other agents what it can do and how to reach it.
We request /.well-known/agent-card.json, the A2A 1.0 address, and the older /.well-known/agent.json when the first isn't there.
Passed. A card at /.well-known/agent-card.json with a name, a description and at least one skill.
Partial. A card only at the older agent.json address, or one missing its name, description or skills: half the points.
Failed. There is a file there, but it isn't a JSON agent card.
When it can't count. Doesn't apply when neither address has a card, so sites without an agent aren't marked down.
Products are described
product_data · 0.4 points
Shopping agents compare products by their data: price, currency, availability and an identifier. Without it they guess from the page, or skip it.
We look for Product schema on the pages we read and check each product for a name, an image, a SKU, GTIN or MPN, and an offer with price, currency and availability.
Passed. Every product we found has all of those.
Partial. Some products are missing parts. Credit is the share that are complete.
Failed. The site sells products, but no page we read has Product schema.
When it can't count. Doesn't apply when we find no sign the site sells products: no shop or cart links and no Product or Offer schema.
Shipping and returns are findable
shop_policies · 0.4 points
Before recommending a shop, agents look for what delivery costs and whether returns are possible.
We look for links to shipping and returns pages on the pages we read, and for shipping and return details in the products' offers.
Passed. Both shipping and returns are covered.
Partial. Only one of them: half the points.
Failed. Neither.
When it can't count. Doesn't apply when the site doesn't sell on its own pages: no cart or checkout links, and its product offers point to other shops, as on a price comparison site.