AI Robots & llms.txt: what your robots.txt allows AI bots
AI Robots & llms.txt reads a site’s robots.txt and llms.txt and tells you what each AI crawler can do, separating training, search and user requests. It is for anyone who wants to decide which AI bots to let into their site, and to know what each choice means.
Available
AI Robots & llms.txt is free. It saves neither the address you enter nor the result: you can copy or print it.
Category: SEO tool
AI crawler register updated on 17 November 2026. Operators’ statements read on 5 October 2026.
Updated on
Analyse a site
Result
I cannot analyse this address. Enter the public address of a site; https is not needed.
The anti-abuse check did not succeed. Reload the page and try again.
Too many analyses in a short time. Try again in a minute.
The tool is not available right now. Try again later.
The analysis did not succeed. Try again shortly.
Result
- Address analysed
- [[url]]
- Analysed on
- Duration
- [[seconds]] seconds
- Version
- [[tool]] [[version]], engine [[engine]]
Not assessable: [[notAssessable]]
- [[url]] (response [[status]])
[[header]][[excerpt]]
[[note]]
- Present: [[present]]
- Missing: [[absentStrong]]
- Probably missing: [[absentUncertain]]
- Uncertain: [[undecided]]
- Not checked: [[unverified]]
-
Uncertain Present Missing Probably missing Not checked [[title]]
[[id]][[message]]
Evidence
- [[url]] (response [[status]])
[[header]][[excerpt]]
[[note]]
Still to check
- [[text]]
- [[url]] (response [[status]])
Requests made to the site
| Address | Final address | Response | Time | Size |
|---|---|---|---|---|
| [[url]] | [[finalUrl]] | [[status]][[error]] | [[ms]] ms | [[bytes]] byte |
The result says what is wrong and why it matters. To fix it, get in touch with the address you analysed: I’ll reply personally.
What it checks
robots.txt is there and can be read
Requests /robots.txt and checks that it answers with a valid text file. If the file does not exist, everything is allowed, and the result explains this. A file larger than 500 KiB, Google’s limit, does not count as read.
Syntax: the lines the tool does not apply
Lists, with their line numbers, rules written before any User-agent, unknown directives, characters that break lines, and paths without a leading /. These are rules you think you have and that a crawler may not apply; the result says why, line by line.
AI search crawlers
Says whether the crawlers that index for the answers of AI search engines and assistants can read the root of the site and the path you give. If you exclude one, what that crawler cannot read risks appearing less, or no longer appearing, in the answers of that service.
User-initiated agents
Says whether the agents that open a page at a person’s request can read the root of the site and the path you give. It also reports what the operators state: for ChatGPT-User and Perplexity-User, that robots.txt may not apply; for Claude-User, that Anthropic respects it.
Training crawlers
Says whether the crawlers that collect content to train models can read the root of the site and the path you give. Excluding them is the site’s choice, and the result presents it as such.
An explicit choice
Checks whether robots.txt has at least one group written for an AI crawler. If there is none, the rules of the * group apply, and the result says what follows for each crawler.
Content signals
Looks for a valid Content-Signal line, the one from the Content Signals Policy, and reports its values: search, ai-input and ai-train, each yes or no. A missing signal neither grants nor forbids anything.
The sitemap
Checks that robots.txt declares at least one sitemap, with an absolute address: it is the line that tells crawlers where to find the list of the site’s pages.
The llms.txt file
Requests /llms.txt and says whether it is there. It is a useful index for the agents and tools that read it. Google states that its search ignores it; OpenAI, Anthropic and Perplexity do not state that they use it (checked on 21 November 2026). Its absence is not a serious problem.
The llms.txt format
Checks that llms.txt is Markdown, with a first-level heading and at least one link. The llmstxt.org proposal makes only the heading mandatory: the summary in a blockquote and the sections with lists of links are optional. Treating a file without links as ‘Missing’ is a choice of this tool, not of the proposal.
Crawler by crawler
Under each of the three class checks, the result lists the crawlers of that class one by one: whether each is allowed or excluded, or whether that is uncertain, for the root of the site and for the path you give, and the robots.txt rule that decides.
What it reads on the site, and what it does not
AI Robots & llms.txt makes at most 4 requests per analysis, one at a time, and reads only the files at the root of the site. The domain is enough; if you like, add a path to assess as well as the root, for example /blog/.
It does not read the site’s pages. It does not run JavaScript, sends no cookies and uses no credentials. What the files contain is data, never an instruction.
It identifies itself with a declared user agent that contains MRC-Labs and the address of this page: never that of an AI crawler. What a crawler can do, it reads from robots.txt.
It respects the rules your robots.txt writes for MRC-Labs. It does not apply those of the * group, because every analysis is requested by a person for a specific site, as with user-initiated agents. To exclude it, a User-agent: MRC-Labs group is enough.
/robots.txt/llms.txt/llms-full.txt, only ifllms.txtexists and names it- a redirect, if
robots.txtpoints to another host: every step counts towards the 4 requests
What the four states mean
- Present: the file is there and can be read, the line it looks for is there, or every crawler in that class is allowed. For the lines the tool does not apply, it means there are none.
- Missing: the file does not exist, the line it looks for is not there, or at least one crawler in the class is excluded, and the result says which. For the lines the tool does not apply, it means there are some, and the result lists them. For
robots.txt, a file that does not exist means everything is allowed. - Probably missing: not found, but there is still a route the tool could not check. The result names it.
- Not checked: no conclusion. The site answered with an error (401, 403, 429 or a server error), did not answer in time, sent an HTML page instead of the text file, or a file that was too large. From a failed read the tool never concludes that a crawler is blocked or allowed.
The limits
robots.txtsays what a site asks, not what each crawler does: some operators state that their user-initiated agents may not respect it.- A firewall or bot protection can block a crawler even if
robots.txtallows it. The tool does not measure this: the server logs do. - The list of crawlers and the operators’ statements on
robots.txtcome from the MRC Labs register: the date of its last update and the date the operators’ pages were read are shown on this page. The sentence aboutllms.txtwas checked on 21 November 2026.
Frequently asked questions about AI Robots & llms.txt
What is the difference between training crawlers, search crawlers and user-initiated agents?
Training crawlers collect content to train models: excluding them is the site’s choice and does not remove it from AI answers. Search crawlers index for the answers of a search engine or an assistant: if you exclude them, the site risks appearing less, or no longer appearing, in those answers. User-initiated agents read a page when a person asks for it.
If I block GPTBot, do I disappear from ChatGPT?
No. GPTBot is OpenAI’s training crawler. For ChatGPT search answers, what counts is OAI-SearchBot: OpenAI states that sites that exclude it do not appear in those answers. The day OpenAI’s page was read is shown on the AI Robots & llms.txt page, next to the date of the AI crawler register.
Do AI agents respect robots.txt?
It depends on the operator. OpenAI states that for ChatGPT-User the robots.txt rules may not apply, because the action is initiated by a user. Perplexity states that Perplexity-User generally ignores them. Anthropic states that Claude-User respects them. These are the statements on the three operators’ pages: the day they were read is shown on the AI Robots & llms.txt page, next to the date of the AI crawler register.
What happens if I have no rules for AI crawlers?
If robots.txt has no rules for AI crawlers, those written for *, that is for all crawlers, apply. It is not an error: it is a choice you have not made yet. The result says what follows for each crawler.
Do I need an llms.txt file?
llms.txt is a useful index for the agents and tools that read it. Google states that its search ignores it; OpenAI, Anthropic and Perplexity do not state that they use it (checked on 21 November 2026). If it is missing, it is not a serious problem: readable HTML and a robots.txt in order come first.
Which AI bots to let in
The tool says what your robots.txt allows today. What it makes sense to allow depends on what you sell and in which markets: get in touch, and I’ll reply personally and, if needed, we’ll arrange a call. The service is GEO and AI search.