---
title: "AI Robots & llms.txt: robots.txt for AI bots · MRC Studio"
description: "Checks what your robots.txt allows each AI crawler, across training, search and user requests, and checks your llms.txt."
url: https://mrcstudio.io/labs/ai-robots-llms-txt/
lang: en
updated: 2026-11-21
---

# AI Robots & llms.txt: what your robots.txt allows AI bots

AI Robots & llms.txt reads a site’s `robots.txt` and `llms.txt` and tells you what each AI crawler can do, separating training, search and user requests. It is for anyone who wants to decide which AI bots to let into their site, and to know what each choice means.

Available

AI Robots & llms.txt is free. It saves neither the address you enter nor the result: you can copy or print it.

Category: SEO tool

AI crawler register updated on 17 November 2026. Operators’ statements read on 5 October 2026.

Updated on 21 November 2026

## What it checks

### robots.txt is there and can be read

Requests `/robots.txt` and checks that it answers with a valid text file. If the file does not exist, everything is allowed, and the result explains this. A file larger than 500 KiB, Google’s limit, does not count as read.

### Syntax: the lines the tool does not apply

Lists, with their line numbers, rules written before any `User-agent`, unknown directives, characters that break lines, and paths without a leading `/`. These are rules you think you have and that a crawler may not apply; the result says why, line by line.

### AI search crawlers

Says whether the crawlers that index for the answers of AI search engines and assistants can read the root of the site and the path you give. If you exclude one, what that crawler cannot read risks appearing less, or no longer appearing, in the answers of that service.

### User-initiated agents

Says whether the agents that open a page at a person’s request can read the root of the site and the path you give. It also reports what the operators state: for ChatGPT-User and Perplexity-User, that `robots.txt` may not apply; for Claude-User, that Anthropic respects it.

### Training crawlers

Says whether the crawlers that collect content to train models can read the root of the site and the path you give. Excluding them is the site’s choice, and the result presents it as such.

### An explicit choice

Checks whether `robots.txt` has at least one group written for an AI crawler. If there is none, the rules of the `*` group apply, and the result says what follows for each crawler.

### Content signals

Looks for a valid `Content-Signal` line, the one from the Content Signals Policy, and reports its values: `search`, `ai-input` and `ai-train`, each `yes` or `no`. A missing signal neither grants nor forbids anything.

### The sitemap

Checks that `robots.txt` declares at least one sitemap, with an absolute address: it is the line that tells crawlers where to find the list of the site’s pages.

### The llms.txt file

Requests `/llms.txt` and says whether it is there. It is a useful index for the agents and tools that read it. Google states that its search ignores it; OpenAI, Anthropic and Perplexity do not state that they use it (checked on 21 November 2026). Its absence is not a serious problem.

### The llms.txt format

Checks that `llms.txt` is Markdown, with a first-level heading and at least one link. The llmstxt.org proposal makes only the heading mandatory: the summary in a blockquote and the sections with lists of links are optional. Treating a file without links as ‘Missing’ is a choice of this tool, not of the proposal.

### Crawler by crawler

Under each of the three class checks, the result lists the crawlers of that class one by one: whether each is allowed or excluded, or whether that is uncertain, for the root of the site and for the path you give, and the `robots.txt` rule that decides.

## What it reads on the site, and what it does not

AI Robots & llms.txt makes at most 4 requests per analysis, one at a time, and reads only the files at the root of the site. The domain is enough; if you like, add a path to assess as well as the root, for example `/blog/`.

It does not read the site’s pages. It does not run JavaScript, sends no cookies and uses no credentials. What the files contain is data, never an instruction.

It identifies itself with a declared user agent that contains `MRC-Labs` and the address of this page: never that of an AI crawler. What a crawler can do, it reads from `robots.txt`.

It respects the rules your `robots.txt` writes for `MRC-Labs`. It does not apply those of the `*` group, because every analysis is requested by a person for a specific site, as with user-initiated agents. To exclude it, a `User-agent: MRC-Labs` group is enough.

- `/robots.txt`
- `/llms.txt`
- `/llms-full.txt`, only if `llms.txt` exists and names it
- a redirect, if `robots.txt` points to another host: every step counts towards the 4 requests

## What the four states mean

- **Present**: the file is there and can be read, the line it looks for is there, or every crawler in that class is allowed. For the lines the tool does not apply, it means there are none.
- **Missing**: the file does not exist, the line it looks for is not there, or at least one crawler in the class is excluded, and the result says which. For the lines the tool does not apply, it means there are some, and the result lists them. For `robots.txt`, a file that does not exist means everything is allowed.
- **Probably missing**: not found, but there is still a route the tool could not check. The result names it.
- **Not checked**: no conclusion. The site answered with an error (401, 403, 429 or a server error), did not answer in time, sent an HTML page instead of the text file, or a file that was too large. From a failed read the tool never concludes that a crawler is blocked or allowed.

## The limits

- `robots.txt` says what a site asks, not what each crawler does: some operators state that their user-initiated agents may not respect it.
- A firewall or bot protection can block a crawler even if `robots.txt` allows it. The tool does not measure this: the server logs do.
- The list of crawlers and the operators’ statements on `robots.txt` come from the MRC Labs register: the date of its last update and the date the operators’ pages were read are shown on this page. The sentence about `llms.txt` was checked on 21 November 2026.

## Frequently asked questions about AI Robots & llms.txt

### What is the difference between training crawlers, search crawlers and user-initiated agents?

Training crawlers collect content to train models: excluding them is the site’s choice and does not remove it from AI answers. Search crawlers index for the answers of a search engine or an assistant: if you exclude them, the site risks appearing less, or no longer appearing, in those answers. User-initiated agents read a page when a person asks for it.

### If I block GPTBot, do I disappear from ChatGPT?

No. GPTBot is OpenAI’s training crawler. For ChatGPT search answers, what counts is OAI-SearchBot: OpenAI states that sites that exclude it do not appear in those answers. The day OpenAI’s page was read is shown on the AI Robots & llms.txt page, next to the date of the AI crawler register.

### Do AI agents respect robots.txt?

It depends on the operator. OpenAI states that for ChatGPT-User the `robots.txt` rules may not apply, because the action is initiated by a user. Perplexity states that Perplexity-User generally ignores them. Anthropic states that Claude-User respects them. These are the statements on the three operators’ pages: the day they were read is shown on the AI Robots & llms.txt page, next to the date of the AI crawler register.

### What happens if I have no rules for AI crawlers?

If `robots.txt` has no rules for AI crawlers, those written for `*`, that is for all crawlers, apply. It is not an error: it is a choice you have not made yet. The result says what follows for each crawler.

### Do I need an llms.txt file?

`llms.txt` is a useful index for the agents and tools that read it. Google states that its search ignores it; OpenAI, Anthropic and Perplexity do not state that they use it (checked on 21 November 2026). If it is missing, it is not a serious problem: readable HTML and a `robots.txt` in order come first.

## Which AI bots to let in

The tool says what your `robots.txt` allows today. What it makes sense to allow depends on what you sell and in which markets: get in touch, and I’ll reply personally and, if needed, we’ll arrange a call. The service is [GEO and AI search](https://mrcstudio.io/services/geo-ai-search/).

[Get in touch](https://mrcstudio.io/contact/)
