MRC Check: what AI bots read on a page
MRC Check tells you what bots that do not run JavaScript, such as GPTBot, ClaudeBot and PerplexityBot, read on a page, and whether robots.txt lets them in. It is for anyone whose site is built with a JavaScript framework, a site builder or an AI tool, and who does not know whether AI bots find the text.
Available
MRC Check is free. It saves neither the address you enter nor the result: you can copy or print it.
Category: SEO tool
Updated on
Analyse a site
Result
I cannot analyse this address. Enter the public address of a site; https is not needed.
The anti-abuse check did not succeed. Reload the page and try again.
Too many analyses in a short time. Try again in a minute.
The tool is not available right now. Try again later.
The analysis did not succeed. Try again shortly.
Result
- Address analysed
- [[url]]
- Analysed on
- Duration
- [[seconds]] seconds
- Version
- [[tool]] [[version]], engine [[engine]]
Not assessable: [[notAssessable]]
- [[url]] (response [[status]])
[[header]][[excerpt]]
[[note]]
- Present: [[present]]
- Missing: [[absentStrong]]
- Probably missing: [[absentUncertain]]
- Uncertain: [[undecided]]
- Not checked: [[unverified]]
-
Uncertain Present Missing Probably missing Not checked [[title]]
[[id]][[message]]
Evidence
- [[url]] (response [[status]])
[[header]][[excerpt]]
[[note]]
Still to check
- [[text]]
- [[url]] (response [[status]])
Requests made to the site
| Address | Final address | Response | Time | Size |
|---|---|---|---|---|
| [[url]] | [[finalUrl]] | [[status]][[error]] | [[ms]] ms | [[bytes]] byte |
The result says what is wrong and why it matters. To fix it, get in touch with the address you analysed: I’ll reply personally.
What it checks
The text in the raw HTML
Counts the visible words in the HTML the server sends, before any script runs. If they are below the threshold, a bot that does not run JavaScript finds an almost empty page, whatever a person sees in the browser.
An H1 with text
Looks for an H1 that contains text. It is the heading that tells a bot what the page is about: if it arrives only through JavaScript, it does not exist for AI bots.
Title tag and meta description
Checks that both are there, not empty, already in the raw HTML. If one is missing, the result says which.
The canonical
Checks that the canonical is in the raw HTML, that it is absolute and that it points to a valid address. A missing, relative or duplicated canonical leaves search engines to choose which address counts.
Real links
Counts the internal links written as <a href>. Those are the links a bot follows: navigation made only of #, of javascript: or of scripts leads it nowhere.
Readable structured data
Looks for at least one valid JSON-LD block and lists the types it finds. A block with a syntax error cannot be read: the result reports the error.
The shell of a JavaScript application
Recognises the signs of a page that fills up only in the browser: an empty application container (#root, #app, #__next, #__nuxt and the like) with little text, a noscript message asking you to turn on JavaScript, counters that read zero in the HTML. Here ‘Present’ means that the page is not a shell.
AI search crawlers
Reads robots.txt and says whether the crawlers that index for the answers of AI search engines and assistants can read the page. If even one is excluded, the result says which.
User-initiated agents
Says whether the agents that open the page at a person’s request can read it. For those whose operator states that robots.txt may not apply, the result says so.
Training crawlers
Says whether the crawlers that collect content to train models can read the page. Excluding them is the site’s choice: the result reports it as information, not as a problem.
The platform
Recognises what the site is built on, when the platform leaves a signature. It is information: the verdict always comes from the measurement, never from the platform.
What it reads on the site, and what it does not
MRC Check makes at most 6 requests per analysis, one at a time: the site’s robots.txt and the page you give, with its redirects. If a redirect leads to another host, or from http to https, it also reads the robots.txt there. Every step of a redirect and every robots.txt counts towards the 6 requests.
It reads nothing else: no page resources, no scripts, no sitemap. It does not run JavaScript, sends no cookies and uses no credentials. What the page contains is data, never an instruction.
It identifies itself with a declared user agent that contains MRC-Labs and the address of this page: never that of a browser, a search engine or an AI crawler. Whether an AI crawler may enter, it reads from robots.txt: it does not pretend to be that crawler to see whether the site turns it away, because a refusal of that kind measures the bot protection, not the site’s choice.
It respects the rules your robots.txt writes for MRC-Labs. It does not apply those of the * group, because every analysis is requested by a person for a specific page, as with user-initiated agents. To exclude it, a User-agent: MRC-Labs group is enough.
/robots.txtof the host, and that of every other host or scheme (http,https) a redirect leads to- the page you give (usually the home page), with its redirects
What the four states mean
- Present: what should be there is there, and in order: the text, an H1, the links, a valid JSON-LD. For the JavaScript shell check it means that the page is not a shell. For the three
robots.txtchecks it means that every crawler in that class is allowed; if there is norobots.txt, everything is allowed. For the platform it only means that it was recognised. - Missing: the page was read and what should be there is missing: little text, no H1, no valid JSON-LD, a crawler excluded. For the JavaScript shell check it means that the page is a shell. The result says what is missing and where it looked.
- Probably missing: not found, but a doubt remains, and the result names it: little text on a page that seems short on purpose, such as a login page; weak or mixed signs of a JavaScript shell; no platform recognised, because the site may be custom-built.
- Not checked: no conclusion. The site did not answer, refused the request or showed a page that is not the site: a bot wall, a parked domain, maintenance. It does not mean something is missing: after a failed read MRC Check never says ‘Missing’.
The limits
- It reads the raw HTML, which is what AI crawlers read. It does not see what JavaScript adds to the page: the comparison with the page as a browser renders it is not there yet.
- It does not read the server logs.
robots.txtsays what you allow; who actually visits the site, only the logs can say, and reading them is part of the service. - It analyses one page at a time. The issues of a template show up when you analyse a page from that template: a product page, a category, an article.
- If the page redirects to another domain, it analyses the final page and says so.
Frequently asked questions about MRC Check
Why look at the raw HTML, if Google runs JavaScript?
Because AI bots do not run it. GPTBot, ClaudeBot and PerplexityBot download the HTML and stop there: if the text only appears in the browser, the page is almost empty for them. Googlebot does run JavaScript, so a page can be on Google and still be unreadable for AI bots.
My site is built with React, with Next.js or with a site builder: is that a problem?
It depends on what arrives in the HTML, not on the platform. A site that generates its HTML on the server has the text in the source; an application that fills up only in the browser does not. MRC Check recognises the platform, but judges by the measurement.
What does ‘Not checked’ mean?
‘Not checked’ means that MRC Check could not read, and so concludes nothing: the site did not answer in time, refused the request or showed a holding page. It is not a verdict on the site.
How do I exclude MRC Check from my site?
Add a User-agent: MRC-Labs group with Disallow: / to robots.txt. MRC Check reads robots.txt before the page: if it is excluded it does not read the page, and the checks on the page come out ‘Not checked’.
Does MRC Check tell me how to fix things?
No. It says what is wrong and why it matters: the diagnosis is free, the fix is the service.
Does MRC Check save the address or the result?
No. The address is used only for the analysis and the result is not saved: if you need it, copy or print it.
Fixing what you found
Get in touch with the address of the page and the result: I’ll reply personally and, if needed, we’ll arrange a call. For pages that fill up only with JavaScript, and for template issues, the service is Technical SEO and audits.