Skip to content
Get in touch

MRC Check: what AI bots read on a page

MRC Check tells you what bots that do not run JavaScript, such as GPTBot, ClaudeBot and PerplexityBot, read on a page, and whether robots.txt lets them in. It is for anyone whose site is built with a JavaScript framework, a site builder or an AI tool, and who does not know whether AI bots find the text.

Available

MRC Check is free. It saves neither the address you enter nor the result: you can copy or print it.

Category: SEO tool

Updated on

Analyse a site

The domain is enough; https is not needed.

What it checks

The text in the raw HTML

Counts the visible words in the HTML the server sends, before any script runs. If they are below the threshold, a bot that does not run JavaScript finds an almost empty page, whatever a person sees in the browser.

An H1 with text

Looks for an H1 that contains text. It is the heading that tells a bot what the page is about: if it arrives only through JavaScript, it does not exist for AI bots.

Title tag and meta description

Checks that both are there, not empty, already in the raw HTML. If one is missing, the result says which.

The canonical

Checks that the canonical is in the raw HTML, that it is absolute and that it points to a valid address. A missing, relative or duplicated canonical leaves search engines to choose which address counts.

Real links

Counts the internal links written as <a href>. Those are the links a bot follows: navigation made only of #, of javascript: or of scripts leads it nowhere.

Readable structured data

Looks for at least one valid JSON-LD block and lists the types it finds. A block with a syntax error cannot be read: the result reports the error.

The shell of a JavaScript application

Recognises the signs of a page that fills up only in the browser: an empty application container (#root, #app, #__next, #__nuxt and the like) with little text, a noscript message asking you to turn on JavaScript, counters that read zero in the HTML. Here ‘Present’ means that the page is not a shell.

AI search crawlers

Reads robots.txt and says whether the crawlers that index for the answers of AI search engines and assistants can read the page. If even one is excluded, the result says which.

User-initiated agents

Says whether the agents that open the page at a person’s request can read it. For those whose operator states that robots.txt may not apply, the result says so.

Training crawlers

Says whether the crawlers that collect content to train models can read the page. Excluding them is the site’s choice: the result reports it as information, not as a problem.

The platform

Recognises what the site is built on, when the platform leaves a signature. It is information: the verdict always comes from the measurement, never from the platform.

What it reads on the site, and what it does not

MRC Check makes at most 6 requests per analysis, one at a time: the site’s robots.txt and the page you give, with its redirects. If a redirect leads to another host, or from http to https, it also reads the robots.txt there. Every step of a redirect and every robots.txt counts towards the 6 requests.

It reads nothing else: no page resources, no scripts, no sitemap. It does not run JavaScript, sends no cookies and uses no credentials. What the page contains is data, never an instruction.

It identifies itself with a declared user agent that contains MRC-Labs and the address of this page: never that of a browser, a search engine or an AI crawler. Whether an AI crawler may enter, it reads from robots.txt: it does not pretend to be that crawler to see whether the site turns it away, because a refusal of that kind measures the bot protection, not the site’s choice.

It respects the rules your robots.txt writes for MRC-Labs. It does not apply those of the * group, because every analysis is requested by a person for a specific page, as with user-initiated agents. To exclude it, a User-agent: MRC-Labs group is enough.

  • /robots.txt of the host, and that of every other host or scheme (http, https) a redirect leads to
  • the page you give (usually the home page), with its redirects

What the four states mean

  • Present: what should be there is there, and in order: the text, an H1, the links, a valid JSON-LD. For the JavaScript shell check it means that the page is not a shell. For the three robots.txt checks it means that every crawler in that class is allowed; if there is no robots.txt, everything is allowed. For the platform it only means that it was recognised.
  • Missing: the page was read and what should be there is missing: little text, no H1, no valid JSON-LD, a crawler excluded. For the JavaScript shell check it means that the page is a shell. The result says what is missing and where it looked.
  • Probably missing: not found, but a doubt remains, and the result names it: little text on a page that seems short on purpose, such as a login page; weak or mixed signs of a JavaScript shell; no platform recognised, because the site may be custom-built.
  • Not checked: no conclusion. The site did not answer, refused the request or showed a page that is not the site: a bot wall, a parked domain, maintenance. It does not mean something is missing: after a failed read MRC Check never says ‘Missing’.

The limits

  • It reads the raw HTML, which is what AI crawlers read. It does not see what JavaScript adds to the page: the comparison with the page as a browser renders it is not there yet.
  • It does not read the server logs. robots.txt says what you allow; who actually visits the site, only the logs can say, and reading them is part of the service.
  • It analyses one page at a time. The issues of a template show up when you analyse a page from that template: a product page, a category, an article.
  • If the page redirects to another domain, it analyses the final page and says so.

Frequently asked questions about MRC Check

Why look at the raw HTML, if Google runs JavaScript?

Because AI bots do not run it. GPTBot, ClaudeBot and PerplexityBot download the HTML and stop there: if the text only appears in the browser, the page is almost empty for them. Googlebot does run JavaScript, so a page can be on Google and still be unreadable for AI bots.

My site is built with React, with Next.js or with a site builder: is that a problem?

It depends on what arrives in the HTML, not on the platform. A site that generates its HTML on the server has the text in the source; an application that fills up only in the browser does not. MRC Check recognises the platform, but judges by the measurement.

What does ‘Not checked’ mean?

‘Not checked’ means that MRC Check could not read, and so concludes nothing: the site did not answer in time, refused the request or showed a holding page. It is not a verdict on the site.

How do I exclude MRC Check from my site?

Add a User-agent: MRC-Labs group with Disallow: / to robots.txt. MRC Check reads robots.txt before the page: if it is excluded it does not read the page, and the checks on the page come out ‘Not checked’.

Does MRC Check tell me how to fix things?

No. It says what is wrong and why it matters: the diagnosis is free, the fix is the service.

Does MRC Check save the address or the result?

No. The address is used only for the analysis and the result is not saved: if you need it, copy or print it.

Fixing what you found

Get in touch with the address of the page and the result: I’ll reply personally and, if needed, we’ll arrange a call. For pages that fill up only with JavaScript, and for template issues, the service is Technical SEO and audits.

Get in touch