Skip to the tool

Robots.txt tester

Type a page address. We read your robots.txt and tell you, crawler by crawler, whether each may read that page and which rule decided it.

Free, no sign up. We read the page from our server and keep nothing.

Reading the page…

What it checks

  • Your robots.txt, read as Google reads it
  • Whether Google and Bing may read the page
  • Whether AI crawlers may read it: GPTBot, ClaudeBot, PerplexityBot and more
  • The exact rule behind each answer
  • Whether it points to your sitemap

How it works

We read robots.txt the way Google does: the most specific group for each crawler wins, then the longest matching rule, and on a tie Allow beats Disallow. A star matches anything and a dollar sign ends a pattern.

Questions

What is robots.txt?

A small file at the root of your site that tells crawlers which pages they may read. It is a request, not a lock: well behaved crawlers follow it.

Should I block AI crawlers?

It is your choice. Blocking them keeps your words out of AI training, but also out of AI answers that could send you buyers. Search crawlers like OAI-SearchBot cite and link to you.

Does blocking a page hide it from Google?

Not always. Google can still list an address it cannot read if other sites link to it. To keep a page out of search, use a noindex tag and let Google read it.

More free tools

Found something to fix? OOVERT fixes it.

We are the brand identity and web studio behind these tools. Brand, website and SEO, done properly in three weeks.

Book a call