Robots.txt tester
Type a page address. We read your robots.txt and tell you, crawler by crawler, whether each may read that page and which rule decided it.
What it checks
- Your robots.txt, read as Google reads it
- Whether Google and Bing may read the page
- Whether AI crawlers may read it: GPTBot, ClaudeBot, PerplexityBot and more
- The exact rule behind each answer
- Whether it points to your sitemap
How it works
We read robots.txt the way Google does: the most specific group for each crawler wins, then the longest matching rule, and on a tie Allow beats Disallow. A star matches anything and a dollar sign ends a pattern.
Questions
What is robots.txt?
A small file at the root of your site that tells crawlers which pages they may read. It is a request, not a lock: well behaved crawlers follow it.
Should I block AI crawlers?
It is your choice. Blocking them keeps your words out of AI training, but also out of AI answers that could send you buyers. Search crawlers like OAI-SearchBot cite and link to you.
Does blocking a page hide it from Google?
Not always. Google can still list an address it cannot read if other sites link to it. To keep a page out of search, use a noindex tag and let Google read it.
More free tools
Found something to fix? OOVERT fixes it.
We are the brand identity and web studio behind these tools. Brand, website and SEO, done properly in three weeks.