Robots.txt Tester and Checker

Which rule actually wins

When several rules match the same URL, Google does not read the file top to bottom and stop at the first match. It picks the longest matching pattern, and if two matching patterns are exactly the same length, Allow beats Disallow. That is the whole precedence rule, and it decides most real cases.

The checker above shows you every rule that matched, not just the verdict, and marks which one won. Seeing the losing Disallow sitting next to the Allow that beat it is the fastest way to understand what your own file is really doing.

Blocked by robots.txt: find the cause, then fix it

A tester tells you a URL is blocked. It does not tell you which of several very different problems you have, and two of them are fixed by doing the opposite of what the other three need. Match the symptom first.

If the symptom is the second row, robots.txt is not your lever at all. Confirm what the page is really telling Google with the noindex checker, which reads the meta robots tag, the X-Robots-Tag header and the canonical in one pass. If the block is not in robots.txt and not in a noindex, the next suspect is your CDN or WAF serving Googlebot something different from what it serves you, which the Googlebot block tester will show you side by side.

Frequently asked questions

What does blocked by robots.txt mean?

It means the site's robots.txt file contains a rule telling that crawler not to fetch the URL, so Googlebot never downloads the page or reads anything on it. It is a crawling instruction, not a penalty and not a removal. Search Console splits it into two states that need opposite fixes: "Blocked by robots.txt" means the page is excluded and not indexed, while "Indexed, though blocked by robots.txt" means Google indexed the URL anyway from links pointing at it, with no description because it never saw the content.

How do I fix a blocked by robots.txt error?

First decide whether the block is a mistake. If the page should be in Google, find the Disallow that matches it, using the checker above to see the exact winning rule, then either delete that line or add a longer Allow rule that carves the page out of it. If the block is intentional and the page is still showing up in results, the fix is the opposite of what people expect: remove the Disallow so Googlebot can crawl the page, and add a noindex instead. A blocked page can never be read, so a noindex on it is never seen.

How do I test if robots.txt is working?

Testing the file loads is not the same as testing the rules. Confirm the file returns 200 at the domain root, then run a real URL you expect to be blocked and a real URL you expect to be crawlable through a tester and check both verdicts match your intent. Add one URL that sits inside a blocked directory but should be an exception, because that is where the longest-match precedence rule decides the outcome and where most files turn out to be wrong.

Is robots.txt legally enforceable?

No. Robots.txt is a voluntary standard, formalised as RFC 9309 in 2022, and it has no legal force on its own. Google, Bing and the other major search engines honour it. Scrapers and malicious crawlers routinely ignore it, and reading it is a fast way for them to find the directories you would rather keep quiet. Never use robots.txt to hide sensitive content. Use authentication for that.

What is a robots.txt checker?

A robots.txt checker takes a URL and tells you whether a crawler is allowed to fetch it under the rules in that site's robots.txt file. A useful one does more than answer yes or no: it names the exact Allow or Disallow line that decided the outcome, because a real file usually has several rules matching the same URL and only one of them wins. The checker above shows every matching rule and marks the winner.

How do I test robots.txt now that the Search Console tester is gone?

Google removed the robots.txt Tester from Search Console on 12 December 2023. Its replacement, the robots.txt report, shows which files Google fetched, when it last fetched them, and how many parse errors it found. It cannot test a URL against your rules and it cannot test rules you have not published yet. Use a third-party tester like the one above: paste the URL, pick the crawler, and either check the live file or paste your own rules to test a change before you ship it.

Which robots.txt rule wins when two rules match the same URL?

The most specific rule wins, and specificity means the longest pattern, not the position in the file. If Disallow: /blog/ and Allow: /blog/public/ both match /blog/public/post, the Allow wins because its pattern is longer. When two matching patterns are exactly the same length, Allow beats Disallow. Many free testers apply first-match-wins instead, which gives the wrong answer on any file that uses Allow to carve an exception out of a blocked directory.

Does robots.txt stop a page being indexed?

No. Robots.txt controls crawling, not indexing. A Disallow tells Googlebot not to fetch the page, but Google can still index the URL if other pages link to it, usually showing it with no description. That is the "Indexed, though blocked by robots.txt" status in Search Console. To keep a page out of Google, allow crawling and add a noindex, so Googlebot can actually fetch the page and read the directive.

Why does my page say blocked when robots.txt looks fine?

Three causes account for most of it. First, the rule matching is not what you assumed: a trailing wildcard or a missing $ makes a pattern match far more than intended, and the checker above shows you the exact line. Second, Google is reading a different group: if your file has a Googlebot block, Google uses that one alone and ignores the * block entirely, without merging them. Third, the file itself is broken, for example a rule sitting above the first User-agent line, which belongs to no group and is silently dropped.

Do Googlebot-Image and Googlebot-News follow the same rules?

Only when there is no group naming them specifically. Google picks the single most specific group that names the crawler and ignores every other group. So Googlebot-Image uses a Googlebot-Image block if one exists, otherwise the Googlebot block, otherwise the * block. It never combines them. That is why blocking Googlebot-Image without realising the site already has a Googlebot block can change image crawling in ways nobody expected. Switch the crawler in the tool above to see each one resolved separately.

Does Google obey Crawl-delay in robots.txt?

No. Google ignores the Crawl-delay directive completely. Bing and Yandex do honour it. If you need Googlebot to slow down, use the crawl rate settings in Search Console or return 503 responses. The checker above flags Crawl-delay lines so you know they are doing nothing for Google.

How often should I check robots.txt?

Checking once tells you about today. Robots.txt is a single shared file that gets edited by developers, migration scripts, plugins and staging environments, and a one-line change can remove an entire section of a site from crawling. Nothing in Search Console alerts you when the file changes. For any page that matters to revenue, re-check on a schedule so you find out the day the rule appears rather than the month the traffic drops.