Free tool
Robots.txt Checker
Paste a URL and see whether Googlebot is actually blocked from crawling it, with the exact rule that decided the answer. Or paste your own rules and test a change before it ever goes live. No signup required.
0 URLs entered · free check covers the first 20
Uses Google’s real precedence: the longest matching rule wins, and Allow beats Disallow on a tie. No signup, no credit card.
Which rule actually wins
When several rules match the same URL, Google does not read the file top to bottom and stop at the first match. It picks the longest matching pattern, and if two matching patterns are exactly the same length, Allow beats Disallow. That is the whole precedence rule, and it decides most real cases.
| URL being tested | Rules that match | Result |
|---|---|---|
| /blog/public/post | Disallow: /blog/ Allow: /blog/public/ | Crawlable. The Allow pattern is longer, so it wins. |
| /page | Allow: /page Disallow: /page | Crawlable. Equal length, so Allow wins the tie. |
| /products.php?id=9 | Disallow: /*? | Blocked. Matching runs against the query string too, not just the path. |
| /report.pdf.html | Disallow: /*.pdf$ | Crawlable. The $ anchors the pattern to the end of the URL. |
The checker above shows you every rule that matched, not just the verdict, and marks which one won. Seeing the losing Disallow sitting next to the Allow that beat it is the fastest way to understand what your own file is really doing.
Blocking a URL does not remove it from Google
Robots.txt controls crawling. It does not control indexing. A Disallow tells Googlebot not to fetch the page, but if other sites link to that URL, Google can still list it in search results, usually with no description underneath. Search Console reports this as Indexed, though blocked by robots.txt.
How to actually remove a page
Allow crawling and add a noindex. Googlebot has to be able to fetch the page to read the directive that removes it. Doing both at once is the classic trap: the page is blocked, so the noindex is never seen, and the URL can sit in the index indefinitely. Once the page has dropped out, you can add the robots.txt rule back to save crawl budget.
To check the other half of that pair, run the same URL through the noindex checker, which reads the meta robots tag, the X-Robots-Tag header and the canonical, and flags the blocked-plus-noindex conflict directly.
The Search Console robots.txt Tester is gone
Google removed the robots.txt Tester from Search Console on 12 December 2023. The replacement, the robots.txt report, is a different thing: it lists which robots.txt files Google found for your top hosts, when each was last fetched, and how many parse issues it saw. It cannot test a URL against your rules, and it cannot test a rule you have not published yet.
Test before you ship
Switch to My own rules, paste the file you are about to deploy, and run your real URLs through it. Nothing is fetched, so you find out that a new Disallow catches your product pages before it reaches production rather than after.
Per-crawler resolution
Google picks one group and ignores the rest. Googlebot-Image uses a Googlebot-Image block if there is one, otherwise Googlebot, otherwise *, never a merge of them. Switch the crawler and the tool re-resolves the group and tells you which one it fell back to.
Sitemaps, including the trap
Every Sitemap: line is extracted, and any sitemap that the same file blocks from being crawled is flagged. A site that declares a sitemap it will not let Google read is more common than it sounds.
Syntax that quietly fails
Rules above the first User-agent line belong to no group and are dropped. Noindex: in robots.txt stopped working in 2019. Crawl-delay is ignored by Google. Anything past 500 KB is never read. All of it is reported.
One shared file, edited by people who are not thinking about SEO
Robots.txt is unusually dangerous for its size. It is a single file that governs the whole site, it gets touched by developers, migration scripts, CMS plugins and staging environments, and one line can remove an entire section from crawling. The most common version of this is a staging file with Disallow: / copied to production during a launch.
Nothing announces it. Search Console will eventually show the coverage drop, but only after Google has recrawled and only if someone is looking. That is the gap a one-off checker cannot close: it tells you the truth about this moment and then forgets.
Keep watching after you close this tab
SearchOptimo re-checks your URLs on a schedule, keeps an index history timeline per URL, and alerts you the day a page drops out of Google. You find out when it happens, not when you next remember to look.
Want to know whether Google has already dropped the blocked pages? Run them through the Google index checker, which answers a different question: not “can it be crawled” but “is it in the index right now”.
Frequently asked questions
- What is a robots.txt checker?
- A robots.txt checker takes a URL and tells you whether a crawler is allowed to fetch it under the rules in that site's robots.txt file. A useful one does more than answer yes or no: it names the exact Allow or Disallow line that decided the outcome, because a real file usually has several rules matching the same URL and only one of them wins. The checker above shows every matching rule and marks the winner.
- How do I test robots.txt now that the Search Console tester is gone?
- Google removed the robots.txt Tester from Search Console on 12 December 2023. Its replacement, the robots.txt report, shows which files Google fetched, when it last fetched them, and how many parse errors it found. It cannot test a URL against your rules and it cannot test rules you have not published yet. Use a third-party tester like the one above: paste the URL, pick the crawler, and either check the live file or paste your own rules to test a change before you ship it.
- Which robots.txt rule wins when two rules match the same URL?
- The most specific rule wins, and specificity means the longest pattern, not the position in the file. If Disallow: /blog/ and Allow: /blog/public/ both match /blog/public/post, the Allow wins because its pattern is longer. When two matching patterns are exactly the same length, Allow beats Disallow. Many free testers apply first-match-wins instead, which gives the wrong answer on any file that uses Allow to carve an exception out of a blocked directory.
- Does robots.txt stop a page being indexed?
- No. Robots.txt controls crawling, not indexing. A Disallow tells Googlebot not to fetch the page, but Google can still index the URL if other pages link to it, usually showing it with no description. That is the "Indexed, though blocked by robots.txt" status in Search Console. To keep a page out of Google, allow crawling and add a noindex, so Googlebot can actually fetch the page and read the directive.
- Why does my page say blocked when robots.txt looks fine?
- Three causes account for most of it. First, the rule matching is not what you assumed: a trailing wildcard or a missing $ makes a pattern match far more than intended, and the checker above shows you the exact line. Second, Google is reading a different group: if your file has a Googlebot block, Google uses that one alone and ignores the * block entirely, without merging them. Third, the file itself is broken, for example a rule sitting above the first User-agent line, which belongs to no group and is silently dropped.
- Do Googlebot-Image and Googlebot-News follow the same rules?
- Only when there is no group naming them specifically. Google picks the single most specific group that names the crawler and ignores every other group. So Googlebot-Image uses a Googlebot-Image block if one exists, otherwise the Googlebot block, otherwise the * block. It never combines them. That is why blocking Googlebot-Image without realising the site already has a Googlebot block can change image crawling in ways nobody expected. Switch the crawler in the tool above to see each one resolved separately.
- Does Google obey Crawl-delay in robots.txt?
- No. Google ignores the Crawl-delay directive completely. Bing and Yandex do honour it. If you need Googlebot to slow down, use the crawl rate settings in Search Console or return 503 responses. The checker above flags Crawl-delay lines so you know they are doing nothing for Google.
- How often should I check robots.txt?
- Checking once tells you about today. Robots.txt is a single shared file that gets edited by developers, migration scripts, plugins and staging environments, and a one-line change can remove an entire section of a site from crawling. Nothing in Search Console alerts you when the file changes. For any page that matters to revenue, re-check on a schedule so you find out the day the rule appears rather than the month the traffic drops.
Test it once. Then never miss the next edit.
Checking by hand catches today’s Disallow. Monitoring catches the one that has not been written yet. Free forever plan, no credit card, cancel anytime.