Skip to content

Robots.txt Tester

Test whether search engine crawlers may fetch a page under a site's robots.txt. The tester downloads robots.txt, picks the group for the user agent you choose, such as Googlebot or Bingbot, and applies the matching rules the way RFC 9309 and Google do: the longest matching rule wins, Allow beats Disallow on ties, and * and $ wildcards are supported. You can also paste your own robots.txt to try changes before you publish them.

Reset

Frequently asked questions

How do I use the tester?

Enter one or more URLs, one per line. The first must be a full URL so the tester knows which site's robots.txt to fetch; the following lines can be full URLs on the same site or paths such as /admin/. Choose the user agent and run the test.

How is the right group of rules chosen?

Crawlers use the group whose User-agent line matches their name, ignoring case. If there is no exact group, the tester uses the longest name that the crawler's name starts with, for example googlebot for Googlebot-Image, and otherwise the group for *. Rules from several groups with the same name are combined.

Which rule wins when several match?

The most specific one, meaning the rule with the longest path pattern. When an Allow and a Disallow rule are equally long, Allow wins. This is the behaviour described in RFC 9309 and used by Google and Bing.

Are wildcards supported?

Yes. An asterisk matches any sequence of characters and a dollar sign at the end anchors the rule to the end of the URL, so Disallow: /*.pdf$ blocks every URL that ends in .pdf. Matching is case-sensitive and includes the query string.

Can I test a robots.txt before publishing it?

Yes. Paste its content into the robots.txt field. The tester then uses your text instead of downloading the live file, so the first line can also be a path.

What if the site has no robots.txt?

When robots.txt returns a 4xx status such as 404, crawlers may fetch everything, and the tester reports every URL as allowed. When it returns a 5xx error, Google treats the whole site as blocked for a while, and the tester shows that too.

Does Disallow keep a page out of search results?

No. Disallow stops crawling, not indexing. A blocked page can still appear in results if other pages link to it, just without a description. Use a noindex meta tag or header on a crawlable page to keep it out of the index.

Why is Crawl-delay flagged?

Google ignores the Crawl-delay directive, while Bing respects it. The tester mentions it so you know that it does not slow down Googlebot.

Is there a size limit?

Google reads the first 500 KiB of a robots.txt file and ignores the rest. The tester reads the same amount and warns when a file is longer.

How many URLs can I test?

Up to 50 per request. Fetching the live robots.txt counts as one request, and each visitor can run a limited number of tests per minute.

About OneClick Pro

OneClick Pro is a free collection of more than 150 web tools for everyday work: converting images and units, formatting and validating code, checking domains and SEO, and working with text, dates and numbers. Every tool runs without registration or subscription, and what you enter is only used to calculate your result and is never stored. Use it as a quick toolbox for one-off tasks, or bookmark the tools you rely on every day.