Labs

Robots.txt Tester

Test any URL against any crawler the way Google does — longest match, wildcards, $ — see the exact line that decides, catch syntax mistakes, and learn which AI bots your robots.txt really blocks.

Enter a site URL or paste robots.txt
robots.txt —
or drop a file here
Results —
Allowed / blocked per URL and crawler, with the deciding line, appear here
Settings Google fallback rules · AI summary on
How It Works

How to Use Robots.txt Tester in 3 Steps

1

Fetch or paste

Enter the site (the tool fetches /robots.txt) or the file’s own URL, or paste the contents. Add the URLs you care about — relative paths are fine — and the crawler names to test; the root / and a sensible default set of crawlers are included when you leave them blank. Press Test.

2

Read the matrix

Each cell says Allowed or Blocked and names the line that decided it. Matching follows RFC 9309 and Google’s implementation: the most specific user-agent group wins, then the longest matching path pattern, with Allow winning ties; * and $ work as wildcards.

3

Fix what the lint found

Issues above the table flag rules outside a group (ignored by crawlers), unknown or misspelled directives, Crawl-delay (ignored by Google), relative Sitemap URLs, the 500 KB limit and a root block for everyone. The AI section lists which LLM crawlers are blocked, allowed or partially blocked.

FAQ

Robots.txt Tester — Frequently Asked Questions

How exactly does matching work?

Google picks the group whose User-agent is the most specific match for the crawler (an exact product token first, then its parent — Googlebot-Image falls back to Googlebot — then *). Within that group every Allow and Disallow is compared to the URL path + query; * matches any run of characters and $ anchors the end. The longest matching pattern wins; if an Allow and a Disallow tie, Allow wins. An empty Disallow: means everything is allowed. This tool implements those rules; other crawlers (Bing, Yandex) agree on the essentials.

Why is a rule marked “outside a group”?

Allow / Disallow lines only count after a User-agent line. Rules at the top of the file before any user-agent — a common copy-paste mistake — are silently ignored by every major crawler, so the tool flags them as errors.

Does blocking GPTBot, ClaudeBot and others actually work?

Those crawlers publish that they honour robots.txt, and the summary tells you whether your file blocks them for the whole site, for parts of it, or not at all. It is a request, not a lock: content stays publicly reachable and bots that ignore the standard are not stopped. Blocking training crawlers (GPTBot, Google-Extended, CCBot) does not remove you from search; blocking search crawlers does.

What about noindex, Crawl-delay and Host?

noindex in robots.txt has been ignored by Google since 2019 — use a meta robots tag or X-Robots-Tag. Crawl-delay is honoured by Bing and Yandex but ignored by Google (set crawl rate in Search Console). Host is Yandex-only. All three are reported as warnings, not errors.

Can it check a staging robots.txt or one that isn’t live yet?

Yes — paste it. Paste mode never contacts any server, and you can test the exact file you are about to deploy. Use the Robots.txt & Sitemap Generator to build one from scratch.

Why does fetching sometimes fail — and what is the relay?

Browsers block JavaScript from reading other websites directly (CORS). The tool first tries a direct request; when the site forbids it, the URL only is sent to a public read-only relay (allorigins.win, then codetabs.com) that fetches the page and returns its HTML. The relay sees the URL you typed, nothing else — Toolity has no server and keeps no record. If the relay is busy or the site blocks bots, paste the page source instead: that path is 100% local.

Is my input sent anywhere?

No. Everything is computed by JavaScript in your browser tab. Nothing is uploaded, logged or stored by Toolity.