robots.txt tester
Check a draft before you deploy it — with the line that decided
1 robots.txt
Try changing this to Googlebot: the example has a Googlebot‑specific group, so every User‑agent: * rule stops applying to it. That is correct RFC behaviour and it surprises almost everyone.
2 URLs to check
One per line. Full URLs or bare paths both work — a full URL is reduced to its path and query, which is all robots.txt matching ever looks at.
3 Verdicts
| Path | Verdict | Deciding rule |
|---|
Paste a robots.txt above and list some URLs — every one gets a verdict here instantly, with the exact line that decided it. No Generate button, no upload.
Honest limits
- This models RFC 9309 matching for the product token you type. It is not a prediction of crawler behaviour: Google ignores
Crawl‑delay, some bots ignore robots.txt entirely, and per‑crawler extensions are out of scope. - It never fetches anything. There is no request to your site and no lookup of a live robots.txt — you paste the draft, which is the whole point.
- robots.txt controls crawling, not indexing. A blocked URL can still appear in results if it is linked from elsewhere; use
noindexfor that. - Matching is case‑sensitive on paths and case‑insensitive on user‑agent names, exactly as the RFC specifies.
Runs entirely in this tab. Your draft and your URLs are never uploaded — nothing you type leaves the page. The page does load one cookieless pageview counter from stats.dankdev.com; it records this page’s address and title, never your robots.txt or the URLs you test.
How the verdict is decided
Group selection first: if any group names your crawler, User‑agent: * is ignored completely — a rule you wrote for everyone does not apply to a bot you named elsewhere. The name has to match your crawler’s product token exactly, ignoring case: Googlebot and googlebot are the same record, but goog matches nothing at all, and a Googlebot‑News group is not Googlebot’s. Consecutive User‑agent lines share one group, and every group naming your crawler is combined into a single rule set before anything is weighed — so a Disallow sitting in a second Googlebot group counts exactly as much as one in the first.
Then, inside that group, the rule with the longest pattern wins. On an exact tie Allow beats Disallow. If nothing matches at all, the path is allowed — that is the RFC default, and it is why an empty robots.txt permits everything.
* matches any run of characters including slashes, and $ anchors to the end of the path — so /*.pdf$ blocks /a/b.pdf but not /a/b.pdf?download=1, which is the single most common way a rule silently fails to do what its author meant.