robots.txt Parser, Validator & Tester
A single misplaced character in robots.txt can silently block your entire site from crawling. This tool parses your file per RFC 9309, flags syntax errors with line numbers, visualizes rules per user-agent, and lets you test any URL to see if it's allowed or blocked.
Tool interface
Validation Results
⚠️ Issues Found
📋 Parsed Rule Groups
🗺️ Declared Sitemaps
🧪 Test a URL
Introduction
The robots.txt file is the first thing crawlers read, yet it's one of the most commonly misconfigured files on the web. A single syntax error can block your entire site from indexing. This tool gives you instant, local validation against the RFC 9309 standard — the official robots.txt specification.
How It Works
- Lines are tokenized and comments (
#) are stripped. - Directives are grouped by
User-agentblocks. - Syntax is validated against RFC 9309 (valid directives, ordering, values).
- The URL tester resolves paths using wildcard
*and end-anchor$patterns, applying longest-match precedence.
Privacy
Your robots.txt content is parsed entirely in your browser using string manipulation. No content is transmitted anywhere. No API calls, no tracking.
Limitations
- Cannot fetch live robots.txt files (CORS restrictions); you must paste the content.
- Does not validate whether sitemap URLs are reachable.
- Bot-specific extensions (e.g., Google's
noindexin robots.txt) are flagged as unknown directives.
FAQ
Why does my Disallow rule not block the URL?
Check if an Allow rule with a longer path overrides it, or if the rule is under a different User-agent group than the bot you're testing.
Is robots.txt case-sensitive?
Directives (User-agent, Disallow, etc.) are case-insensitive per RFC 9309. Path values are case-sensitive. User-agent names are case-insensitive.
Is my robots.txt content uploaded?
No. Everything is parsed locally in your browser. Nothing is transmitted.