robots.txt Checker
Free robots.txt Checker
Fetch and validate a site's robots.txt live — User-agent groups, Allow/Disallow rules, Sitemap references and common issues, all in one report.
robots.txt Checker
Fetch and validate a site's crawler directives and sitemap references.
Enter a domain above to run a real, live check.
Capabilities
What it checks
Directive parsing
Every User-agent group's Allow, Disallow and Crawl-delay rules.
Sitemap discovery
Extracts every Sitemap: directive referenced in the file.
Issue flagging
Warns on site-wide blocks and other common misconfigurations.
Why use it
Benefits
- Confirm a site isn't accidentally blocking search engines after a deploy.
- Audit crawler rules before a migration or redesign.
- Export the parsed report as JSON for documentation.
FAQ
Frequently Asked Questions
It fetches robots.txt, parses each User-agent group's Allow/Disallow rules, and flags common issues like blocking all crawlers.