Test robots.txt access rules
A robots.txt file gives crawlers instructions for paths on a site. Enter a public domain, a path such as /private/page, and a crawler token to test Allow and Disallow rules, including a wildcard or end-of-path anchor. The tool fetches robots.txt over HTTPS and tests the supplied path for that agent. Path and agent are preserved in the domain URL as query parameters; public history stores only the domain, so following a history link tests / with Googlebot.
This tool checks path rules only. It does not validate indexing, meta robots directives, or a crawler’s complete behavior. When robots.txt is unavailable, the tool reports that status explicitly; confirm important cases in the crawler’s own testing tool.