Robots.txt Tester

A robots.txt tester evaluates whether a crawler is permitted to request a particular path according to the rules published at the site's /robots.txt location. It is a crawl-control check, not an indexing check. A URL can be allowed by robots.txt and still carry a noindex directive, canonicalize elsewhere, return an error, or remain unindexed for other reasons.

Robots rules are easy to misread because matching depends on user-agent groups, path prefixes and the most specific applicable rule. A single broad Disallow can prevent search crawlers from fetching templates, sections or resources that the site owner expected to be accessible. Testing the exact URL and crawler is safer than scanning the file by eye.

1. 1Enter the full URL you want to test, not only the domain.
2. 2Choose or enter the crawler name that matters to your audit, such as Googlebot.
3. 3Run the test and review the fetched robots.txt file, matching group and winning Allow or Disallow rule.
4. 4If no rule matches, remember that the default outcome is normally allowed.

• Allowed means the tested crawler is not blocked by the applicable robots.txt rules for that path.
• Blocked means a matching Disallow rule wins for the selected crawler and path.
• No robots.txt or a fetch failure is different from an explicit Allow. Read the fetch message before drawing a conclusion.

• Blocking an entire directory when only one private path was intended.
• Testing Googlebot while the file contains a more specific Googlebot-News or AdsBot group.
• Assuming robots.txt removes a URL from search results.
• Using robots.txt to protect confidential content.

• Narrow broad Disallow patterns and retest the exact affected paths.
• Use meta robots or X-Robots-Tag when the requirement is about indexing rather than crawling.
• Use authentication for private content; robots.txt is publicly readable and is not an access-control system.

• Keep rules as simple as the site architecture allows.
• Retest after migrations, CMS changes and staging-to-production releases.
• Link your sitemap from robots.txt when useful, but do not assume that makes every sitemap URL indexable.

This tool evaluates published robots rules and the requested crawler. It does not prove that Google has crawled, indexed or ranked the URL, and it cannot override server-side bot blocking that happens outside robots.txt.