X-Robots-Tag Checker

The X-Robots-Tag is an HTTP response header that can carry indexing and snippet directives outside HTML. A checker reads the header exactly as delivered by the server or CDN and explains detected values.

Header-level directives are easy to miss in source-code reviews. They are commonly used for PDFs, files and server-wide policies, and a broad server rule can accidentally noindex entire asset classes or page sections.

1. 1Enter the resource URL, including non-HTML files if relevant.
2. 2Run the request and review all X-Robots-Tag header values.
3. 3Compare the header with HTML meta robots when checking webpages.
4. 4Trace unexpected directives to CDN, web-server or application configuration.

• noindex in the header is an explicit indexing exclusion.
• Crawler-specific forms can scope a directive to a named bot.
• Multiple header lines may combine into one effective policy.

• Server configuration adds noindex to every PDF when only one directory should be excluded.
• CDN rule persists after application tags are fixed.
• Conflicting header and HTML directives make debugging difficult.

• Remove or narrow the server/CDN rule that injects unwanted directives.
• Keep one authoritative policy where possible.
• Retest the raw response after cache purge.

• Use X-Robots-Tag when controlling non-HTML indexing.
• Document header policies in infrastructure code.
• Include response-header checks in launch QA.

The checker reports current headers from one request. Cached, geo-specific or authenticated responses can differ.