🤖 Robots.txt Tester

robots.txt is a plain-text file at the root of a domain that tells search-engine crawlers which parts of the site they're allowed to visit. Get it wrong — block the wrong folder, or accidentally disallow the whole site with a stray "Disallow: /" — and pages can silently vanish from search results with no error message anywhere. This tool fetches a site's real robots.txt, parses its rules, and can test whether a specific path is allowed or blocked for a given crawler.

How to Use This Tool

Enter a domain or URL, optionally a specific path you want to test (like /blog/some-post/), and optionally a user-agent (leave blank to test against the default "*" rules that apply to all crawlers). Click Test Robots.txt.

What This Tool Checks

Fetches robots.txt from the domain's root, parses all Allow/Disallow rules grouped by user-agent, lists any Sitemap: directives declared in the file, and — if you supplied a path — determines whether that specific path is allowed or blocked.

Understanding Your Results

A "blocked" verdict on a path means crawlers respecting robots.txt (which includes Googlebot) will not crawl that URL — it can still appear in search results if linked from elsewhere, but Google won't be able to read its content to judge relevance. A missing robots.txt (404) is not an error — it just means the site allows unrestricted crawling by default.

How to Fix Common Problems

If an important page is blocked, find the matching Disallow rule in your robots.txt and either remove it or add a more specific Allow rule above it for that path. Be especially careful with WordPress sites — plugins occasionally write "Disallow: /wp-admin/" (fine, that's meant to be blocked) but sometimes overly broad rules leak in and block real content too.

SEO Best Practices

Keep robots.txt as simple as possible — the more rules you have, the more likely one silently breaks something. Always declare your sitemap location in robots.txt with a "Sitemap:" line so crawlers can find it easily. Test any change here before assuming it worked; robots.txt mistakes are invisible until someone notices traffic dropped.

Example

A common accidental mistake: a staging-site robots.txt with "User-agent: * / Disallow: /" gets pushed to production during a site migration, and the entire live site becomes invisible to Google overnight. Checking robots.txt after any migration catches this instantly.

Common Mistakes to Avoid

Don't use robots.txt to try to hide sensitive information — it's a public file, and Disallow doesn't prevent the URL from being listed (just crawled); use noindex or authentication instead. Don't assume Disallow removes an already-indexed page — for that you need a noindex meta tag, which requires the page to be crawlable in the first place.