⚙️ Robots.txt Generator

robots.txt is a small but high-stakes file — a single wrong line can accidentally block your whole site from search engines. Writing it by hand is error-prone if you're not sure of the syntax. This tool builds a clean, correctly-formatted robots.txt from simple inputs, including your sitemap reference, so you get a valid file without guessing at the format.

How to Use This Tool

Optionally enter your sitemap URL and any paths you want to block (one per line), keep "allow all other pages" checked, and click Generate. Copy the result into a plain-text file named robots.txt at your site's root.

What This Tool Checks

This generates a robots.txt file from your inputs — it doesn't fetch or analyze an existing one (use the Robots.txt Tester for that). It simply produces correct syntax.

Understanding Your Results

The generated file allows all crawling by default and blocks only the specific paths you listed. The Sitemap line helps search engines find your sitemap. Review the paths you're blocking to make sure you're not accidentally hiding content you want indexed.

SEO Best Practices

Keep robots.txt minimal — only block what genuinely shouldn't be crawled (admin areas, cart/checkout, internal search results). Always include your sitemap URL. After uploading, verify with the Robots.txt Tester that important pages are still allowed.

Example

A typical WordPress robots.txt blocks /wp-admin/ and adds the sitemap line, while allowing everything else — exactly what this generator produces from those inputs.

Common Mistakes to Avoid

Don't block resources like CSS or JS folders — Google needs them to render and understand your pages. Don't use robots.txt to hide private data; it's a public file and Disallow doesn't provide security.