smart_toy Robots.txt Generator

Generate a custom robots.txt file.

Set your own Allow/Disallow rules, sitemap, and crawl-delay. Free, instant, no signup.

edit_note Rules
description Result 
User-agent: *
Disallow:

Fill in the form and click Generate.

How to Use

  1. 1

    List the paths to block in "Disallow paths," one per line, like /admin/.

  2. 2

    Optionally add Allow paths to carve out exceptions inside a blocked folder.

  3. 3

    Add your sitemap URL and a crawl-delay if you want them, then click Generate.

  4. 4

    Download the file and upload it to your site's root as robots.txt.

Frequently Asked Questions

Upload it to the root of your website, so it's reachable at yoursite.com/robots.txt. It must be at the root - a subfolder won't work.
Disallow blocks crawlers from a path. Allow can carve out an exception inside a disallowed folder, letting crawlers access one specific file or subfolder within it.
Not always. It stops crawling, but a disallowed URL can still be indexed (without content) if other sites link to it. Use a noindex meta tag for a guaranteed removal.