--- title: "Robots.txt Generator - Build Crawl Rules Online | CanDoYa" description: "Free robots.txt generator for valid crawler rules, Allow and Disallow paths, and sitemap URLs. Preview, copy, or download your file in the browser." url: https://candoya.com/tools/robots-txt-generator/ locale: en type: browser-tool --- # Free Robots.txt Generator Category: SEO & Web ## What is a robots.txt generator? A robots.txt generator builds the plain-text file that tells search engines and other well-behaved crawlers which URL paths they may request. Add user-agent groups, Allow and Disallow paths, and sitemap URLs, then copy or download standards-aware output without memorizing the directive syntax. Everything is generated locally in your browser. A `robots.txt` file belongs at the root of a site, such as `https://example.com/robots.txt`. This builder follows the core Robots Exclusion Protocol fields defined by RFC 9309 and adds the widely supported `Sitemap` field. It checks user-agent tokens, path formatting, and absolute sitemap URLs while you type. Presets give you a safe starting point for an open site, WordPress, or a complete crawl block. Robots rules control crawling, not access or guaranteed indexing. Anyone can read the file, and crawlers can choose whether to honor it. Use authentication or server permissions for private content, and use a supported `noindex` method when the goal is to keep a page out of search results. ## How to use it 1. **Choose a starting point** - Load the starter, open-site, WordPress, or block-all preset, or clear the form for a blank configuration. 2. **Set crawler rules** - Use * for all crawlers or enter a named user-agent, then add one Allow or Disallow path per line. 3. **Add sitemap URLs** - Enter any complete sitemap or sitemap-index URLs. The preview and checks update as you type. 4. **Copy or download robots.txt** - Review full-site blocking warnings, then copy the output or download the ready-to-upload robots.txt file. ## Who it's for - **Site owners** creating a first robots.txt file without hand-writing every directive. - **SEO specialists** checking crawl rules before they are placed on a production domain. - **Developers** documenting crawler access for admin areas, search pages, or generated URL spaces. - **WordPress administrators** starting with a focused wp-admin rule and an exception for admin-ajax.php. ## Expert note An Allow rule does not grant access by itself. Crawlers compare matching Allow and Disallow paths and use the most specific match; when equally specific rules conflict, RFC 9309 says Allow should win. That is why WordPress can disallow /wp-admin/ while allowing the longer /wp-admin/admin-ajax.php path. ## Frequently asked questions ### Is this robots.txt generator free? Yes. The generator is free to use with no account, sign-up, or usage limit. You can create multiple user-agent groups and sitemap entries, then copy the output or download it as robots.txt. ### Is my robots.txt configuration uploaded? No. The rules and sitemap URLs you enter are processed locally in your browser. The generator does not upload your configuration or send it to a server. ### How many rules or sitemap URLs can I add? The tool sets no fixed limit on groups, paths, or sitemap URLs. Search crawlers may impose their own file-size handling limits, so keep the final file focused and avoid duplicate rules. ### Where should I upload robots.txt? Place the UTF-8 plain-text file at the top-level /robots.txt path for the exact scheme and host it controls. For example, rules for https://www.example.com belong at https://www.example.com/robots.txt, not in a subdirectory. ### Does robots.txt stop a page from appearing in Google? Not reliably. A Disallow rule asks compliant crawlers not to request a URL, but Google may still know the URL from links and show it without a snippet. Use an appropriate noindex directive when you need to prevent indexing, and do not block the crawler from seeing that directive. ### Should I add Crawl-delay? Only when you know the target crawler supports it. Crawl-delay is not part of the core RFC 9309 rule syntax, and Google does not support it. This generator leaves it out so the main workflow stays portable and standards-aware. ### Can robots.txt protect private files? No. A robots.txt file is public and its paths can reveal where content lives. It is a voluntary crawling instruction, not authentication or authorization. Protect sensitive resources with login controls, server permissions, or another real access-control layer. ## Related tools - [Meta Tag Generator](https://candoya.com/tools/meta-tag-generator/) - [Open Graph Generator](https://candoya.com/tools/open-graph-generator/) - [SERP Snippet Preview](https://candoya.com/tools/serp-snippet-preview/) - [UTM Link Builder](https://candoya.com/tools/utm-link-builder/) - [Breadcrumb Schema Generator](https://candoya.com/tools/breadcrumb-schema-generator/) --- *Markdown edition of , published for AI agents and other automated readers. Index of key pages: .*