How to Use Robots.txt Generator: Complete Guide (2026)

What is Robots.txt Generator?

A robots.txt generator builds the plain-text file crawlers request at the site root. ToolsMinify's free generator starts from allow all, block all, or block admin, then turns User-agent, Allow, Disallow, and an optional sitemap URL into copy-ready robots.txt. Save the file at https://your-domain/robots.txt so Googlebot and others know which paths they may fetch. Everything runs in your browser.

Generate a robots.txt file with allow-all, block-all, or block-admin presets. Add Allow/Disallow paths and a sitemap URL - free, instant, and private.

Why use Robots.txt Generator?

• Presets for allow all, block all, and block admin so you are not writing directives from memory
• Live robots.txt as you edit User-agent, Allow paths, and Disallow paths
• Optional Sitemap: line with an absolute URL at the end of the file
• Paths are trimmed, de-duplicated, and given a leading slash when needed
• Runs entirely in your browser - drafts never leave the device

How to use Robots.txt Generator - step by step

1. Step 1: Choose a preset: allow all, block all, or block admin.
2. Step 2: Edit the User-agent if you need a named crawler instead of *.
3. Step 3: Add or change Disallow and Allow paths (one per line).
4. Step 4: Optionally paste your sitemap URL.
5. Step 5: Copy the robots.txt output and save it at https://your-domain/robots.txt.

The Robots.txt Generator lives under SEO Tools. Open the tool page, enter your input in the main field, and results update instantly. Use the copy button to paste output into documents, code editors, or spreadsheets.

Common use cases

Launch a public site

Use allow all plus your XML sitemap URL so crawlers can fetch every public path and discover URLs from one file.

Keep staging out of the index

Use block all (Disallow: /) on a staging host. Add noindex on the pages themselves if that host is ever reachable from the open web.

Hide CMS admin areas

Use block admin to Disallow /admin/ and /wp-admin/ while blog posts and product pages stay crawlable.

Fine-tune a named crawler

Set User-agent to Googlebot, mix Allow and Disallow prefixes, and keep a Sitemap line for URL discovery.

Examples

• Public site, allow everything: Preset allow all emits User-agent: * and Allow: /. Add a sitemap URL so crawlers can find your XML sitemap in the same file.
• Staging site, block all crawlers: Preset block all emits Disallow: / so Googlebot and others stay out of a staging or private host. Pair this with noindex if the host is ever public.
• Block CMS admin paths: Preset block admin writes Disallow: /admin/ and Disallow: /wp-admin/ while the rest of the site stays crawlable.
• Named crawler plus custom paths: Set User-agent to Googlebot, Disallow /tmp, Allow /public/, and a sitemap URL to produce a mixed group plus a Sitemap: line.

Pro tips

• The file must live at the site root. example.com/blog/robots.txt is ignored.
• robots.txt is not a security control and does not noindex a URL. Use noindex when a page must stay out of results.
• Use absolute https sitemap URLs. Relative paths are easy to misread.
• Google ignores Crawl-delay. Skip it and use Search Console if you need to slow Googlebot.
• After deploy, test the live file in Google's robots.txt tester and request indexing for key URLs.

Frequently asked questions

Where do I put robots.txt?

Save the file at the site root: https://www.example.com/robots.txt. Crawlers only fetch that exact path. A file in a subdirectory is ignored.

What is the difference between Allow and Disallow?

Disallow lists path prefixes crawlers should skip. Allow lists prefixes they may fetch, useful when a parent path is disallowed. An empty path list in this generator defaults to Allow: / (the whole site).

Does robots.txt hide pages from Google?

No. It asks crawlers not to fetch those URLs. Blocked pages can still appear in search if they are linked elsewhere. Use noindex on the page when you need it kept out of results.

Should I add a Sitemap line?

Yes when you have an XML sitemap. The Sitemap: directive is independent of User-agent groups and helps Google, Bing, and others discover URLs. Use an absolute https URL.

Is Crawl-delay included?

No. Google ignores Crawl-delay. Use Search Console crawl settings if you need to slow Googlebot. This generator sticks to User-agent, Allow, Disallow, and Sitemap.

Does my file leave the device?

No. The Robots.txt Generator runs entirely in your browser. Paths and sitemap URLs are not uploaded or stored on a server.

Related tools you might need

Explore other seo tools on ToolsMinify. Related utilities are linked on the Robots.txt Generator page to help you complete your workflow without leaving the site.