Where do I put robots.txt?

Save the file at the site root: https://www.example.com/robots.txt. Crawlers only fetch that exact path. A file in a subdirectory is ignored.

What is the difference between Allow and Disallow?

Disallow lists path prefixes crawlers should skip. Allow lists prefixes they may fetch, useful when a parent path is disallowed. An empty path list in this generator defaults to Allow: / (the whole site).

Does robots.txt hide pages from Google?

No. It asks crawlers not to fetch those URLs. Blocked pages can still appear in search if they are linked elsewhere. Use noindex on the page when you need it kept out of results.

Should I add a Sitemap line?

Yes when you have an XML sitemap. The Sitemap: directive is independent of User-agent groups and helps Google, Bing, and others discover URLs. Use an absolute https URL.

Is Crawl-delay included?

No. Google ignores Crawl-delay. Use Search Console crawl settings if you need to slow Googlebot. This generator sticks to User-agent, Allow, Disallow, and Sitemap.

Does my file leave the device?

No. The Robots.txt Generator runs entirely in your browser. Paths and sitemap URLs are not uploaded or stored on a server.

SEO Tools

Popular

Robots.txt Generator

Build a robots.txt file in your browser.

Preset

Start from a common file, then edit paths or add a sitemap URL.

User-agent

Use * for every crawler, or a name like Googlebot.

Disallow paths

Allow paths

Sitemap URL

Optional. Written as a Sitemap: line at the end of the file.

robots.txt

User-agent: *
Allow: /

2 lines

Save this file as robots.txt at the root of your site (example.com/robots.txt).

About Robots.txt Generator

Build a robots.txt file in your browser. Start from allow all, block all, or block admin, then edit User-agent, Allow, and Disallow paths and optionally add a sitemap URL. Copy the file and save it at the root of your site so crawlers know what they may fetch. Nothing is uploaded.

How to Use Robots.txt Generator

Choose a preset: allow all, block all, or block admin.
Edit the User-agent if you need a named crawler instead of *.
Add or change Disallow and Allow paths (one per line).
Optionally paste your sitemap URL.
Copy the robots.txt output and save it at https://your-domain/robots.txt.

Robots.txt Generator Examples

Public site, allow everything

Preset allow all emits User-agent: * and Allow: /. Add a sitemap URL so crawlers can find your XML sitemap in the same file.

Staging site, block all crawlers

Preset block all emits Disallow: / so Googlebot and others stay out of a staging or private host. Pair this with noindex if the host is ever public.

Block CMS admin paths

Preset block admin writes Disallow: /admin/ and Disallow: /wp-admin/ while the rest of the site stays crawlable.

Named crawler plus custom paths

Set User-agent to Googlebot, Disallow /tmp, Allow /public/, and a sitemap URL to produce a mixed group plus a Sitemap: line.

Read the full guide

How to Use Robots.txt Generator: Complete Guide (2026) - 5 min read

Frequently Asked Questions about Robots.txt Generator

Where do I put robots.txt?: Save the file at the site root: https://www.example.com/robots.txt. Crawlers only fetch that exact path. A file in a subdirectory is ignored.
What is the difference between Allow and Disallow?: Disallow lists path prefixes crawlers should skip. Allow lists prefixes they may fetch, useful when a parent path is disallowed. An empty path list in this generator defaults to Allow: / (the whole site).
Does robots.txt hide pages from Google?: No. It asks crawlers not to fetch those URLs. Blocked pages can still appear in search if they are linked elsewhere. Use noindex on the page when you need it kept out of results.
Should I add a Sitemap line?: Yes when you have an XML sitemap. The Sitemap: directive is independent of User-agent groups and helps Google, Bing, and others discover URLs. Use an absolute https URL.
Is Crawl-delay included?: No. Google ignores Crawl-delay. Use Search Console crawl settings if you need to slow Googlebot. This generator sticks to User-agent, Allow, Disallow, and Sitemap.
Does my file leave the device?: No. The Robots.txt Generator runs entirely in your browser. Paths and sitemap URLs are not uploaded or stored on a server.

Open Graph Tag Generator

Generate Open Graph and Twitter Card meta tags from a title, description, URL, and image. Free OG tag generator with live HTML and a social preview.

Meta Description Checker & SERP Preview

Free SERP length checker and meta description preview. Live character counter and Google snippet preview - or generate a draft. Fits the 150-160 limit.

Slug Generator

Turn any title or phrase into a clean, SEO-friendly URL slug. Lowercase, strip accents, and choose hyphen or underscore separators - free, instant, and private.