A robots.txt file is a small text file at the root of a host that tells crawlers which URL paths they may fetch. It is not a password, not a noindex tag, and not a way to hide secrets. Get it right and Googlebot, Bingbot, and others spend crawl budget on the pages you care about. Get it wrong and you block CSS, hide a sitemap, or leak a staging site into search. This guide covers the rules that still matter in 2026, the difference between Disallow and noindex, and the mistakes that cost rankings. When you want a file you can paste at https://your-domain/robots.txt, use the Robots.txt Generator - it runs in your browser and never uploads the paths.
Quick answer
Save robots.txt at the site root of each host (https://www.example.com/robots.txt). Use User-agent groups with Allow and Disallow path prefixes, plus a Sitemap: line with an absolute https URL. Allow: / is the usual public-site default. Disallow: / is for staging. robots.txt only asks crawlers not to fetch; blocked URLs can still appear in search if they are linked. Use a noindex robots meta tag (or X-Robots-Tag) when you need a URL kept out of results. Never use robots.txt to hide private data - the file itself is public.
What robots.txt is (and is not)
robots.txt is the Robots Exclusion Protocol: a voluntary set of rules crawlers look up before they request other URLs on that host. Well-behaved bots (Googlebot, Bingbot, many SEO crawlers) honor it. Scrapers and some AI fetchers may ignore it. Treat the file as a crawl hint, not a security control.
- It is a public text file. Anyone can open https://your-host/robots.txt and read every Disallow you wrote.
- It is per host. www.example.com and example.com each need their own file if both serve content.
- It does not authenticate users. A Disallow line does not stop a browser, a scraper, or a person with the URL.
- It does not remove URLs from Google on its own. That is what noindex (and removals in Search Console) are for.
Search engines still need to fetch CSS, JavaScript, and images to render a page the way a person sees it. Blocking those assets is a classic way to tank how Google understands the layout. A good robots.txt is boring: allow the public site, block a few admin prefixes, point at the sitemap.
Where the file must live
Crawlers only request one path: /robots.txt on that host. A file anywhere else is ignored.
- Correct: https://www.example.com/robots.txt
- Ignored: https://www.example.com/blog/robots.txt or https://www.example.com/static/robots.txt
- www and apex are different hosts. If both respond, each needs a file (or one should redirect to the other before crawlers look).
- Serve it as text/plain with HTTP 200. An HTML soft 404, a login wall, or a 500 looks like a missing or broken file.
- Keep it at the site root of the canonical host you want indexed, matching the host in Search Console.
The syntax that actually matters
A robots.txt file is groups of rules. Each group starts with one or more User-agent lines, then Allow and Disallow lines for that group. Sitemap lines sit outside the groups and apply to every crawler that understands them.
- User-agent: * - the default group for crawlers that do not have a more specific group.
- User-agent: Googlebot - a named group. Google uses the most specific matching group, not * plus Googlebot together.
- Disallow: /admin/ - skip URL paths that start with /admin/.
- Allow: / - allow the whole site. An empty Disallow is the older way to say the same thing.
- Sitemap: https://www.example.com/sitemap.xml - an absolute https URL to your XML sitemap.
- Comments start with #. Blank lines separate groups. Paths are case-sensitive.
Google supports * (any sequence) and $ (end of URL) in Allow and Disallow paths. Bing is close. Do not invent directives Google ignores. Crawl-delay is the usual example: Google does not use it. Use Search Console crawl settings if you need to slow Googlebot.
Allow vs Disallow
Disallow lists path prefixes a crawler should skip. Allow lists prefixes it may fetch, which is useful when a parent path is disallowed. Google picks the longest matching path, not the first line in the file.
- Disallow: / means skip everything on this host. Use that on staging, not on the public site.
- Allow: / (or no Disallow lines) means fetch the whole host.
- Disallow: /tmp plus Allow: /tmp/public/ lets Googlebot into the public folder while skipping the rest of /tmp.
- A more specific Allow can reopen a path that a shorter Disallow closed.
- Trailing slashes matter. /admin and /admin/ are different prefixes. Match the URLs your CMS actually serves.
Longest match wins
If one line says Disallow: /blog and another says Allow: /blog/how-to, Google treats /blog/how-to as allowed because that prefix is longer. Do not rely on line order. Write the prefixes you mean, then test a few URLs in Search Console.
robots.txt vs noindex
This is the distinction that saves (or wrecks) indexation. robots.txt and noindex do different jobs. Mixing them is how URLs get stuck in search with no title and no snippet.
- robots.txt Disallow - please do not fetch this URL. The crawler may still list the URL if other pages link to it.
- noindex (meta robots or X-Robots-Tag) - you may fetch this URL, but do not show it in results.
- If you Disallow a URL, Google often cannot see a noindex on that page, because it never downloads the HTML.
- To drop a public URL from the index, allow the crawl and put noindex on the page. Then wait for recrawl.
- To hide a private host, do not rely on robots.txt alone. Use HTTP auth, IP allowlists, or keep it off the public internet.
A thank-you page, a faceted filter URL, or an internal search results page usually wants noindex (and a crawl) so Google can see the tag. A staging host wants Disallow: / plus real access control. A CMS admin area can use both: Disallow the prefix so you do not waste crawl budget, and noindex if those URLs are ever reachable without a login.
Best practices for a public site
Most brochure and tool sites need a short, boring file. Start from allow-all, add a sitemap, then Disallow only the prefixes that should not burn crawl budget.
- Allow the whole public site (User-agent: * and Allow: /).
- Add one Sitemap: line with the absolute https URL of your XML sitemap.
- Optionally Disallow CMS admin paths such as /admin/ or /wp-admin/.
- Do not Disallow CSS, JavaScript, font, or image folders Google needs to render the page.
- Do not Disallow your locale prefixes (/es/, /pt/, /fr/, /de/) if those URLs are meant to rank.
- Keep parameter cleanup in Search Console or canonical tags, not in a maze of Disallow rules.
- Re-check the live file after every deploy. A leftover Disallow: / from staging is a common outage.
Staging, preview, and private hosts
If a host should never appear in search, Disallow: / is the robots.txt half of the job. It is not the whole job. Links, sitemaps, and Search Console properties can still surface the URLs.
- Put staging on its own host (staging.example.com), not on a public path of production.
- Disallow: / on that host, and serve noindex on every HTML response if the host is reachable at all.
- Prefer HTTP authentication or a VPN. robots.txt will not stop a crawler that ignores the protocol.
- Do not submit a staging sitemap to the production Search Console property.
- When you clone production to staging, replace the production robots.txt immediately. The allow-all file is the dangerous one here.
The Sitemap line
The Sitemap: directive is independent of User-agent groups. Google, Bing, and others use it as a discovery hint. It does not override Disallow. If you list a URL in the sitemap and also Disallow it, you are sending mixed signals.
- Use an absolute https URL. Sitemap: /sitemap.xml is not valid.
- You can list more than one sitemap (index plus child sitemaps, or locale-specific files).
- The sitemap host should match the robots.txt host when you can. Cross-host sitemaps need extra Search Console setup.
- Keep the sitemap to indexable URLs. Thank-you pages, cart URLs, and staging clones do not belong there.
Wildcards, end anchors, and comments
Google's robots.txt parser supports two extra tokens in paths. Use them sparingly. A long wildcard file is harder to test than a few exact prefixes.
- * matches any sequence of characters. Disallow: /*.json$ skips URLs that end in .json.
- $ matches the end of the URL. Without it, *.json would also match .json.html.
- # starts a comment through the end of the line. Do not put secrets, API keys, or internal hostnames in comments.
- If a crawler does not support * or $, it may treat those characters as literals. Stick to simple prefixes when you can.
Worked example
A public marketing site that also has a CMS admin area might ship this file. The generator's allow-all preset plus a sitemap and two Disallow lines produces the same shape.
- User-agent: *
- Allow: /
- Disallow: /admin/
- Disallow: /wp-admin/
- Sitemap: https://www.example.com/sitemap.xml
That file tells every crawler to fetch the public site, skip the two admin prefixes, and look at the XML sitemap. It does not noindex the admin URLs (login should already hide those). It does not block /_next/ or /images/. After you paste it, open the live /robots.txt URL in a browser and confirm the bytes match what you copied.
Common mistakes
- Putting the file in a subdirectory. Crawlers only fetch /robots.txt.
- Disallowing CSS, JavaScript, or font paths so Google cannot render the page.
- Using Disallow to 'hide' a page you still want removed from the index. Use noindex and allow the crawl.
- Leaving Disallow: / on production after a staging clone.
- A relative Sitemap: line. It must be an absolute https URL.
- Blocking og:image paths. Social crawlers still need to fetch the image. See Open Graph tags complete guide.
- Case mismatches (/Admin vs /admin/). Paths are case-sensitive.
- Crawl-delay lines that Google ignores, which give a false sense of rate control.
- Writing secrets in comments. The file is public.
- Assuming every bot honors the file. Some scrapers ignore it. That is an access-control problem, not a robots.txt problem.
Frequently asked questions
Does robots.txt hide pages from Google?
No. It asks Googlebot not to fetch those URLs. If other pages link to a blocked URL, Google can still list it with little or no snippet. To keep a URL out of results, allow the crawl and add noindex.
Where do I put robots.txt?
At the site root of the host: https://www.example.com/robots.txt. That exact path is the only one crawlers request. A file in a folder is ignored.
Should every site allow all crawlers?
Public sites that want to rank usually start with User-agent: * and Allow: /. Staging, preview, and private hosts should Disallow: / and add real access control. Named groups (Googlebot, Bingbot) are for exceptions, not for everyday files.
Do I need a Sitemap line?
Yes when you have an XML sitemap. It is a discovery hint, not a ranking switch, and it does not override Disallow. Use an absolute https URL. You can also submit the same sitemap in Search Console.
Is Crawl-delay still useful?
Not for Google. Google ignores Crawl-delay. If Googlebot is too aggressive, use the crawl settings in Search Console. Other bots may honor Crawl-delay, but this guide (and the generator) stick to User-agent, Allow, Disallow, and Sitemap.
How do I test the file?
Open the live /robots.txt URL and read it. Then use URL Inspection in Search Console on a few allowed and disallowed URLs. For a click-by-click walkthrough of the builder, see How to Use Robots.txt Generator.
Build the file in your browser
Start from allow all, block all, or block admin, then copy the live output. Nothing is uploaded:
- Robots.txt Generator - presets, Allow/Disallow paths, and an optional sitemap URL.
- How to Use Robots.txt Generator - step-by-step guide for the same tool.
- Open Graph Tag Generator - share cards that social crawlers must still be allowed to fetch.
- Slug Generator - clean URL slugs for the pages you want crawled.
- All SEO tools - the full category index.
Generate robots.txt - free
Open the Robots.txt Generator, pick a preset, add paths and a sitemap URL, and copy the file. Save it at the site root. Everything runs in your browser.