What does a robots.txt file actually control?
A robots.txt file is a plain-text list of crawl rules served at the root of a site. It tells compliant crawlers which URL paths they may fetch. It doesn't hide pages, password-protect anything, or remove URLs from search results; it only governs crawling. The format is standardised as the Robots Exclusion Protocol in RFC 9309.
Google is blunt about the limits in its robots.txt introduction: the file "is not a mechanism for keeping a web page out of Google." A disallowed URL can still be indexed if other sites link to it, and it then appears without a description.
How the generator turns your inputs into rules
The tool writes exactly one group: a User-agent line, any comments you typed, one or more Disallow lines, and an optional Sitemap line.
| Input | What ends up in the file |
|---|---|
User-agent (default *) |
User-agent: *, or whatever token you type. A blank field falls back to * |
| Paths, one per line | One Disallow: line per path, trimmed, in the order typed. Blank lines are skipped |
Lines starting with # |
Copied into the group as comments, in place, and never turned into rules |
| No paths, Allow all checked | Disallow: with no value, which blocks nothing |
| No paths, Allow all unchecked | Disallow: /, which blocks the entire site for that user-agent |
| Sitemap URL | A blank line, then Sitemap: <your URL> |
Every path must start with / or with a * wildcard. RFC 9309 defines a rule's path as starting with /, and Google also accepts a leading *. If any line breaks that rule, for example admin/ or a full URL like https://example.com/admin, the tool shows an error naming the offending lines instead of writing a file crawlers would ignore. A list that contains only comments counts as no paths, so the Allow all checkbox decides the Disallow line. There's no Allow field; add Allow lines by hand if you need exceptions inside a blocked folder.
Example: blocking an admin area and cart
With these three paths in the input (the blank line and extra spaces are dropped):
/admin/
/cart
/search?q=
and the sitemap https://example.com/sitemap.xml, the output is:
User-agent: *
Disallow: /admin/
Disallow: /cart
Disallow: /search?q=
Sitemap: https://example.com/sitemap.xml
Type a comment such as # Private areas above /admin/ and it appears in the output between the User-agent line and Disallow: /admin/, exactly where you put it.
Note what Disallow: /cart really matches. Rules are prefix matches against the path, and Google's robots.txt specification uses /fish as its example: it matches /fish.html, /fish/salmon.html and /fishheads, but not /Fish.asp, because matching is case-sensitive. So /cart would also block a page at /cartography. Add a trailing slash (/cart/) or an end-of-URL anchor (/cart$) when you mean one exact thing. Google supports * and $ in paths, and the generator passes them through untouched.
Publishing the file
- Generate the rules and click Copy.
- Save them as a UTF-8 plain-text file named exactly
robots.txt. - Upload it to the web root so it loads at
https://yourdomain/robots.txt. A file in a subfolder is ignored. - Open that URL in a browser and confirm the text matches.
- If you added a Sitemap line, check the sitemap itself with the sitemap validator. The Sitemap value must be a fully qualified URL including protocol and host.
Google generally caches robots.txt for up to 24 hours, so changes can take a day to apply. It also ignores anything beyond 500 KiB, a limit you're unlikely to hit with a hand-written file.
Mistakes that block the wrong thing
Unchecking Allow all by accident. With no paths listed, that checkbox is the difference between allowing everything and Disallow: /. For User-agent: * the second version asks every compliant crawler to skip your whole site.
Using Disallow to de-index. If a page is already in search results and you disallow it, Google can no longer crawl it to see a noindex tag, so the URL may linger. Google's guide to blocking indexing says the page must not be blocked by robots.txt for noindex to work, and that noindex inside robots.txt isn't supported. Add the tag with the meta tag generator and leave the URL crawlable.
Relying on it for secrets. The file is public, and the protocol is voluntary. Listing /private-reports/ tells everyone where to look. Protect sensitive paths with authentication.
Expecting one group to merge with another. If you write a group for Googlebot and another for *, Googlebot reads only its own group.
Blocking AI training crawlers
You can target a specific bot by typing its token in the User-agent field. Google documents Google-Extended as a product token that controls whether content Google crawls may be used for Gemini training and grounding, and says it doesn't affect inclusion or ranking in Google Search. Entering Google-Extended, no paths and Allow all unchecked produces a two-line group with Disallow: /. Other AI vendors publish their own tokens; check their documentation for the exact name.
To see a page roughly as a non-JavaScript crawler fetches it, try the search engine spider simulator.
Last updated