Robots.txt Rules Explained: Crawl Control, Not Access Control

Direct answer: A robots.txt file tells compliant crawlers which URL paths they may request. It is a crawl-management protocol, not authentication and not a reliable removal mechanism. If the goal is to keep a page out of a supported search index, the crawler generally needs to reach a noindex directive; if the goal is privacy, use access control.

How robots.txt rules work

A robots file is read from the origin root at /robots.txt. A group begins with one or more User-agent lines and is followed by rules such as Allow and Disallow. A Sitemap directive can point crawlers toward a sitemap, while comments document why a rule exists. The file can contain several groups, but a crawler applies the group it recognizes according to its own published interpretation.

User-agent: *
Disallow: /private/
Allow: /private/public-example.html
Sitemap: https://example.com/sitemap.xml

The example says that the broad /private/ path is not intended for crawling, with one explicit exception. It does not stop someone who already knows the URL from requesting it.

Robots.txt, noindex and password protection

• Crawl management: robots rules can reduce requests for paths a crawler should not fetch.
• Indexation: use a crawlable noindex response or another search-engine-supported mechanism when a page should not enter an index.
• Access control: use authentication, authorization, or network controls for private material. Never put credentials or secrets in a robots file.

How to test your robots.txt rules

1. Fetch the exact origin-root file and confirm the response is the intended plain text.
2. Read each group as a unit; look for an accidental Disallow: /.
3. Check that important CSS, JavaScript, and image paths are not blocked without a clear reason.
4. Validate sitemap URLs as absolute HTTP(S) URLs and confirm they describe the same site.
5. Test representative crawler/path pairs locally; do not treat a local interpreter as proof of every bot’s behavior.

Our browser-side Robots Policy Generator helps draft and inspect policy text. It never fetches a remote URL and is not a security control.

Robots.txt is not a noindex directive

Use robots.txt to manage crawler access, not as a substitute for removing a page from search results. When a page must remain crawlable enough for search engines to see an exclusion signal, use the appropriate noindex vs. disallow control instead.

Rule examples: a directory, an exception and a file ending

For Google, path matching is case-sensitive. A more specific matching rule takes precedence; an equally specific Allow wins over Disallow. These illustrative patterns are not a ready-made policy for every website:

Rule	Purpose	Check before using it
Disallow: /drafts/	Exclude paths beginning with /drafts/ from crawling.	Confirm no public pages live in that directory.
Allow: /drafts/example.html	Permit a specific exception.	Test the full path in the same crawler group.
Disallow: /*.pdf$	Match paths ending in .pdf for crawlers supporting these patterns.	This does not remove a PDF from search; a query string changes the ending.

Which robots.txt file applies?

Check the exact protocol, hostname and port of the URL. A policy served from one subdomain is not automatically the policy for another. A 200 response is only the first check: confirm the body is robots text rather than an HTML error page or login screen. When testing AI crawler groups, consult that crawler operator’s documentation rather than assuming every bot implements Google’s behavior.

For the discovery side of crawl management, see how to select and validate XML sitemap URLs. To understand where crawling fits, start with how Google Search discovers, indexes and serves pages.

Sources & further reading

• Google: robots.txt matching and scope
• Google: blocking indexing with noindex