Seo
Robots.txt Rules Explained: Crawl Control, Not Access Control
How robots.txt controls compliant crawler requests—and why it is not authentication or a reliable noindex mechanism.
How robots.txt controls compliant crawler requests—and why it is not authentication or a reliable noindex mechanism.
Direct answer: A robots.txt file tells compliant crawlers which URL paths they may request. It is a crawl-management protocol, not authentication and not a reliable removal mechanism. If the goal is to keep a page out of a supported search index, the crawler generally needs to reach a noindex directive; if the goal is privacy, use access control.
What a robots file controls
A robots file is read from the origin root at /robots.txt. A group begins with one or more User-agent lines and is followed by rules such as Allow and Disallow. A Sitemap directive can point crawlers toward a sitemap, while comments document why a rule exists. The file can contain several groups, but a crawler applies the group it recognizes according to its own published interpretation.
User-agent: *
Disallow: /private/
Allow: /private/public-example.html
Sitemap: https://example.com/sitemap.xmlThe example says that the broad /private/ path is not intended for crawling, with one explicit exception. It does not stop someone who already knows the URL from requesting it.
Three jobs that are easy to confuse
- Crawl management: robots rules can reduce requests for paths a crawler should not fetch.
- Indexation: use a crawlable
noindexresponse or another search-engine-supported mechanism when a page should not enter an index. - Access control: use authentication, authorization, or network controls for private material. Never put credentials or secrets in a robots file.
Review checklist
- Fetch the exact origin-root file and confirm the response is the intended plain text.
- Read each group as a unit; look for an accidental
Disallow: /. - Check that important CSS, JavaScript, and image paths are not blocked without a clear reason.
- Validate sitemap URLs as absolute HTTP(S) URLs and confirm they describe the same site.
- Test representative crawler/path pairs locally; do not treat a local interpreter as proof of every bot’s behavior.
Our browser-side Robots Policy Generator helps draft and inspect policy text. It never fetches a remote URL and is not a security control.
This guide separates documented platform behavior from interpretation and identifies the source behind material claims.