What is robots.txt?

How does robots.txt work?

A crawler checks the robots.txt file at the root of a host and applies rules for its user-agent token. Rules can allow or disallow matching paths. The scope depends on the host, protocol, and port where the file is served.

Participating crawlers follow the protocol, but robots.txt is not a firewall and cannot stop arbitrary clients from requesting public content.

• User-agent groups identify the crawlers to which instructions apply.
• Path rules express the crawling permissions within each group.
• Separate hosts require their own relevant robots.txt configuration.

Why does robots.txt matter for B2B SaaS?

SaaS websites commonly include marketing pages, documentation, account routes, and generated search URLs. Crawl rules can guide participating bots away from low-value public paths while preserving access to useful pages.

AI providers may expose separate tokens for training, search, and user-directed retrieval. A broad block can therefore have consequences that differ from the publisher’s intended training preference or visibility goal.

• Inventory each crawler’s documented purpose before writing a rule.
• Keep valuable documentation accessible to intended search crawlers.
• Protect private application data with authentication rather than crawl instructions.

What is an example of robots.txt?

A publisher could disallow GPTBot while allowing OAI-SearchBot because OpenAI documents these as independent controls for different purposes.

The exact policy should reflect the company’s preferences and current provider documentation. A separate rule might prevent compliant crawlers from exploring a public internal-search path.

Neither rule makes a confidential resource secure or proves that a page has been removed from an existing index.

• Use the exact documented token for the intended crawler.
• Check how broad rules interact with more specific groups.
• Do not assume another provider interprets nonstandard directives identically.

How do you improve or measure robots.txt?

Review the live file alongside server logs, indexing status, and firewall behavior. A crawler can be allowed in robots.txt yet blocked by a content delivery network or authentication challenge.

Conversely, blocking crawling can prevent a search engine from seeing a page-level noindex instruction. Test representative public URLs after policy changes and document the reason for each important crawler-specific rule.

• Confirm that the root file returns the intended public response.
• Check affected paths instead of testing only the homepage.
• Review rules after migrations, subdomain launches, and security changes.

Frequently asked questions about robots.txt