What is robots.txt?

The short answer

Robots.txt is a text file at a website root that tells compliant crawlers which URL paths they may or may not request.

Step by step

1. Request https://example.com/robots.txt.
2. Check the User-agent, Allow, Disallow, and Sitemap directives.
3. Confirm important public pages are not unintentionally disallowed.
4. Use noindex when you need indexing control; robots.txt is primarily a crawl directive.

Technical details

• Robots rules are host-specific and are not an access-control mechanism. A malicious client can ignore them.

Common mistakes

• Blocking the whole site with Disallow: /.
• Using robots.txt to hide private information.
• Adding unsupported directives such as Host for Googlebot.

Limitations

• SiteIndex does not currently expose a standalone robots checker; Website Score checks robots.txt availability and sitemap discovery.