What is robots.txt?
Learn how robots.txt guides compliant crawlers and how it differs from an XML sitemap.
The short answer
Robots.txt is a text file at a website root that tells compliant crawlers which URL paths they may or may not request.
Use the free Website Score CheckerUse the free HTTP Status Code CheckerStep by step
- Request https://siteindex.app/robots.txt.
- Check the User-agent, Allow, Disallow, and Sitemap directives.
- Confirm important public pages are not unintentionally disallowed.
- Use noindex when you need indexing control; robots.txt is primarily a crawl directive.
Technical details
- Robots rules are host-specific and are not an access-control mechanism. A malicious client can ignore them.
Common mistakes
- Blocking the whole site with Disallow: /.
- Using robots.txt to hide private information.
- Adding unsupported directives such as Host for Googlebot.
Limitations
- SiteIndex does not currently expose a standalone robots checker; Website Score checks robots.txt availability and sitemap discovery.
Related SiteIndex tool
Measure a public website's SEO, performance, accessibility, security, and technical health.
Use Website ScoreLearn moreCheck the HTTP response for a URL and review the status, server, and redirect chain.
Use HTTP Status CheckerLearn more