Robots.txt Checker

What is robots.txt?

robots.txt is a text file at the root of your domain (/robots.txt) that tells crawlers which pages or sections they're allowed or not allowed to crawl. It's a directive, not a security measure — compliant bots respect it; malicious scrapers may not. Google, Bing, and other search engine crawlers all respect robots.txt.

What does Disallow: / mean?

Disallow: / blocks the specified user-agent from crawling any page on your site. If applied to User-agent: *, this blocks all crawlers from the entire site. This is a common mistake that prevents search engines from indexing any of your content.

Should I block crawlers from my admin area?

Yes — but robots.txt alone is not sufficient security. Bots (including malicious ones) can ignore it. Use server-side authentication and IP allowlisting for admin areas. robots.txt's role is to save crawl budget, not provide security. Add Disallow: /admin/ to prevent it from appearing in search results.

Does robots.txt affect indexing?

Blocking a URL in robots.txt prevents crawling but not necessarily indexing. Google can still index a blocked URL if it sees links pointing to it — it just can't read the content. To prevent indexing, add a noindex meta tag on the page itself (requires the page to be crawlable) or use the X-Robots-Tag header.