Robots.txt Generator
Generate a robots.txt file to control how search engine crawlers access your website.
This document is a template for informational purposes only and does not constitute legal advice. Consult a qualified attorney for your specific needs.
Did this tool work for you?
How to use this calculator
- 1
Enter your sitemap URL.
- 2
Set which paths to disallow (admin pages, private areas, etc.).
- 3
Choose whether to block AI training crawlers and bad bots.
- 4
Set a crawl delay if your server needs breathing room.
- 5
Copy or download and upload as robots.txt to your site root.
Frequently asked questions
What is a robots.txt file?
A robots.txt file tells search engine crawlers which pages or sections of your site they should or should not access. It's placed in the root of your website (e.g. https://example.com/robots.txt) and read by crawlers before they visit any other page.
Does blocking a page in robots.txt hide it from Google?
No — robots.txt prevents crawling but not indexing. Google can still index a page it hasn't crawled if other sites link to it. To prevent indexing, use a noindex meta tag or X-Robots-Tag header instead.
Should I block AI crawlers?
This is a personal choice. Blocking AI crawlers (GPTBot, Claude-Web, CCBot, etc.) prevents your content from being used to train AI models. It has no effect on Google search rankings. Many publishers are choosing to block them to protect their content.
What happens if I have no robots.txt?
Without a robots.txt, all crawlers are allowed to access all public pages. This is fine for most sites. A missing robots.txt returns a 404, which crawlers treat as "allow all".
Free Robots.txt Generator
robots.txt vs noindex — what's the difference?
robots.txt controls crawling (whether search engines visit the page). The noindex meta tag controls indexing (whether search engines include the page in results). A common mistake is using robots.txt to hide pages — blocked pages can still appear in search results if linked from other sites. Use noindex for true invisibility.
Should you block SEO crawlers?
Tools like Ahrefs and Semrush use their own crawlers to build backlink databases. Blocking them in robots.txt stops them crawling your site but won't remove your site from their databases (they get data from other sources too). Block them if you're concerned about competitor research, but it's largely a cosmetic measure.
Learn more from an authoritative source:
Google Search CentralKeyword Density Calculator
Calculate keyword density and frequency for any text. Paste your content and keyword to see how often it appears and whether density is in the recommended SEO range.
Meta Description Checker
Check your meta description length, character count, pixel width, and SEO quality. Get instant feedback on whether it will display fully in Google search results.
Title Tag Checker
Check your SEO title tag for optimal length, pixel width, and keyword placement. See how it will appear in Google search results.
Reading Level Calculator
Calculate the reading level and readability score of any text using the Flesch-Kincaid formula. Find out what grade level your content targets.
Results are estimates for informational purposes only and do not constitute professional financial, medical, legal, or technical advice. Read full disclaimer →