Robots.txt Checker
Fetch and analyse the robots.txt file for any domain — see crawl rules, user-agent blocks, and sitemap references.
Did this tool work for you?
How to use this calculator
- 1
Enter the domain name (without https://).
- 2
Click "Check Now" to fetch https://domain.com/robots.txt.
- 3
Results show the number of user-agent rules, disallow/allow directives, and sitemap references.
- 4
The raw file content is also shown for detailed inspection.
Frequently asked questions
Which URL does the checker fetch?
https://<host>/robots.txt, where the host is taken from whatever you type. A full URL or a path is reduced to the host name, so typing https://www.example.com/shop/item fetches https://www.example.com/robots.txt. The host is lowercased and a trailing dot is removed, and www.example.com and example.com are treated as different hosts.
What limits apply?
The server allows 10 seconds, follows up to 8 redirects with the host re-checked at every hop, and reads at most 2 MB. Only public hosts are allowed, so localhost, private IP ranges and names ending in .local or .internal are refused. Beyond that, each client address is capped at 15 requests per minute across the site’s page-fetching tools.
What do User-agents, Disallow rules, Allow rules and Sitemap refs count?
Each is the number of lines in the file that begin with that directive, ignoring case, across the whole file. Counts are not grouped by crawler. An empty “Disallow:” line counts as a rule, comment lines are not counted, and a comment at the end of a directive line is not stripped from its value.
What does it show when the file is missing?
Any non-successful status produces the status code plus a tip that many sites omit robots.txt. The tool does not distinguish a 404 from a server error such as 500, and crawlers may treat those cases differently, so read the status code itself. If the server answers 200 with an HTML page, the tool reports “File found: Yes” with zero rules.
Can it tell me whether a particular URL is blocked?
No. It summarises the file and shows its text, but it does not match any URL or user-agent against the rules. To work it out, find the group for the crawler in question, then apply the most specific matching Allow or Disallow path.
How much of the file is displayed?
The File Content box shows the first 3,000 characters and then “… (truncated)”. The counts use the text read from the server, up to 2 MB, so they can include lines you cannot see in the box.
Does it validate syntax or check my sitemap links?
No. It does not report unknown directives, malformed lines or conflicting rules, and it does not fetch the sitemaps listed. Sitemap lines are listed and counted, and zero sitemap references shows as a warning.
What the robots.txt checker fetches and how it summarises the file
The request and its limits
Enter a domain and the server requests https://that-host/robots.txt. Only the host name is used from your input, so any path, query or port you type is dropped. The request times out after 10 seconds, follows up to 8 redirects, validates each hop as a public host, and reads at most 2 MB. Because the file is requested server-side, it reflects what a visitor outside your network is served. The allowance of 15 requests a minute per client address is pooled with the other page-fetching tools on the site.
Input forms and the URL they produce
These examples show the host-only behaviour, which matters when a site serves different files on its www and bare domains.
| Input | URL requested |
|---|---|
| example.com | https://example.com/robots.txt |
| https://www.example.com/shop/item?id=5 | https://www.example.com/robots.txt |
| EXAMPLE.com. | https://example.com/robots.txt |
| localhost | Refused: not a public host |
What the Summary lists
A successful fetch gives a Summary and the file text. The Summary reports File found, the number of User-agent lines, Disallow lines, Allow lines and Sitemap lines, and then each Sitemap URL on its own row. The Sitemap count turns to a warning when it is zero. Counting is line-based and case-insensitive over the whole file, so five groups with one Disallow each and one group with five Disallow lines both show five rules. Use the File Content box when the grouping matters, since that is where you see which rules belong to which crawler.
Reading the file text
Look first for a group that starts with User-agent: * and check what follows. A single Disallow: / under that group asks every compliant crawler to stay away from the whole site, which is correct for a staging host and a serious mistake on a live one. An empty Disallow line means nothing is blocked for that group. Check for groups addressed to named crawlers, because a crawler that has its own group follows it instead of the wildcard group. Look for Sitemap lines with full URLs. Remember that robots.txt applies per host and protocol, so the file you fetched says nothing about other subdomains.
What the tool leaves to you
The checker does not test URLs against rules, validate syntax, resolve the sitemaps listed, or tell a missing file from a server error. A blocked URL can still be indexed without its content if other pages link to it; robots.txt is about crawling, and a noindex directive needs the page to be fetchable. To draft a new file, the robots.txt generator writes one from your settings.
Learn more from an authoritative source:
Google Search CentralSitemap Validator
Fetch and validate the XML sitemap for any domain — count URLs, check for lastmod dates, and detect sitemap indexes.
Robots.txt Generator
Generate a robots.txt file to control how search engine crawlers access your website.
Meta Tag Generator
Generate HTML meta tags for SEO, social sharing, and viewport — ready to paste into your page head.
Open Graph Tag Generator
Generate Open Graph and Twitter Card meta tags to control how your page looks when shared on social media.
Keyword Density Calculator
Calculate keyword density and frequency for any text. Paste your content and keyword to see how often it appears and whether density is in the recommended SEO range.
Meta Description Checker
Check your meta description length, character count, pixel width, and SEO quality. Get instant feedback on whether it will display fully in Google search results.
Results are estimates for informational purposes only and do not constitute professional financial, medical, legal, or technical advice. Read full disclaimer →