Robots.txt Generator

What does each setting write into the file?

The access select decides the main group: “Allow all” and “Same as allow all” write User-agent: * followed by one Disallow line per path you list and one Allow line per allowed path, while “Block everything” writes User-agent: * with Disallow: /. A crawl delay above 0 adds a Crawl-delay line to the main group. The two bot options append one separate group per name. A sitemap URL adds a Sitemap line at the end.

Is the first select about Googlebot specifically?

No. Its label now reads “Access for all crawlers (User-agent: *)” because every option applies to the wildcard group. Nothing in the output names Googlebot. A crawler that has its own group, such as the AI and tool bots below, follows that group instead of the wildcard group.

Why do I get a bare Disallow: line when I list no paths?

An empty Disallow value means nothing is disallowed for that group. The tool writes it when the disallow box is empty so the User-agent: * group is not left without a rule. If you list paths, they replace it.

How do I enter several paths?

Separate them with commas or put each on its own line. Each one gets a leading / if it lacks one, so *.pdf is written as /*.pdf. Allow paths use the same rules. Paths are not tested against your site, so a typo is written as typed.

Does blocking a path keep it out of search results?

No. robots.txt controls crawling, not indexing. A blocked URL that other pages link to can still be indexed without its content. To keep a page out of results, use a noindex directive on a page that crawlers are allowed to fetch.

Which bots are in the built-in lists, and are the names reliable?

AI list: GPTBot, ChatGPT-User, Google-Extended, anthropic-ai, Claude-Web, CCBot, Omgilibot, FacebookBot, PerplexityBot, YouBot. Other list: AhrefsBot, SemrushBot, MJ12bot, DotBot, BLEXBot, Nikto. These are the tool’s lists, not a vendor-verified catalogue; names change and crawlers are added, so confirm each against the operator’s documentation. The AI list mixes training crawlers with fetchers that act on a user’s request, and Google-Extended is a robots.txt token that does not affect Google Search inclusion.

Does Crawl-delay work, and is robots.txt a security measure?

Googlebot ignores Crawl-delay, though some other crawlers read it; the tool writes only whole seconds above 0. robots.txt is a public file that compliant crawlers choose to follow. Anyone can read it, so listing a private path there advertises it. Protect private areas with authentication.