Robots.txt Generator
Robots.txt Generator
Build a robots.txt file visually: configure User-agent rules, Allow/Disallow paths, Crawl-delay, and Sitemap URL. Also checks an existing robots.txt against 16 known AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and more) to show which are allowed or blocked.
The * user-agent applies rules to all crawlers. Place your robots.txt at the root of your domain.
Robots.txt Generator
Build a robots.txt file visually: configure User-agent rules, Allow/Disallow paths, Crawl-delay, and Sitemap URL. Also checks an existing robots.txt against 16 known AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and more) to show which are allowed or blocked.
Common uses
- Block Googlebot from the /admin/ path
- Set a crawl-delay for all bots
- Reference a sitemap URL in robots.txt
- Check whether a robots.txt blocks GPTBot or ClaudeBot
- See which AI crawlers a website allows
Also searched as
- robots.txt generator
- block gptbot in robots.txt
- block ai crawlers robots.txt
- check if robots.txt blocks claudebot
- robots.txt disallow admin folder
- robots.txt crawl-delay example
- which ai bots can crawl my site
- robots.txt allow all example
- block google-extended ai training
- import and edit existing robots.txt
Frequently asked questions
Is my data uploaded to a server?
No, with one exception: the AI Crawler Check's "fetch remote robots.txt" option sends a request directly to the domain you enter — that's the only way to check its actual file. The Generator tab is fully local.
How do I block GPTBot and other AI crawlers?
Add a User-agent rule for each bot with Disallow: / in the Generator tab. Note that robots.txt is a request that well-behaved crawlers honor, not an enforcement mechanism.
How can I check which AI crawlers my robots.txt allows?
Paste your robots.txt (or fetch a site's) into the AI Crawler Check tab. It tests the root-level rule against 16 known AI crawlers, including GPTBot, ClaudeBot, PerplexityBot and Google-Extended.
Can I add a sitemap and crawl-delay?
Yes. The generator lets you set Allow/Disallow paths, Crawl-delay and a Sitemap URL.