Robots.txt Generator

Build a robots.txt file visually: configure User-agent rules, Allow/Disallow paths, Crawl-delay, and Sitemap URL. Also checks an existing robots.txt against 16 known AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and more) to show which are allowed or blocked.

Ln: 1  Col: 1 59 chars · 59 B13px

The * user-agent applies rules to all crawlers. Place your robots.txt at the root of your domain.

Robots.txt Generator

Build a robots.txt file visually: configure User-agent rules, Allow/Disallow paths, Crawl-delay, and Sitemap URL. Also checks an existing robots.txt against 16 known AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and more) to show which are allowed or blocked.

Common uses

  • Block Googlebot from the /admin/ path
  • Set a crawl-delay for all bots
  • Reference a sitemap URL in robots.txt
  • Check whether a robots.txt blocks GPTBot or ClaudeBot
  • See which AI crawlers a website allows

Frequently asked questions

Is my data uploaded to a server?

No, with one exception: the AI Crawler Check's "fetch remote robots.txt" option sends a request directly to the domain you enter — that's the only way to check its actual file. The Generator tab is fully local.