AI robots.txt generator
Decide which AI crawlers can read your site. Let ChatGPT, Claude and Perplexity cite you in answers, and keep your content out of AI training if you want to.
How the generator works
Pick which AI bots to allow, and get a robots.txt you can upload in minutes.
Pick a preset
Start from allowing AI search while blocking training, or allow or block every AI bot.
Fine-tune each bot
Switch 16 AI crawlers on or off, from GPTBot and ClaudeBot to PerplexityBot and Google-Extended.
Keep your existing rules
Paste your current robots.txt and the AI rules are added to it, with your private paths still protected.
Upload it
Copy or download the file and add it to the root of your site, at yourdomain.com/robots.txt.
Want AI search to cite you, not just crawl you?
We build AEO strategies that get B2B SaaS brands mentioned and cited in ChatGPT, Perplexity and AI Overviews.
Frequently Asked Questions
What is robots.txt?
robots.txt is a plain text file at the root of your site that tells crawlers which pages they may visit. Search engines and AI companies check it before crawling, and you can set different rules for each crawler by name.
Should I block AI crawlers?
It depends on the crawler. AI search and assistant bots, like OAI-SearchBot and PerplexityBot, fetch pages to answer questions and cite sources, so blocking them can keep you out of AI answers. Training crawlers, like GPTBot and CCBot, collect content to train models. Most B2B SaaS companies allow AI search and decide case by case on training.
Does blocking Google-Extended remove me from AI Overviews?
No. AI Overviews and AI Mode are part of Google Search and use Googlebot. Google-Extended only controls whether your content is used for Gemini training and grounding, and blocking it doesn't affect your Google rankings.
Do AI crawlers respect robots.txt?
The documented crawlers from major AI companies say they follow it. Fetchers that act on a user's request, like ChatGPT-User and Perplexity-User, may not always apply the rules, and some lesser-known crawlers ignore robots.txt entirely. It's a request, not a lock.
How do I add robots.txt to my site?
In Webflow, paste it into Site settings > SEO > robots.txt. In WordPress, use your SEO plugin's file editor or upload the file to your site's root folder. In HubSpot, edit it in your website settings under SEO and crawlers. Then check it loads at yourdomain.com/robots.txt.
