AI crawler robots.txt generator
Choose who reads your site.
Choose which AI crawlers may use your website, and get a robots.txt file to copy. Allow AI search so your pages can appear in ChatGPT, Claude and Perplexity answers, block AI training, or decide crawler by crawler. It's free and runs in your browser.
# robots.txt with AI crawler rules, made at https://vagejs.com/tools/ai-crawler-robots-txt/
# Search engines and other crawlers
User-agent: *
Allow: /
# AI crawlers you allow
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Google-Extended
User-agent: PerplexityBot
User-agent: Applebot-Extended
Allow: /Save it as robots.txt at the root of your domain (example.com/robots.txt). On vagejs, skip the file: seo: { robots: { aiTraining: "disallow", aiSearch: "allow" } } writes it on every build. How
How it works
- Start from a common choice, such as allowing AI search and blocking AI training, or pick crawler by crawler.
- Keep the rule for everyone else so Google, Bing and other search engines can still crawl your site, and add your sitemap's address if you have one.
- Copy the file and save it as
robots.txtat the root of your domain, for exampleexample.com/robots.txt.
Which AI crawler does what
| Crawler | Company | What it does | If you block it |
|---|---|---|---|
| GPTBot | OpenAI | Collects pages to train OpenAI's models | Your pages aren't used for training; ChatGPT search isn't affected |
| OAI-SearchBot | OpenAI | Finds pages to show in ChatGPT search | Your pages can't appear in ChatGPT search results |
| ClaudeBot | Anthropic | Collects pages to train Anthropic's models | Your pages aren't used for training |
| Claude-SearchBot | Anthropic | Finds pages for Claude's search results | Your pages can't appear in Claude's search results |
| Google-Extended | Controls use for Gemini training and grounding | Gemini doesn't use your pages; Google Search isn't affected | |
| PerplexityBot | Perplexity | Finds pages to show in Perplexity answers | Your pages can't appear in Perplexity answers |
| Applebot-Extended | Apple | Controls use for training Apple's models | Your pages aren't used for training; Apple search features aren't affected |
The list comes from each company's own crawler documentation and is maintained in vagejs's rules.
Questions
Does robots.txt stop AI crawlers?
It asks them to stay away, and the crawlers above say they follow it. It isn't a lock: a crawler that ignores robots.txt can still read public pages. To keep something private, don't publish it.
If I block GPTBot, will my site disappear from ChatGPT?
No. GPTBot only collects pages for training. ChatGPT search uses a separate crawler, OAI-SearchBot. Block GPTBot and allow OAI-SearchBot to stay out of training but still show up in ChatGPT search.
Does blocking Google-Extended hurt my Google ranking?
No. Google-Extended only controls whether Gemini may use your pages. Google Search crawls with Googlebot, which the "everyone else" rule covers.
Where does robots.txt go?
At the root of your domain: https://example.com/robots.txt. Crawlers don't look for it anywhere else, so a robots.txt inside a folder has no effect.
Can my website write this file for me?
With vagejs, yes: seo: { robots: { aiTraining: "disallow", aiSearch: "allow" } } in vage.config.ts writes robots.txt on every build, with the same crawler list. See SEO and AI search. To check whether AI search can reach your live site, read why AI search can't see your website, and for every step from robots.txt to a live test, see how to make your website visible in AI search.
Start with one command.
You'll have a site running in a minute.
npm create vagejs@latest my-site