AI visibility checker
Can search and AI crawlers reach your site?
Enter a website address to see whether search engines and AI crawlers can reach its home page: Googlebot, Bingbot, OpenAI's, Anthropic's and Perplexity's crawlers. It also checks for robots.txt, a sitemap, llms.txt, structured data and noindex. It's free and needs no sign-up.
Enter a website address and press Check.
You'll see, for each search engine and AI crawler:
✓ it can read the site, or
✗ why not: blocked in robots.txt, or blocked by the server
and whether the site has robots.txt, a sitemap, llms.txt,
structured data and no noindex.We check the address you enter and don't store it. Our crawler requests use each crawler's published name but come from our server, so a site that only blocks fake crawlers may still let the real ones in. How vagejs checks your own site
What does it check?
- robots.txt rules for each crawler, read the way crawlers read them: the group that names the crawler, otherwise the
*group, with the longest matching rule winning. A server error on robots.txt counts as "stay away", as the robots.txt standard says. - What the server does. We request the page with each crawler's published user agent and report errors, Cloudflare blocks and challenge pages.
- noindex, in a
<meta name="robots">tag or anX-Robots-Tagheader, which keeps a page out of search results. - Helpful files: robots.txt, a sitemap (at
/sitemap.xmlor listed in robots.txt), llms.txt, and structured data (JSON-LD) on the page.
The crawler list and the robots.txt rules are the same ones vage verify uses on vagejs sites, with each crawler's purpose taken from its owner's documentation.
How do I read the results?
- Search engines decide whether you appear in Google and Bing at all. A block here is almost always a mistake.
- AI search crawlers find pages to show and link in AI answers, such as ChatGPT search, Claude and Perplexity. Block them and your site can't be cited there.
- AI training crawlers collect pages to train models. Blocking them is a choice that doesn't affect search results.
- AI assistants fetch a page when a person asks about it. They don't follow robots.txt, so only server blocks are shown for them.
To change what crawlers may do, the AI crawler robots.txt generator writes the rules for you.
What can't it tell you?
Our requests come from our own server. Many sites block anything that claims to be Googlebot or Bingbot but doesn't come from Google's or Microsoft's addresses, so a block shown for a search engine may only affect fake crawlers; Google Search Console and Bing Webmaster Tools show what the real ones get. A passing check also doesn't mean a page will be indexed, ranked or cited. It means nothing obvious is in the way.
Questions
Do you store the addresses I check?
No. The address is sent to our checker in the request body, used for the check and not saved or logged.
Why did the check fail on a big website?
Large sites often block unknown tools, including this one. When that happens the checker says so, and the result tells you nothing about what real crawlers get.
How do I keep my own site visible?
Allow the crawlers you want in robots.txt, keep a sitemap, and don't block them at your host or CDN. vagejs writes robots.txt, the sitemap and llms.txt from your pages, and vage verify runs these same crawler checks on your live site. See SEO and AI search. For the whole process, step by step, read how to make your website visible in AI search.
Start with one command.
You'll have a site running in a minute.
npm create vagejs@latest my-site