How to make your website visible in AI search
Let AI search read your site, then check that it can.

Each step below stands on its own, so you can jump to the one you need. They work for any website, and the vagejs parts show how a vagejs site handles each one for you.

1. Check what crawlers can reach today

Start by finding out where you stand. Enter your address in our free AI visibility checker. It requests your home page as Googlebot, Bingbot and the crawlers of OpenAI, Anthropic and Perplexity, and reports:

• which crawlers your robots.txt lets in, and which it shuts out,
• server errors, Cloudflare blocks and challenge pages,
• a noindex tag or header, and
• whether you have a sitemap, llms.txt and structured data.

A block on a search engine or an AI search crawler is the thing to fix first. A block on an AI training crawler may be exactly what you chose.

2. Make sure your words are in the HTML

Many AI crawlers read your HTML as it arrives and don't run JavaScript. If your site builds its text in the browser, they see a nearly empty page.

To test a page, open it, choose "View page source" in your browser and search for a sentence you can see on screen. In a terminal, the same test is:

curl -s https://example.com/about/ | grep -i "a sentence from the page"

No match means the text arrives later, from JavaScript. vagejs builds every page into plain HTML before anyone visits and sends no JavaScript unless you add some, so this test passes on every page of a vagejs site.

3. Allow AI search crawlers in robots.txt

AI companies use separate crawlers for search and for training. The search crawlers find pages to show and link in AI answers:

Training crawlers such as GPTBot and ClaudeBot collect pages to train models. Blocking them doesn't affect AI search. Google-Extended controls whether Gemini may use your pages, and blocking it doesn't affect Google Search. Google says you don't need special AI files to appear in its own AI features; what helps there is the usual: crawlable pages, useful content and a working sitemap.

Read your current file with:

curl -s https://example.com/robots.txt

Look for Disallow: / under User-agent: * or under one of the search crawlers above. Long "block all AI" lists copied from forums often shut out the search crawlers too. To write a file that blocks training and allows search, or to choose crawler by crawler, use the AI crawler robots.txt generator. If you're still deciding which to allow, our post on why AI search can't see your website explains the three kinds of AI visitors.

On a vagejs site, robots.txt is written on every build and allows every crawler by default. To treat the two kinds differently, set both choices in vage.config.ts:

seo: {
  robots: { aiTraining: "disallow", aiSearch: "allow" },
},

robots.txt is a polite request, not a lock. The crawlers above say they follow it.

4. Remove the blocks you can't see

A perfect robots.txt doesn't help if something else turns crawlers away. Two common culprits:

• An X-Robots-Tag: noindex header sent by your host with every page. It doesn't show in the page, but it asks search engines to leave the page out. Check with curl -sI https://example.com/ | grep -i x-robots-tag; no output is good.
• Bot protection at your host or CDN (the network in front of your site). Some settings block AI bots in one click, or answer them with a "Checking your browser..." page instead of your content. You fix these in your host's or CDN's dashboard.

The AI visibility checker reports both. Its requests come from our server, not from the crawler companies' own addresses, so a block shown for Googlebot or Bingbot may only affect fake crawlers. Google Search Console and Bing Webmaster Tools show what the real ones get.

5. Publish a sitemap and a clear summary for every page

A sitemap tells crawlers which pages exist. A unique title and description on each page tells them, and the people reading search results, what each page is about.

vagejs writes sitemap.xml from your published pages on every build, leaving out drafts and noindex pages, and vage check reports missing or duplicate titles and descriptions, broken links and invalid structured data before you publish:

npx vage check

If your site isn't in regular Google results either, start there: our post on why your AI-built website isn't showing up on Google covers the usual causes. For Google, add your sitemap in Search Console.

6. Add llms.txt if you like

llms.txt is a short Markdown guide to your site for AI tools: your site's name, a summary and links to the pages that matter. It's a proposal, not a standard. Some AI tools read it, no search engine requires it, and it won't change your Google rankings. It costs little to publish.

To write one by hand, use the llms.txt generator and save the result at the root of your site. vagejs writes llms.txt from your published pages on every build, so it never goes out of date. Two optional settings add more for AI tools:

seo: {
  llmsTxt: { full: true }, // also writes llms-full.txt with the full text of every listed page
  markdownCopies: true,    // a plain Markdown copy of each page, such as /docs/cli.md
},

7. Test the live site after every deploy

Your files can be right and your host can still say no. After you deploy a vagejs site, run:

npx vage verify

It compares the live site with your files and fetches your home page as 10 search and AI crawlers: Googlebot, Bingbot, OAI-SearchBot, GPTBot, ChatGPT-User, Claude-SearchBot, ClaudeBot, Claude-User, PerplexityBot and Perplexity-User. It reports block and challenge pages, robots.txt rules that shut a crawler out of the whole site, noindex headers, and http:// or www. addresses that don't redirect to your real one. A rule that blocks Googlebot or Bingbot is an error; a blocked training crawler is a warning that says it's fine if that was your choice.

New vagejs sites include a GitHub Actions workflow that runs vage verify every day once you set the VAGE_VERIFY_URL variable, so a host setting that changes quietly gets noticed. See deployment for setup.

For a site that isn't built with vagejs, run the AI visibility checker again after each change to your host or robots.txt.

8. Write pages worth citing

Once crawlers can read your site, what they show depends on the page. Tooling removes mistakes; it can't make a page worth quoting. Pages that answer a real question clearly, show who wrote them and are kept up to date give AI search, and people, a reason to pick them. Put the answer near the top, use headings that say what each section covers, and keep facts such as prices, hours and contact details on your own pages where crawlers can read them.

What to expect

• Fast: fixing a robots.txt rule, a noindex header or a host bot setting takes minutes, and the checker or vage verify shows the result straight away.
• Slow: crawlers come back on their own schedule, so it can take a while before AI answers reflect a change.
• Not in your control: whether an AI tool mentions or links your site. Each company decides what its tools show, and those decisions change. Being readable is the step everything else depends on, not a promise of citations.
• Not tracked: vagejs doesn't watch what AI assistants say about you, and the checker only tests your home page.

A clean vage check says the same on every run: passing checks don't guarantee indexing, rankings, rich results or AI citations.

Related

• SEO and AI search: every file vagejs writes for search engines and AI tools, and the settings for each.
• AI visibility checker, AI crawler robots.txt generator and llms.txt generator: the free tools used in this guide.
• AGENTS.md generator: if an AI coding assistant builds your site, give it written rules, such as asking before it deploys.
• Why ChatGPT and Perplexity can't see your website and why your AI-built website isn't showing up on Google: the problems behind these steps, in more depth.
• Features: everything vagejs does, including the files search engines and AI assistants read.