# SEO and AI search

> What vagejs writes for search engines and AI assistants on every build (meta tags, structured data, sitemap, robots.txt, llms.txt) and the choices that stay yours.

Search engines and AI assistants both need the same basics: a clear title and summary for each page, an honest address, and a map of the site. vagejs writes these on every build, so you don't have to remember them, and `vage check` stops the build when one goes missing.

## On every page

From the `title` and `description` in a page's frontmatter, vagejs adds:

| Tag | Why it matters |
| --- | --- |
| `<title>` and `meta description` | What people see in search results |
| Canonical URL | Tells search engines which address is the real one |
| Open Graph tags (`og:title`, `og:description`, `og:url`, `og:image`, `og:site_name`) | The preview card when someone shares the link |
| `twitter:card` | A large image card when the page has an `image` |
| `robots: noindex` | Only on pages where you set `noindex: true` |

If your layout already writes one of these, vagejs leaves yours alone.

## Structured data (JSON-LD)

Structured data describes a page in a format search engines read directly. vagejs adds `WebSite` data to the home page and `Article` data to posts with `type: article`, including the author and date when the page lists them. `Organization` data is opt-in, because only you know the facts:

```ts
seo: {
  structuredData: {
    organization: { name: "Acme Studio", url: "https://acme.example", logo: "/logo.png" },
  },
},
```

`vage check` reports structured data that is broken or doesn't match the page.

## Sitemap and robots.txt

`sitemap.xml` lists every published page and leaves out drafts and `noindex` pages. `robots.txt` allows every crawler by default and points to the sitemap. Both need `site` in `vage.config.ts`.

## AI crawlers

Some AI companies use separate crawlers for training their models and for answering questions in AI search. You can treat them differently:

```ts
seo: {
  robots: { aiTraining: "disallow", aiSearch: "allow" },
},
```

-   `aiTraining` covers GPTBot, ClaudeBot and Google-Extended.
-   `aiSearch` covers OAI-SearchBot and Claude-SearchBot.

Nothing changes until you set them. The crawler names come from each company's own documentation, and `npx vage guidelines` shows the source and the date it was last checked. Keep in mind that `robots.txt` is a polite request: well-behaved crawlers follow it, but it doesn't lock anything.

## llms.txt

`llms.txt` is a short, plain-text guide to your site for AI tools: your site's name, its description and a list of pages with their summaries. vagejs writes it on every build. Turn on `llmsTxt: { full: true }` to also write `llms-full.txt`, which contains the full text of every listed page.

> llms.txt is a proposed convention, not a search-engine standard. It costs nothing to publish, and some AI tools read it, but don't expect it to change your Google rankings.

## Telling search engines about changes

After you publish, `npx vage indexnow --changed-since HEAD~1` tells IndexNow search engines (such as Bing) which pages changed. It asks before sending anything. For Google, add your sitemap in Search Console. `vage connect gsc` and `vage report search` then bring your search data into the terminal.

## What vagejs can't do for you

Good tooling removes mistakes. It can't make a page worth ranking. Search engines and AI assistants still reward pages that answer a real question clearly, show who wrote them, and are kept up to date. Write for the person reading, and let vagejs handle the details.