@rsc-kit/mcp/guides/seo-files.md

An MCP server over what an rsc-kit build decided: the routes, why each one is static or not, and what it costs the browser.

# robots, sitemap and llms.txt

> The files a site describes itself with, from a file beside the root layout — written the way Next writes them, stored at build when they can be.

A crawler asks for `/robots.txt` and `/sitemap.xml` before it reads a page,
and a model asks for `/llms.txt`. Each comes from a file beside the root
layout, named for what it answers:

| file | answers | returns |
| --- | --- | --- |
| `src/app/robots.ts` | `/robots.txt` | `MetadataRoute.Robots` |
| `src/app/sitemap.ts` | `/sitemap.xml` | `MetadataRoute.Sitemap` |
| `src/app/llms.ts` | `/llms.txt` | `MetadataRoute.Llms` |
| `src/app/llms-full.ts` | `/llms-full.txt` | a string |

The same names and shapes as Next, so a port copies them across unchanged.

```ts title="src/app/robots.ts"
import type { MetadataRoute } from '@rsc-kit/core/metadata';

export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      { userAgent: '*', allow: '/', disallow: ['/api/', '/studio'] },
      { userAgent: 'GPTBot', disallow: '/' },
    ],
    sitemap: '/sitemap.xml',
  };
}
```

```ts title="src/app/sitemap.ts"
import type { MetadataRoute } from '@rsc-kit/core/metadata';
import { db } from '@/lib/db';

export default async function sitemap(): Promise<MetadataRoute.Sitemap> {
  const posts = await db.post.findMany({ select: { slug: true, updatedAt: true } });

  return [
    { url: '/', changeFrequency: 'weekly', priority: 1 },
    { url: '/pricing' },
    ...posts.map((post) => ({ url: `/blog/${post.slug}`, lastModified: post.updatedAt })),
  ];
}
```

```ts title="src/app/llms.ts"
import type { MetadataRoute } from '@rsc-kit/core/metadata';

export default function llms(): MetadataRoute.Llms {
  return {
    title: 'Example',
    summary: 'What the site is, in one sentence.',
    sections: [
      { title: 'Pages', links: [{ title: 'Pricing', url: '/pricing', description: 'Per restoration, no subscription' }] },
    ],
  };
}
```

A relative url in any of them is made absolute with the root layout's
`metadataBase`; without one, a relative url is a build error that says so.
Any of the three may return a string instead, served as written.

## A sitemap you do not write

With no `sitemap.ts`, the build writes `/sitemap.xml` itself, from what it
already knows: every page it stored and every url `generateStaticParams`
listed, each with the build as `lastModified`. Left out: anything under a
`middleware.ts` (a guard means not for everyone), a page the build could not
render, and the not-found page. The build's table says so:

```
○  /sitemap.xml
   written by the build: 5 urls; a sitemap.ts beside the root layout replaces it
```

It needs the root layout's `metadataBase` for the host; without one the line
says it was not written. A `sitemap.ts` replaces it entirely — the moment you
want a page the build cannot see (a post per database row without
`generateStaticParams`) or a `changeFrequency`, write one.

## How fresh

Three choices, and the function decides — the same rule every route follows:

| you write | served | fresh |
| --- | --- | --- |
| nothing | the build's own sitemap, stored | every deploy |
| a `sitemap.ts` that reads the database | stored at build | every deploy |
| a `sitemap.ts` that awaits `connection()` | rendered per request | every crawl |

```ts title="src/app/sitemap.ts"
import { connection } from '@rsc-kit/core/request';

export default async function sitemap(): Promise<MetadataRoute.Sitemap> {
  await connection(); // per request — the same mark a page uses

  const posts = await db.post.findMany({ select: { slug: true, updatedAt: true } });

  return posts.map((post) => ({ url: `/blog/${post.slug}`, lastModified: post.updatedAt }));
}
```

Without the `connection()` line the same function runs once at build and the
answer is stored; the build's table shows `○` for a stored one and `ƒ` for
one that runs per request. Reading the database at build is fine — it is the
request that makes a route dynamic, not the data.

## How they are served

Each file becomes an api route, and that decides the rest. A `sitemap.ts`
that reads nothing per request — the database counts as nothing, the request
does not — is answered once at build and stored, like any frozen route, and
served from the file after. One that reads `cookies()` or awaits
`connection()` stays dynamic and runs per request. The build's table lists
them with the other routes and says which.

They run no middleware, on purpose: a guard on the root layout's directory
would otherwise answer a crawler's request for `robots.txt` with a 401.

The urls are typed like every other route, so `route('/sitemap.xml')` is a
link the build checks.

## Files as written

A hand-written file beside the root layout is served at the root as it is:
`robots.txt`, `sitemap.xml` (or `sitemap-posts.xml`), `llms.txt`,
`llms-full.txt`, and the others a site is asked for there — `humans.txt`,
`security.txt`, `ads.txt`. In development they are read from `app/`; a build
copies them beside the client output. A file and a function for the same url
is a build error naming both.

## Coming from Next

`app/robots.ts` and `app/sitemap.ts` carry across unchanged. `app/llms.txt`
in Next is usually a `route.ts`; here it is `llms.ts` with a shape, or a file
as written. Next's `generateSitemaps()` for a sitemap split across files is
not here: a `sitemap-posts.xml` written by hand, or a `route.ts` under
`app/sitemap/`, covers it.