@awesomate/hosting-mcp/skill/awesomate-seo/references/ai-discoverability.md

Awesomate MCP server — lets Claude manage your Awesomate WordPress hosting, plan, limits, n8n automations, and build Node/static apps + databases

# AI discoverability — crawlers, rendering, llms.txt

How AI assistants (ChatGPT, Claude, Perplexity, Gemini, Copilot) find and cite
a site — and the two silent failure modes that make a "perfectly SEO'd" site
invisible to them anyway. Crawler/rendering tables adapted from
[claude-blog](https://github.com/AgriciDaniel/claude-blog) (MIT © AgriciDaniel).

## Google needs nothing special — say so when asked

Google's official guidance
(https://developers.google.com/search/docs/fundamentals/ai-optimization-guide):
AI Overviews and AI Mode run on Google's normal Search ranking systems. No
special markup, no AI schema, no llms.txt — **Google Search ignores llms.txt
entirely**. Normal SEO (the main checklist) *is* the Google-AI strategy. If
the user asks about "GEO"/"AEO" tactics or paid AI-optimization services,
this is the honest answer: for Google it's just SEO; for other assistants
the extras below help.

llms.txt still earns its place for **non-Google** assistants — keep writing
it, but present it as that, never as a Google requirement.

## Three kinds of AI bot — three different decisions

| Kind | Bots | Blocking means |
|---|---|---|
| Search-indexing | OAI-SearchBot, Claude-SearchBot, PerplexityBot, **Bingbot** | The site can't be cited in that assistant's answers. **Allow these.** Bingbot belongs here because Copilot and several assistants ground on Bing's index. |
| Training | GPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extended, Meta-ExternalAgent | Owner's policy choice about model training; doesn't remove the site from answers |
| User-triggered retrieval | ChatGPT-User, Perplexity-User, Claude-User | Fetches when a live user asks; may not fully respect robots.txt |

The default `User-agent: * / Allow: /` robots.txt already allows all of them —
no explicit allowlist needed. Explicit stanzas only matter when the owner
wants to opt out of **training** while staying visible in answers:

```
# Stay citable, opt out of model training
User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: *
Allow: /
Sitemap: https://DOMAIN/sitemap.xml
```

Never block the search-indexing bots "for safety" — that silently removes the
site from ChatGPT/Claude/Perplexity answers.

Note for Awesomate-managed domains: the CDN prepends a managed section to the
site's robots.txt (a Content Signals preamble plus disallows for the
*training*-bot family). Search-indexing bots stay allowed, so the site remains
citable — but the served robots.txt is the site file *plus* that preamble.
Always check the live `https://DOMAIN/robots.txt`, not just the file you wrote.

## robots.txt allows it — but the CDN can still 403 it

Cloudflare can block AI crawlers at the edge, and steers new zones into doing
so. The crawler gets a 403 or a challenge page before it ever reads robots.txt,
so a perfect robots.txt proves nothing. Always verify from the outside, with the
**retrieval** bots — the ones whose access decides whether the site is cited:

```bash
for ua in \
  "Mozilla/5.0 (compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot)" \
  "Mozilla/5.0 (compatible; Claude-SearchBot/1.0; +https://anthropic.com)" \
  "Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)" \
  "Mozilla/5.0 (compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)" \
  "Mozilla/5.0 (compatible; ChatGPT-User/1.0; +https://openai.com/bot)" \
  "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"; do
  printf '%s  %s\n' "$(curl -s -o /dev/null -w '%{http_code}' -A "$ua" https://DOMAIN/)" "$ua"
done
```

200 = reachable. 403, 503, or an HTML page mentioning Cloudflare = blocked at
the edge.

**Training bots (GPTBot, ClaudeBot, CCBot, Bytespider, Amazonbot…) are a
different question.** On Awesomate-managed domains (`*.awesomate.site`,
`*.awesomate.app`) they get `403 Your request was blocked.` **by design** — that
is policy, not a fault, and it does not affect citation. Don't report it and
don't ask for it to be lifted. On the user's own domain, letting training bots
in is the user's choice.

Where to fix a blocked **retrieval** bot:

- **Awesomate-managed domain**: the zone is managed by Awesomate — the user
  can't change it. Raise it via `awesomate_support`.
- **Custom domain on the user's own Cloudflare account**: in **AI Crawl
  Control**, set each blocked retrieval bot (OAI-SearchBot, Claude-SearchBot,
  PerplexityBot, Perplexity-User, ChatGPT-User) to **Allow**; training bots are
  their call. If Cloudflare's **managed robots.txt** is on, check the live
  `/robots.txt` doesn't disallow the retrieval bots either. Don't write a WAF
  skip on `cf.verified_bot_category "AI Crawler"`: that category is the
  training crawlers, and PerplexityBot isn't a verified bot (since Aug 2025),
  so a category rule can't match it anyway.

Verified 2026-09-02 across awesomate.ai, hub.awesomate.ai and eight live client
sites: all four retrieval bots got **200**. **That is now stale:** on
08/10/2026 PerplexityBot was found getting 403 on Awesomate-managed zones
(OAI-SearchBot, Claude-SearchBot, Perplexity-User, ChatGPT-User and Bingbot were
fine). It is a known Cloudflare fault being fixed — PerplexityBot should get
200; if a managed site still shows 403 for it, raise it with support rather
than treating it as expected. Re-check per site rather than assuming either way.

## Most AI crawlers do not run JavaScript

**Googlebot, Gemini and Applebot render JS. GPTBot, ClaudeBot, PerplexityBot,
OAI-SearchBot and Claude-SearchBot do not** — they read the **raw HTML
response**. So a client-rendered app (React/Vue/Next SPA without SSR) is
invisible to ChatGPT, Claude and Perplexity regardless of its meta tags,
schema or llms.txt.

`awesomate_site_audit` checks this automatically (gate G1) — run it rather
than eyeballing the page in a browser, which renders the JS and hides the
problem.

Red flags — check the raw response, not the browser:

```bash
curl -s https://DOMAIN/page | head -100
```

- Body is essentially an empty `<div id="root">` / `<div id="app">` /
  `<div id="__next">`
- Full HTML under ~5KB for a content page
- The content exists only inside `<script>` tags as JSON

Fix: keep public, indexable content in static or server-rendered HTML — for
app projects, static marketing/landing pages in front of the app is usually
enough. Don't rebuild a working app over this, but never claim AI
discoverability for client-rendered content.

## The accidental opt-out — check before blaming anything else

A page can be perfectly built, reachable and fast, and still be barred from
every AI answer by one directive. Sweep for these before investigating
anything subtler:

- `<meta name="robots" content="noindex">` — out of search AND AI answers
- `nosnippet` or `max-snippet:0` — nothing may be quoted, which is the same as
  being uncitable even though the page still ranks
- an `X-Robots-Tag` response **header** carrying any of the above (easy to miss:
  it is not in the HTML, so viewing source shows nothing)
- a staging `robots.txt` that shipped to production with `Disallow: /`
- in Search Console, the Search generative AI control left off

The usual causes are an SEO plugin default, a staging file promoted by mistake,
or a "private launch" setting nobody turned back off. `awesomate_site_audit`
covers the meta and header cases (gate G3); the Search Console control has to be
checked by the owner in their own account.

Note that blocking `Google-Extended` does **not** remove a site from AI
Overviews or AI Mode — that control is about training, not answers. Don't
"fix" it expecting a visibility change.

## llms.txt rules

- Optional inventory for non-Google assistants — Google ignores it.
- Under 10KB (assistants may truncate or skip larger files).
- Markdown links: `[Title](URL): one-line description`. Only real,
  high-value pages; only true facts.
- Supplements sitemap.xml, never replaces it. Update both when content changes.