skills-ws
Version:
85 agent skills for AI coding assistants — marketing, growth, web3, dev, design & operations. Built for OpenClaw, Claude Code, Cursor, and Codex.
1,137 lines (918 loc) • 45.3 kB
Markdown
name: programmatic-seo
description: "Build template-driven SEO pages at scale (Next.js/Astro) that survive Google's scaled-content-abuse and site-reputation-abuse policies: page patterns, data pipelines, canonical/index management, schema, indexing monitoring. Use when user mentions programmatic SEO, pSEO, or directory/location/comparison/integration pages at scale."
# Programmatic SEO — Build Thousands of High-Quality Pages at Scale
## Core Philosophy
Programmatic SEO is NOT "spin 10,000 thin pages and pray." It's building genuinely useful pages where **the combination of data creates unique value**. Every page must answer a question someone is actually asking.
**The Golden Rule:** If you removed the template chrome and just looked at the data, would the page still be useful? If not, don't build it.
### What Google actually enforces (as of Jun 2026)
pSEO is not banned — *low-value scaled content* is. Know the live policies, because they decide whether your project ships or gets deindexed. Verify current wording at the Search Central spam policies page (`developers.google.com/search/docs/essentials/spam-policies`):
- **Scaled content abuse** (March 2024 core/spam update, since folded into standing policy): "generating many pages... primarily to manipulate ranking and not help users." The trigger is *intent + lack of unique value*, not the method or the use of AI. AI-assisted generation is allowed; AI-assisted *thin spam at scale* is not.
- **Site reputation abuse** ("parasite SEO", enforced from May 2024, tightened since): hosting third-party content with little oversight to exploit a host's ranking signals. Relevant if you let partners/users publish templated pages on your domain — you own quality control.
- **Expired-domain abuse**: don't buy aged domains to fast-track a pSEO farm.
- **Helpful Content signals** are now baked into the core ranking system (not a separate update). "People-first content," demonstrated **E-E-A-T** (Experience, Expertise, Authoritativeness, Trust), and genuine first-hand data are the durable wins. Pure aggregation with no added value is fragile.
Practical read: every template must add information a user can't trivially get elsewhere (proprietary data, computed insight, real reviews, fresh pricing). "AI wrote unique-sounding paragraphs over the same three facts" is exactly what the scaled-content policy targets.
## 1. Page Pattern Playbook
### 1.1 Location Pages — "[Service] in [City]"
**When to use:** Local services, marketplaces, delivery, real estate, jobs.
**Data you need per location:**
- Population, demographics, cost of living
- Local competitors / providers
- Geo-specific stats (median home price, avg salary, weather)
- Real reviews or testimonials from that area
- Local regulations or requirements
**URL structure:**
```
/plumbers/austin-tx
/plumbers/austin-tx/drain-cleaning
```
**Quality signals to include:**
- Map embed or service area polygon
- Local phone number or office address
- Area-specific pricing ("Average drain cleaning in Austin: $150–$280")
- Nearby areas linked ("Also serving: Round Rock, Cedar Park, Georgetown")
**Next.js implementation:**
> **Next.js version note (App Router, Next 15+ → mid-2026).** Since Next 15, `params` and `searchParams` are **async** — they are `Promise`s you must `await`. The pre-15 synchronous shape (`params: { service: string }`) no longer type-checks. Examples below use the async form. On Next 14 these were synchronous; if you must support 14, drop the `Promise<>` wrapper and the `await`. A `Metadata` return type from `next` is also recommended for `generateMetadata`.
```tsx
// app/[service]/[location]/page.tsx
import { notFound } from 'next/navigation';
import type { Metadata } from 'next';
import { getLocationData, getServiceData } from '@/lib/data';
import { generateLocationSchema } from '@/lib/schema';
// params is async in Next 15+ — type it as a Promise and await it.
type Params = Promise<{ service: string; location: string }>;
// Pre-render only validated combos (see §6). Other slugs render on-demand
// because dynamicParams defaults to true; we noindex/404 invalid ones there.
export async function generateStaticParams() {
const combos = await getServiceLocationCombos();
return combos.map(({ service, location }) => ({
service: service.slug,
location: location.slug,
}));
}
// On-demand rendering for slugs not in generateStaticParams, revalidated daily.
export const revalidate = 86400;
export async function generateMetadata(
{ params }: { params: Params },
): Promise<Metadata> {
const { service: serviceSlug, location: locationSlug } = await params;
const location = await getLocationData(locationSlug);
const service = await getServiceData(serviceSlug);
if (!location || !service) return {};
return {
title: `${service.name} in ${location.city}, ${location.state} — Top ${location.providerCount}+ Providers`,
description: `Find trusted ${service.name.toLowerCase()} in ${location.city}. Compare ${location.providerCount} local pros, read ${location.reviewCount} reviews, and get free quotes.`,
alternates: {
canonical: `/${serviceSlug}/${locationSlug}`,
},
};
}
export default async function LocationPage({ params }: { params: Params }) {
const { service: serviceSlug, location: locationSlug } = await params;
const location = await getLocationData(locationSlug);
const service = await getServiceData(serviceSlug);
if (!location || !service) notFound();
const providers = await getProviders(service.id, location.id);
const stats = await getLocalStats(service.id, location.id);
const faqs = generateLocalFAQs(service, location, stats);
const nearbyLocations = await getNearbyLocations(location.id, service.id);
return (
<>
<script
type="application/ld+json"
dangerouslySetInnerHTML={{
__html: JSON.stringify(generateLocationSchema(service, location, providers, faqs)),
}}
/>
<h1>{service.name} in {location.city}, {location.state}</h1>
{/* Unique local context — NOT just the template */}
<LocalStatsBar stats={stats} city={location.city} />
{/* Provider listings with real data */}
<ProviderGrid providers={providers} />
{/* Area-specific pricing data */}
<PricingTable service={service} location={location} stats={stats} />
{/* Genuine FAQ with local answers */}
<FAQSection faqs={faqs} />
{/* Internal linking to nearby areas */}
<NearbyAreas locations={nearbyLocations} service={service} />
{/* Internal linking to related services */}
<RelatedServices location={location} currentService={service} />
</>
);
}
```
### 1.2 Comparison Pages — "[Product A] vs [Product B]"
**When to use:** SaaS directories, review sites, marketplaces.
**URL structure:**
```
/compare/notion-vs-coda
/compare/slack-vs-teams-vs-discord (three-way)
```
**Critical: avoid thin comparisons.** Every comparison page needs:
- Feature-by-feature breakdown with actual data
- Pricing comparison (current, verified)
- Use-case recommendations ("Best for X: Product A. Best for Y: Product B.")
- Unique pros/cons per product
- User sentiment data (review aggregates, NPS if available)
```tsx
// app/compare/[slug]/page.tsx
import type { Metadata } from 'next';
type Params = Promise<{ slug: string }>;
export async function generateStaticParams() {
const comparisons = await getPopularComparisons();
// Only generate pages for combinations with search volume
return comparisons
.filter(c => c.monthlySearchVolume > 50)
.map(c => ({ slug: c.slug }));
}
// Generate bidirectional — "A vs B" and "B vs A" both resolve and stay
// crawlable; canonical consolidates ranking to the higher-volume variant.
export async function generateMetadata(
{ params }: { params: Params },
): Promise<Metadata> {
const { slug } = await params;
const comparison = await getComparison(slug);
const canonical = comparison.searchVolume.aVsB > comparison.searchVolume.bVsA
? `${comparison.productA.slug}-vs-${comparison.productB.slug}`
: `${comparison.productB.slug}-vs-${comparison.productA.slug}`;
return {
title: `${comparison.productA.name} vs ${comparison.productB.name} (${new Date().getFullYear()}) — Features, Pricing, Verdict`,
alternates: { canonical: `/compare/${canonical}` },
};
}
```
### 1.3 Integration Pages — "[Your Product] + [Integration]"
**When to use:** SaaS products with integrations, API platforms, automation tools.
**URL structure:**
```
/integrations/salesforce
/integrations/salesforce/setup-guide
```
**Unique value per page:**
- What specific data syncs between products
- Step-by-step setup with screenshots
- Use-case examples ("When a deal closes in Salesforce, automatically create an invoice in [Your Product]")
- Limitations and workarounds
- Pricing impact (does this integration require a specific plan?)
### 1.4 "X for Y" Pages — "[Tool/Concept] for [Audience]"
**When to use:** Products serving multiple verticals or personas.
**URL structure:**
```
/solutions/project-management-for-agencies
/solutions/crm-for-real-estate
```
**Each page needs:**
- Industry-specific pain points (not generic)
- Tailored feature highlights (same features, different framing)
- Social proof from that vertical (logos, quotes, case studies)
- Industry-specific terminology and workflows
- Compliance or regulatory callouts relevant to that vertical
### 1.5 Directory / Listing Pages
**URL structure:**
```
/tools/email-marketing (category)
/tools/email-marketing/mailchimp (individual listing)
```
**Aggregation pages (category level) must include:**
- Curated top picks with brief rationale
- Filterable/sortable table or grid
- Quick comparison of top 3–5
- Last-updated date (freshness signal)
## 2. Data Source Strategies
### 2.1 APIs (Best for Fresh Data)
```typescript
// lib/data-sources/api.ts
import pThrottle from 'p-throttle';
// Always throttle API calls during build
const throttle = pThrottle({ limit: 5, interval: 1000 });
const fetchWithRetry = throttle(async (url: string, retries = 3): Promise<any> => {
for (let attempt = 0; attempt < retries; attempt++) {
try {
const res = await fetch(url, {
headers: { Authorization: `Bearer ${process.env.API_KEY}` },
next: { revalidate: 86400 }, // ISR: rebuild daily
});
if (!res.ok) throw new Error(`${res.status}: ${res.statusText}`);
return res.json();
} catch (e) {
if (attempt === retries - 1) throw e;
await new Promise(r => setTimeout(r, 2 ** attempt * 1000));
}
}
});
// Cache API results to avoid hammering during build/dev.
// `unstable_cache` still ships in Next 15/16 (note the `unstable_` prefix) and is
// fine to use today. The forward-looking replacement is the `'use cache'`
// directive + cacheLife/cacheTag (see below). Pick ONE and be consistent.
import { unstable_cache } from 'next/cache';
export const getProductData = unstable_cache(
async (productSlug: string) => {
const data = await fetchWithRetry(`https://api.example.com/products/${productSlug}`);
return transformProductData(data);
},
['product-data'],
{ revalidate: 86400, tags: ['products'] }
);
```
**Modern alternative — `'use cache'` (Next 15.x stable / 16):** the new cache model is opt-in via the `cacheComponents` flag in `next.config.js` (this flag was named `dynamicIO` in earlier 15.x canaries — check your version). Inside a cached scope you set freshness with `cacheLife` and invalidation keys with `cacheTag`. Verify the directive's stability for your exact version at `nextjs.org/docs`.
```typescript
import { cacheLife, cacheTag } from 'next/cache';
export async function getProductData(productSlug: string) {
'use cache';
cacheLife('days'); // preset: seconds|minutes|hours|days|weeks|max, or { stale, revalidate }
cacheTag(`product-${productSlug}`); // revalidateTag(`product-${slug}`) busts just this entry
const data = await fetchWithRetry(`https://api.example.com/products/${productSlug}`);
return transformProductData(data);
}
```
### 2.2 Database (Best for Scale + Control)
```typescript
// lib/data-sources/db.ts
import { prisma } from '@/lib/prisma';
export async function getLocationData(slug: string) {
return prisma.location.findUnique({
where: { slug },
include: {
stats: true,
providers: { where: { active: true }, orderBy: { rating: 'desc' }, take: 20 },
nearbyLocations: { take: 8 },
},
});
}
// For generateStaticParams — paginate to avoid memory issues
export async function* getAllLocationSlugs() {
let cursor: string | undefined;
while (true) {
const batch = await prisma.location.findMany({
select: { slug: true },
take: 1000,
...(cursor ? { skip: 1, cursor: { slug: cursor } } : {}),
orderBy: { slug: 'asc' },
});
if (batch.length === 0) break;
for (const item of batch) yield item.slug;
cursor = batch[batch.length - 1].slug;
}
}
```
### 2.3 Scraping + Enrichment Pipeline
> **Scraping compliance checklist — do this before writing the scraper.** Scraping for *commercial republishing* (which pSEO is) carries legal and contractual risk; the naive "open page, grab pricing" loop is a liability. Treat this as the minimum bar:
> 1. **Prefer an official API or licensed feed.** Almost every "scrape pricing" task has an API, partner feed, or affiliate data export that is faster, cleaner, and contractually safe. Scrape only as a last resort.
> 2. **Honor `robots.txt`.** Fetch and parse it per origin; skip disallowed paths. Robots is not a law, but ignoring it is the first thing cited against you.
> 3. **Read the Terms of Service.** Many sites' ToS forbid scraping and especially *republishing* their data. Republishing facts you scraped can implicate copyright, database rights (EU `sui generis`), and unfair-competition claims. **Get legal review before commercial reuse**, and prefer attribution + linking back.
> 4. **Identify yourself.** Set a descriptive `User-Agent` with a contact URL (`MyBot/1.0 (+https://example.com/bot)`). No spoofing real browsers to evade blocks.
> 5. **Rate-limit per domain** and add jittered exponential backoff; back off hard on `429`/`503`. Never run unbounded concurrency against one host.
> 6. **Record provenance.** Store `sourceUrl` + `fetchedAt` for every scraped value so you can show "as of <date>", expire stale data, and audit disputes.
> 7. **Cache politely.** Re-fetch on a schedule (e.g. weekly), not on every build. Conditional requests (ETag/If-Modified-Since) save everyone bandwidth.
```typescript
// scripts/enrich-data.ts
// Run as a scheduled job (cron), NOT at build time. tsx scripts/enrich-data.ts
import { chromium, type Browser, type Page } from 'playwright';
import pThrottle from 'p-throttle';
import robotsParser from 'robots-parser';
import { prisma } from '@/lib/prisma';
const USER_AGENT =
'MyCompanyEnrichBot/1.0 (+https://example.com/bot-info; bot@example.com)';
// Per-domain throttle: at most 1 request / 2s to any single host.
const throttlesByHost = new Map<string, ReturnType<typeof pThrottle>>();
function hostThrottle(host: string) {
if (!throttlesByHost.has(host)) {
throttlesByHost.set(host, pThrottle({ limit: 1, interval: 2000 }));
}
return throttlesByHost.get(host)!;
}
// Cache robots.txt per origin so we fetch it once.
const robotsByOrigin = new Map<string, Awaited<ReturnType<typeof loadRobots>>>();
async function loadRobots(origin: string) {
const robotsUrl = `${origin}/robots.txt`;
try {
const res = await fetch(robotsUrl, { headers: { 'User-Agent': USER_AGENT } });
const body = res.ok ? await res.text() : '';
return robotsParser(robotsUrl, body);
} catch {
// Fail CLOSED on robots fetch error: if we can't confirm we're allowed, skip.
return robotsParser(robotsUrl, 'User-agent: *\nDisallow: /');
}
}
async function isAllowed(url: string) {
const origin = new URL(url).origin;
if (!robotsByOrigin.has(origin)) robotsByOrigin.set(origin, await loadRobots(origin));
return robotsByOrigin.get(origin)!.isAllowed(url, USER_AGENT) ?? false;
}
async function withBackoff<T>(fn: () => Promise<T>, retries = 3): Promise<T> {
for (let attempt = 0; ; attempt++) {
try {
return await fn();
} catch (e) {
if (attempt >= retries) throw e;
const wait = 2 ** attempt * 1000 + Math.random() * 500; // jittered backoff
await new Promise(r => setTimeout(r, wait));
}
}
}
async function enrichOne(browser: Browser, product: { id: string; name: string; pricingUrl: string }) {
const url = product.pricingUrl;
if (!(await isAllowed(url))) {
console.warn(`robots.txt disallows ${url} — skipping ${product.name}`);
return;
}
const host = new URL(url).host;
await hostThrottle(host)(async () => {
let page: Page | undefined;
try {
page = await browser.newPage({ userAgent: USER_AGENT });
await withBackoff(() => page!.goto(url, { waitUntil: 'domcontentloaded', timeout: 30_000 }));
const pricing = await extractPricing(page);
await prisma.product.update({
where: { id: product.id },
// Provenance: record WHERE and WHEN, so the page can say "as of <date>".
data: { pricing, sourceUrl: url, lastEnriched: new Date() },
});
} catch (e) {
console.error(`Failed to enrich ${product.name} (${url}):`, e);
} finally {
await page?.close(); // ALWAYS close — otherwise pages leak and the run OOMs.
}
})();
}
async function enrichProductData() {
const browser = await chromium.launch();
try {
const products = await prisma.product.findMany({
where: {
OR: [
{ lastEnriched: null },
{ lastEnriched: { lt: new Date(Date.now() - 7 * 86_400_000) } }, // > 7 days old
],
},
take: 100,
});
// Sequential per host via throttle; products on different hosts still interleave.
for (const product of products) await enrichOne(browser, product);
} finally {
await browser.close();
}
}
enrichProductData().catch((e) => {
console.error(e);
process.exit(1);
});
```
### 2.4 CSV / Spreadsheet (Quick Start)
Good for prototyping. Use a CMS or database for production.
```typescript
// lib/data-sources/csv.ts
import { parse } from 'csv-parse/sync';
import { readFileSync } from 'fs';
import path from 'path';
const dataDir = path.join(process.cwd(), 'data');
export function loadLocations() {
const raw = readFileSync(path.join(dataDir, 'locations.csv'), 'utf-8');
return parse(raw, { columns: true, cast: true }) as Location[];
}
```
## 3. URL Structure Best Practices
### Rules
1. **Flat over deep.** `/plumbers/austin-tx` beats `/services/home/plumbing/us/texas/austin`.
2. **Slugs, not IDs.** `/compare/notion-vs-coda` not `/compare/12345`.
3. **Consistent separators.** Hyphens only. No underscores, no camelCase.
4. **Include geo qualifiers.** `austin-tx` not just `austin` (disambiguation).
5. **Lowercase everything.** Redirect uppercase variants.
6. **Trailing slash: pick one.** Enforce via middleware and redirect the other.
### Middleware for URL Normalization
```typescript
// middleware.ts (Next.js)
import { NextResponse } from 'next/server';
import type { NextRequest } from 'next/server';
export function middleware(request: NextRequest) {
const { pathname } = request.nextUrl;
// Force lowercase
if (pathname !== pathname.toLowerCase()) {
const url = request.nextUrl.clone();
url.pathname = pathname.toLowerCase();
return NextResponse.redirect(url, 301);
}
// Remove trailing slash (except root)
if (pathname.length > 1 && pathname.endsWith('/')) {
const url = request.nextUrl.clone();
url.pathname = pathname.slice(0, -1);
return NextResponse.redirect(url, 301);
}
return NextResponse.next();
}
```
## 4. Canonical Strategy
### Decision Matrix
| Scenario | Canonical |
|----------|-----------|
| "A vs B" and "B vs A" exist | Point both to higher search volume variant |
| Location + service page | Self-referencing canonical |
| Paginated listings (page 2+) | **Self-referencing canonical** on each page; do NOT canonical page 2+ to page 1 (see note) |
| Filtered views (`/tools?category=email`) | Canonical to unfiltered `/tools` unless filtered URL has its own search intent |
| HTTP vs HTTPS | Always HTTPS |
| www vs non-www | Pick one, redirect the other, canonical to winner |
| Duplicate content across locales | Use `hreflang`, self-referencing canonicals per locale |
**Pagination — the `rel=prev/next` myth.** Google **stopped using `rel=prev/next` as an indexing signal years ago** (announced 2019) and does not use it today. Modern pagination guidance:
- Give every page a **self-referencing canonical** (`/tools/email-marketing?page=3` → itself). Canonicalizing page 2+ to page 1 hides the items that only appear deeper, so they never get discovered or indexed.
- Make pagination **crawlable with real `<a href>` links** — not buttons that only work with JS, and not infinite scroll with no underlying URLs.
- Give each paginated page a **distinct `<title>`/meta** (e.g. append "— Page 3") so they aren't flagged as duplicates.
- Only `noindex` **truly low-value** variants (e.g. arbitrary filter/sort permutations). Keep them `follow` so equity still flows.
- `rel=prev/next` is harmless if already present (other engines may use it), but don't build new work around it.
### Implementation
```tsx
// Always set canonical in generateMetadata. params is async in Next 15+.
export async function generateMetadata(
{ params }: { params: Promise<{ service: string; location: string }> },
) {
const { service, location } = await params;
return {
alternates: {
canonical: `https://example.com/${service}/${location}`,
},
};
}
```
## 5. Internal Linking at Scale
Internal linking is the #1 lever for programmatic SEO. Do it systematically.
### Link Architecture Patterns
```
Hub Page (/plumbers)
├── Location Pages (/plumbers/austin-tx)
│ ├── links to nearby locations
│ ├── links to sub-services (/plumbers/austin-tx/drain-cleaning)
│ └── links back to hub
├── Location Pages (/plumbers/denver-co)
└── ...
```
### Automatic "Related" Links
```typescript
// lib/internal-links.ts
export async function getRelatedPages(
currentPage: { type: string; tags: string[]; locationId?: string; slug: string },
limit = 6
) {
// 1. Same type, overlapping tags (most relevant)
const byTags = await prisma.page.findMany({
where: {
type: currentPage.type,
tags: { hasSome: currentPage.tags },
slug: { not: currentPage.slug },
},
orderBy: { traffic: 'desc' },
take: limit,
});
if (byTags.length >= limit) return byTags;
// 2. Nearby locations (for location pages)
if (currentPage.locationId) {
const nearby = await prisma.page.findMany({
where: {
type: currentPage.type,
locationId: { in: await getNearbyLocationIds(currentPage.locationId) },
},
take: limit - byTags.length,
});
return [...byTags, ...nearby];
}
return byTags;
}
```
### Breadcrumbs (Every Page)
```tsx
function Breadcrumbs({ items }: { items: { label: string; href: string }[] }) {
const schema = {
'@context': 'https://schema.org',
'@type': 'BreadcrumbList',
itemListElement: items.map((item, i) => ({
'@type': 'ListItem',
position: i + 1,
name: item.label,
item: `https://example.com${item.href}`,
})),
};
return (
<>
<script type="application/ld+json" dangerouslySetInnerHTML={{ __html: JSON.stringify(schema) }} />
<nav aria-label="Breadcrumb">
<ol className="flex gap-2 text-sm text-gray-500">
{items.map((item, i) => (
<li key={item.href} className="flex items-center gap-2">
{i > 0 && <span>/</span>}
{i === items.length - 1 ? (
<span aria-current="page">{item.label}</span>
) : (
<a href={item.href}>{item.label}</a>
)}
</li>
))}
</ol>
</nav>
</>
);
}
```
## 6. Preventing Thin Content
Thin content is the #1 killer of pSEO projects. Google will deindex entire sections.
### Quality gates — measure *unique value*, not word count
Google does **not** rank by word count, and a "300+ words" rule is trivially gamed by padding. Word/character length is at best a weak proxy. Gate on signals that actually correlate with usefulness, and treat length as one minor input among several:
| Signal | What it measures | Example gate |
|--------|------------------|--------------|
| **Unique data points** | How many distinct facts this page carries that a sibling page does NOT | ≥ 5 page-specific values (price, counts, named entities) |
| **Source coverage** | Real providers/items/competitors backing the page | ≥ 3 entities with non-placeholder data |
| **Entity completeness** | Required fields populated, no "N/A" filler | 0 critical fields missing |
| **Freshness age** | How stale the underlying data is | `lastEnriched` within 30 days |
| **Duplicate similarity** | Near-duplicate body vs. other pages of the same type | shingled/MinHash similarity < 0.8 |
| **Manual spot-check** | Human review of a random sample | 20 random pages/launch sign-off |
```typescript
// lib/quality-gate.ts
interface QualityCheck {
pass: boolean;
reason?: string;
}
const THIRTY_DAYS_MS = 30 * 86_400_000;
export function qualityGate(pageData: any, pageType: string): QualityCheck {
// Generic gates that apply to every page type.
if (pageData.lastEnriched && Date.now() - +new Date(pageData.lastEnriched) > THIRTY_DAYS_MS)
return { pass: false, reason: 'Underlying data is stale (>30 days)' };
// similarityScore is precomputed against same-type pages (MinHash/shingles, 0–1).
if (typeof pageData.similarityScore === 'number' && pageData.similarityScore > 0.8)
return { pass: false, reason: 'Near-duplicate of another page (>0.8 similarity)' };
const checks: Record<string, () => QualityCheck> = {
location: () => {
if (!pageData.providers || pageData.providers.length < 3)
return { pass: false, reason: 'Fewer than 3 real providers' };
if (!pageData.stats?.avgPrice)
return { pass: false, reason: 'No pricing data' };
// Count page-SPECIFIC facts, not characters: anything that varies per location.
if (countUniqueDataPoints(pageData) < 5)
return { pass: false, reason: 'Too few location-specific data points' };
return { pass: true };
},
comparison: () => {
if (!pageData.productA?.features || !pageData.productB?.features)
return { pass: false, reason: 'Missing feature data' };
if (!pageData.productA?.pricing || !pageData.productB?.pricing)
return { pass: false, reason: 'Missing pricing data' };
// A real comparison needs differentiators, not just two spec sheets.
if (countDistinguishingFacts(pageData.productA, pageData.productB) < 5)
return { pass: false, reason: 'No meaningful differences surfaced' };
return { pass: true };
},
};
return checks[pageType]?.() ?? { pass: true };
}
// In generateStaticParams, filter out low-quality pages
export async function generateStaticParams() {
const allPages = await getAllPageData();
return allPages
.filter(p => qualityGate(p, 'location').pass)
.map(p => ({ slug: p.slug }));
}
```
### Content Enrichment Strategies
1. **Computed insights:** "Austin plumbers charge 23% less than the national average"
2. **Aggregated stats:** Review sentiment analysis, rating distributions
3. **Temporal data:** "Prices rose 12% since last year" / a real "Updated {month} {year}" derived from the data's `lastEnriched`, never a hardcoded or page-load date
4. **Cross-references:** "Compared to Denver, Austin has 2x more licensed plumbers per capita"
5. **User-generated:** Reviews, Q&A, community contributions
6. **AI-generated summaries:** Use LLMs to synthesize unique descriptions from structured data — but always fact-check against the source data
## 7. Index Management
### robots.txt — block crawl *traps*, not your content
Critical distinction: **`robots.txt` `Disallow` blocks crawling, not indexing.** A disallowed URL can still get indexed (from links) — and because Google can't fetch it, it will **never see your `canonical` or `noindex` tag** on that URL. So:
- **Use `Disallow` only for genuine infinite crawl traps** (every `sort`/`filter` permutation, calendar pickers, session-id URLs) where you never want the crawler to spend budget.
- **Do NOT blanket-block pagination** (`?page=`). Page 2+ is how crawlers discover deeper items; blocking it strands that inventory. Instead keep paginated pages crawlable and let the on-page self-canonical/`noindex` do the work (see §4).
- **To keep something out of the index, use `noindex` (meta/header) and leave it crawlable** — the opposite of `Disallow`.
```txt
User-agent: *
Allow: /
# Block genuine crawl traps (combinatorial filter/sort URLs add no unique pages)
Disallow: /*?sort=
Disallow: /*?filter=
# NOTE: do NOT add `Disallow: /*?page=` — pagination must stay crawlable so
# deeper items get discovered. Control its indexing with on-page tags instead.
# Block non-public sections (these should ALSO send noindex if ever reachable)
Disallow: /drafts/
Disallow: /preview/
Sitemap: https://example.com/sitemap-index.xml
```
### Sitemap Strategy for Large Sites
A single sitemap file is capped at **50,000 URLs / 50 MB uncompressed** — split before you hit either. (Chunking at 10k as below keeps files small and fast to regenerate.) Next 15+ also ships a native `app/sitemap.ts` exporting `MetadataRoute.Sitemap`, plus `generateSitemaps()` for sharding — prefer that for typed, framework-managed sitemaps. The hand-rolled route handlers below give you full control and work on any framework; both are valid.
```typescript
// app/sitemap-index.xml/route.ts
export async function GET() {
const pageTypes = ['locations', 'comparisons', 'integrations', 'tools'];
const sitemaps: string[] = [];
for (const type of pageTypes) {
const count = await getPageCount(type);
const chunks = Math.ceil(count / 10000);
for (let i = 0; i < chunks; i++) {
sitemaps.push(`https://example.com/sitemaps/${type}-${i}.xml`);
}
}
const xml = `<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
${sitemaps.map(url => `<sitemap><loc>${url}</loc></sitemap>`).join('\n ')}
</sitemapindex>`;
return new Response(xml, { headers: { 'Content-Type': 'application/xml' } });
}
// app/sitemaps/[type]-[chunk].xml/route.ts
// Route Handler params are async in Next 15+ — await them.
export async function GET(
_: Request,
{ params }: { params: Promise<{ type: string; chunk: string }> },
) {
const { type, chunk: chunkStr } = await params;
const chunk = parseInt(chunkStr);
const pages = await getPagesByType(type, { skip: chunk * 10000, take: 10000 });
const xml = `<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
${pages.map(p => `<url>
<loc>https://example.com${p.path}</loc>
<lastmod>${p.updatedAt.toISOString()}</lastmod>
<changefreq>${p.changefreq ?? 'weekly'}</changefreq>
</url>`).join('\n ')}
</urlset>`;
return new Response(xml, { headers: { 'Content-Type': 'application/xml' } });
}
```
### Noindex Pages That Don't Pass Quality Gates
```tsx
export async function generateMetadata({ params }: { params: Promise<{ slug: string }> }) {
const { slug } = await params;
const data = await getPageData(slug);
const quality = qualityGate(data, 'location');
// Keep follow:true so internal-link equity still flows out of a thin page.
return {
...(quality.pass ? {} : { robots: { index: false, follow: true } }),
};
}
```
## 8. Astro Implementation (Static-First)
Astro is excellent for pSEO — static by default, fast builds, great for content sites.
```astro
// src/pages/[service]/[location].astro
import Layout from '@/layouts/Base.astro';
import { getLocationData, getServiceData, getAllCombos } from '@/lib/data';
import LocalStats from '@/components/LocalStats.astro';
import ProviderGrid from '@/components/ProviderGrid.astro';
import FAQSection from '@/components/FAQSection.astro';
export async function getStaticPaths() {
const combos = await getAllCombos();
return combos
.filter(c => qualityGate(c, 'location').pass)
.map(c => ({
params: { service: c.serviceSlug, location: c.locationSlug },
props: { serviceId: c.serviceId, locationId: c.locationId },
}));
}
const { serviceId, locationId } = Astro.props;
const location = await getLocationData(locationId);
const service = await getServiceData(serviceId);
const providers = await getProviders(serviceId, locationId);
const stats = await getLocalStats(serviceId, locationId);
<Layout
title={`${service.name} in ${location.city}, ${location.state}`}
description={`Find ${service.name.toLowerCase()} in ${location.city}. ${location.providerCount}+ pros, ${location.reviewCount} reviews.`}
canonical={`/${service.slug}/${location.slug}`}
>
<h1>{service.name} in {location.city}, {location.state}</h1>
<LocalStats stats={stats} city={location.city} />
<ProviderGrid providers={providers} />
<FAQSection service={service} location={location} stats={stats} />
</Layout>
```
## 9. Build & Deploy at Scale
### Incremental Static Regeneration (Next.js)
For 100k+ pages, don't rebuild everything on every deploy.
```typescript
// In your page — only pre-render high-traffic pages
export async function generateStaticParams() {
const topPages = await getTopPages(1000);
return topPages.map(p => ({ slug: p.slug }));
}
// dynamicParams = true (default) means other slugs render on-demand
export const revalidate = 86400; // Revalidate daily
```
### Build Performance Tips
1. **Parallelize data fetching** in `generateStaticParams`
2. **Cache API responses** to disk during build
3. **Use database connection pooling** (PgBouncer or similar)
4. **Chunk builds** — deploy in batches if build times exceed CI limits
5. **Use `dynamicParams: true`** + ISR instead of pre-rendering everything
## 10. Monitoring & Dashboards
### What to Track
| Metric | Tool | Alert Threshold |
|--------|------|-----------------|
| Indexed pages | GSC → Indexing report / URL Inspection API | Drop >10% week-over-week |
| Pages submitted vs. indexed ratio | GSC Indexing report | <70% of submitted URLs indexed |
| Avg position by page type | GSC Search Analytics | Decline >5 positions |
| Crawl stats / soft 404s | GSC Crawl Stats + server logs | >50% 4xx/soft-404 in crawl |
| Thin / near-duplicate pages | Custom crawler | Quality-gate fail or similarity >0.8 (see §6) |
| Broken internal links | Screaming Frog / custom | Any internal 404 |
| Core Web Vitals (field) | CrUX / GSC | LCP >2.5s, INP >200ms, CLS >0.1 |
| Organic traffic by template | GA4 + GSC | Drop >20% month-over-month |
Note: **INP (Interaction to Next Paint) replaced FID as a Core Web Vital in March 2024** — track INP, not FID.
### GSC API Monitoring Script
Run this weekly (cron). Two things matter for correctness:
1. **Use a rolling window, never hardcoded dates.** GSC Search Analytics data lags ~2–3 days, so query "the 28 days ending 3 days ago" and compare it to the immediately prior 28 days — so the alert is *trend*, not an absolute one-off.
2. **Search Analytics ≠ index status.** A page only appears here once it has had an *impression*. It's a proxy for "indexed and ranking somewhere." For true index status, use the **URL Inspection API** (`urlInspection.index.inspect`, quota ~2,000/day) on a sample, or read the Indexing report in the GSC UI.
```typescript
// scripts/monitor-indexing.ts — run weekly via cron. tsx scripts/monitor-indexing.ts
import { google } from 'googleapis';
const SITE_URL = process.env.GSC_SITE_URL ?? 'https://example.com';
const DAY = 86_400_000;
// GSC data lags; offset the window end by `lagDays`.
function rollingWindow(endOffsetDays: number, lengthDays: number) {
const end = new Date(Date.now() - endOffsetDays * DAY);
const start = new Date(+end - (lengthDays - 1) * DAY);
const iso = (d: Date) => d.toISOString().slice(0, 10); // YYYY-MM-DD
return { startDate: iso(start), endDate: iso(end) };
}
async function queryPageCount(
sc: ReturnType<typeof google.searchconsole>,
window: { startDate: string; endDate: string },
pathRegex: string,
) {
const res = await sc.searchanalytics.query({
siteUrl: SITE_URL,
requestBody: {
...window,
dimensions: ['page'],
dimensionFilterGroups: [{
filters: [{ dimension: 'page', operator: 'includingRegex', expression: pathRegex }],
}],
rowLimit: 25000, // paginate with startRow if a template exceeds 25k URLs
},
});
return res.data.rows?.length ?? 0;
}
async function checkIndexingHealth() {
const auth = new google.auth.GoogleAuth({
keyFile: process.env.GSC_KEY_FILE ?? 'service-account.json',
scopes: ['https://www.googleapis.com/auth/webmasters.readonly'],
});
const sc = google.searchconsole({ version: 'v1', auth });
// Templates to watch, keyed by URL regex.
const templates: Record<string, string> = {
locations: '/plumbers/',
comparisons: '/compare/',
};
const current = rollingWindow(3, 28); // 28 days ending 3 days ago
const prior = rollingWindow(31, 28); // the 28 days before that
for (const [name, regex] of Object.entries(templates)) {
const [nowCount, prevCount, expected] = await Promise.all([
queryPageCount(sc, current, regex),
queryPageCount(sc, prior, regex),
getExpectedPageCount(name),
]);
const indexRatio = expected ? nowCount / expected : 0;
const wowDelta = prevCount ? (nowCount - prevCount) / prevCount : 0;
console.log(
`[${name}] ranking-visible: ${nowCount}/${expected} (${(indexRatio * 100).toFixed(1)}%), ` +
`period-over-period: ${(wowDelta * 100).toFixed(1)}%`,
);
if (indexRatio < 0.7) console.error(` ⚠ Only ${(indexRatio * 100).toFixed(1)}% of ${name} pages visible in search.`);
if (wowDelta < -0.1) console.error(` ⚠ ${name} dropped ${(wowDelta * -100).toFixed(1)}% vs prior period.`);
}
}
checkIndexingHealth().catch((e) => {
console.error(e);
process.exit(1);
});
```
## 11. Schema Markup at Scale
**Valid syntax ≠ a rich result.** Schema.org markup parses fine for any type, but Google only *renders rich results* for specific types and reserves the right to show none. At pSEO scale, prioritize types that are still broadly eligible **and** reflect content visible on the page (markup must match on-page content or it's a structured-data spam violation):
| Schema type | Rich-result status (as of Jun 2026) | Use for |
|-------------|-------------------------------------|---------|
| `BreadcrumbList` | Broadly shown | Every page |
| `ItemList` / `Product` | Shown (Product needs price/availability) | Directory & listing pages |
| `LocalBusiness` | Shown for genuine businesses | Location/provider pages |
| `Review` / `AggregateRating` | Shown, but **only for content the page is genuinely about**; self-serving/site-wide ratings are ineligible | Provider/product pages with real reviews |
| `FAQPage` | **Sharply restricted since Aug 2023** — rich results now limited mainly to authoritative gov/health sites. For most sites it renders nothing | Keep only as on-page UX; do NOT add at scale expecting SERP real estate |
| `HowTo` | **Deprecated as a rich result** (rolled back, ~2023) | Don't rely on it |
Rule of thumb: ship `BreadcrumbList` + the page's primary type (`Product`/`LocalBusiness`/`ItemList`) everywhere; add `Review`/`AggregateRating` only where real, on-page reviews exist. Validate with the Rich Results Test (`search.google.com/test/rich-results`) and confirm current eligibility at `developers.google.com/search/docs/appearance/structured-data`. Never mark up ratings/reviews/FAQs that aren't actually visible to the user.
```typescript
// lib/schema.ts
export function generateLocalBusinessSchema(service: Service, location: Location, providers: Provider[]) {
return {
'@context': 'https://schema.org',
'@type': 'ItemList',
name: `${service.name} in ${location.city}, ${location.state}`,
numberOfItems: providers.length,
itemListElement: providers.slice(0, 10).map((p, i) => ({
'@type': 'ListItem',
position: i + 1,
item: {
'@type': 'LocalBusiness',
name: p.name,
address: {
'@type': 'PostalAddress',
addressLocality: location.city,
addressRegion: location.state,
},
aggregateRating: p.reviewCount > 0 ? {
'@type': 'AggregateRating',
ratingValue: p.avgRating,
reviewCount: p.reviewCount,
} : undefined,
telephone: p.phone,
},
})),
};
}
// NOTE: FAQPage rich results are heavily restricted (see table above). Only emit
// this if the Q&A is genuinely on-page; do not roll it out site-wide expecting
// SERP FAQ accordions — for most sites that real estate no longer exists, and
// marking up hidden/duplicated FAQs risks a structured-data spam action.
export function generateFAQSchema(faqs: FAQ[]) {
return {
'@context': 'https://schema.org',
'@type': 'FAQPage',
mainEntity: faqs.map(faq => ({
'@type': 'Question',
name: faq.question,
acceptedAnswer: {
'@type': 'Answer',
text: faq.answer,
},
})),
};
}
```
## 12. Pre-Launch Checklist
### Data Quality
- [ ] Every page passes quality gate (minimum data thresholds met)
- [ ] No duplicate pages (check slugs for collisions)
- [ ] Data is current (enrichment pipeline ran within last 7 days)
- [ ] Spot-check 20 random pages manually for accuracy
### Technical SEO
- [ ] Every page has unique `<title>` and `<meta description>`
- [ ] Self-referencing canonical on every page
- [ ] Sitemap submitted and all URLs return 200
- [ ] robots.txt doesn't block template pages OR pagination (`?page=`)
- [ ] Structured data validates in Rich Results Test and matches on-page content
- [ ] Breadcrumbs with schema on every page
- [ ] Internal links: every page reachable within 3 clicks from homepage
- [ ] No orphan pages (every page has at least 1 inbound internal link)
- [ ] URL redirects for any slug changes (301, not 302)
### Performance
- [ ] LCP < 2.5s on template pages
- [ ] CLS < 0.1
- [ ] Pages work without JavaScript (SSR/SSG)
- [ ] Images have width/height attributes and lazy loading
### Content Quality
- [ ] Each page carries genuinely unique data (passes the §6 quality gates — unique facts, source coverage, low duplicate-similarity), not just padded word count
- [ ] No boilerplate-only pages (data swap ≠ unique value)
- [ ] Headings are descriptive, not generic
- [ ] Last-updated dates reflect real data freshness, not `new Date()` theater
### Monitoring
- [ ] GSC property verified and sitemap submitted
- [ ] Indexing monitoring script running weekly
- [ ] Core Web Vitals monitoring active
- [ ] 404 monitoring for broken internal links
- [ ] Alerting set up for >10% index drop
## 13. Common Mistakes
1. **Building pages nobody searches for.** Validate demand with keyword research BEFORE building templates.
2. **Same template, zero unique data.** If the only difference between pages is the city name swapped in, that's thin content. Google will nuke it.
3. **Ignoring internal linking.** Pages with no inbound links don't get crawled.
4. **Generating all pages at once.** Start with 100. Validate they get indexed. Then scale to 1,000. Then 10,000.
5. **No freshness signals.** "Last updated" dates, recent reviews, current pricing — these signal pages are maintained.
6. **Blocking crawlers accidentally.** Triple-check robots.txt.
7. **No fallback for missing data.** If an API is down during build, do you generate empty pages? Always have quality gates.
## 14. Scaling Playbook
### Phase 1: Validate (100 pages)
- Build 1 template, 100 pages
- Submit to GSC, wait 2–4 weeks
- Track: index rate, impressions, click-through rate
- **Gate:** >70% indexed, some impressions → proceed
### Phase 2: Expand (1,000 pages)
- Refine template based on Phase 1 data
- Add 900 more pages
- Implement internal linking hub
- **Gate:** Consistent indexing, growing impressions → proceed
### Phase 3: Scale (10,000+ pages)
- Add new page types (comparisons, integrations)
- Build cross-linking between page types
- Set up automated data enrichment pipeline
- Implement ISR for freshness without full rebuilds
### Phase 4: Optimize
- A/B test title tags and meta descriptions
- Add schema markup variants
- Build topical authority with supporting blog content
- Monitor and prune underperforming pages