@lunora/cli/skills/lunora-migration-helper/SKILL.md

The Lunora CLI: init, dev, deploy, codegen, run, reset, and migrate commands

---
name: lunora-migration-helper
description: Plans Lunora schema and data migrations with widen-migrate-narrow. Use for
    breaking schema changes, backfills, table reshaping, online data migrations
    (`defineMigration` + `lunora migrate up`), the `.global()` D1 SQL flow, and the
    pre-deploy schema-drift gate.
---

# Lunora Migration Helper

Safely change a Lunora schema and migrate data when making breaking changes.

## When to Use

- Adding required fields to existing tables.
- Changing field types or structure.
- Splitting/merging tables, renaming/removing fields.
- Reshaping `.global()` (D1-backed) tables.

## When Not to Use

- Greenfield schema with no data at rest.
- Adding **optional** fields that need no backfill.
- Adding new tables or indexes with no correctness concern.

## Two Storage Layers — Know Which You Are Migrating

Lunora tables live in one of two backends, and they migrate differently:

- **ShardDO SQLite (default `root`, and `.shardBy(key)` tables).** State lives in
  the per-app / per-shard Durable Object. Data is reshaped with **online data
  migrations** — `defineMigration` declarations run by `lunora migrate up`,
  resumable per shard.
- **`.global()` tables (D1).** Replicated to D1 for cross-region reads. Their
  structural DDL gets versioned **SQL migrations** via `lunora migrate generate`,
  applied by `@lunora/d1`'s runner at deploy time.

A breaking change to a `.global()` table needs a generated SQL migration; a data
backfill (either layer) is an online `defineMigration`. Both follow the same
**widen → migrate → narrow** discipline.

## Key Principle: Widen, Migrate, Narrow

The schema-drift gate (and D1 itself) will not let a breaking change deploy
without an accompanying migration. So every breaking change is staged:

1. **Widen** — make the schema accept both old and new shapes (add the new field
   as `v.optional`, keep the old one). Update reads to handle both; start writing
   the new shape for new rows. Deploy.
2. **Migrate** — backfill existing rows to the new shape (an online
   `defineMigration` run with `lunora migrate up`; plus `lunora migrate generate`
   for `.global()` structural DDL). Verify completeness with `lunora migrate
status`.
3. **Narrow** — make the field required / drop the old field, remove the
   both-shapes read code. Deploy.

### Prefer new fields over changing types

When changing a field's shape, add a new field rather than mutating the existing
one — safer transition, easier rollback.

### Don't delete data prematurely

Prefer deprecating: mark the old field `v.optional` with a `// deprecated:` code
comment explaining why it existed. Delete only once you are sure nothing reads
it.

## Safe Changes (No Migration Needed)

```ts
// Adding an optional field — safe.
users: defineTable({
    name: v.string(),
    bio: v.optional(v.string()),
});

// Adding a new table — safe.
posts: defineTable({ userId: v.id("users"), title: v.string() }).index("by_user", ["userId"]);

// Adding an index — safe.
users: defineTable({ name: v.string(), email: v.string() }).index("by_email", ["email"]);
```

## Online Data Migrations (the backfill workhorse)

For backfilling/reshaping rows, declare a migration with `defineMigration` from
`@lunora/server`. It transforms one document at a time, runs **inside each
shard's** Durable Object in keyset batches, and is **resumable** — per-shard
progress is tracked in a reserved `__lunora_migrations` table, so an interrupted
run resumes where it stopped. Codegen discovers declarations and emits them into
the registry the DO and CLI look up by `id`.

```ts
// lunora/migrations/backfill-display-name.ts
import { defineMigration } from "@lunora/server";

export default defineMigration({
    id: "backfill-display-name",
    table: "users",
    batchSize: 200, // optional; defaults to the runner's batch size
    up: (doc) => {
        if (typeof doc.displayName === "string") {
            return; // already migrated — return undefined to skip (not counted as changed)
        }
        return { ...doc, displayName: doc.name ?? "Anonymous" };
    },
    // optional reverse transform applied by `lunora migrate down`
    down: (doc) => {
        const { displayName, ...rest } = doc as Record<string, unknown>;
        return rest;
    },
});
```

The transform must preserve row identity — the runner always keeps the original
`_id` / `_creationTime`, so do not change them.

### Run it

```bash
lunora migrate create backfill-display-name   # scaffold the migration file
lunora codegen                                 # discover + register it
lunora migrate up backfill-display-name --dry-run   # preview, no rows rewritten
lunora migrate up backfill-display-name        # run across shards (keyset batches)
lunora migrate status backfill-display-name    # per-shard progress
lunora migrate down backfill-display-name      # revert (if `down` defined)
```

Useful flags: `--batch-size <n>`, `--steps <n>` (cap batches this run), and
`--prod --url <worker> --yes` to target production (with `LUNORA_ADMIN_TOKEN`).

## `.global()` (D1) Structural Migration Flow

```bash
# 1. Edit lunora/schema.ts (widen: add the optional new field to the .global() table).
lunora codegen

# 2. Generate the SQL migration by diffing schema against the snapshot baseline.
lunora migrate generate --name=add_user_status

# Writes lunora/migrations/<timestamp>_add_user_status.sql and updates
# lunora/migrations/.snapshot.json. Review the SQL before committing.

# 3. Deploy — @lunora/d1's runner applies pending migrations.
lunora deploy
```

`lunora migrate generate` only considers `.global()` tables (root/sharded tables
are not D1-backed). Run it after each schema edit in the widen and narrow steps;
backfill data with an online migration between them.

## The Schema-Drift Gate

`lunora deploy` (and `verify` / `prepare`) run a **pre-deploy schema-drift gate**:
it compares the committed structural baseline (`lunora/.lunora-schema.json`)
against the snapshot codegen produced this run. Breaking drift **without an
accompanying data migration blocks the deploy**. The baseline is only re-blessed
_after_ the deploy succeeds, so a failed deploy never advances it past a change
that never shipped.

If the gate blocks you: that is the signal to stage the change (widen first) or
add the migration — not to bypass it.

## Common Pitfalls

1. **Making a field required before backfilling.** The drift gate / D1 rejects
   the deploy because existing rows lack it. Widen first.
2. **Reshaping rows by hand instead of `defineMigration`.** A hand-rolled
   `internalMutation` that `.collect()`s a large table hits transaction limits
   and is not resumable. Use `defineMigration` — it batches and tracks per-shard
   progress.
3. **Not writing the new shape during the migration window.** Rows created mid-
   migration get missed, leaving unmigrated data after it "completes." Start
   dual-writing in the widen step.
4. **Skipping the dry run.** `lunora migrate up <id> --dry-run` validates the
   transform before it touches real rows.
5. **Deleting a field prematurely.** Deprecate with `v.optional` + a comment;
   delete only once nothing references it.
6. **Migrating the wrong layer.** A `.global()` structural change needs `lunora
migrate generate` (SQL); a data backfill needs a `defineMigration`. Check the
   table's `.global()` / `.shardBy()` modifier first.

## Checklist

- [ ] Identified the change and which layer it touches (ShardDO vs `.global()`).
- [ ] Widened the schema to accept both shapes; `lunora codegen` clean.
- [ ] Updated reads to handle both shapes; started writing the new shape.
- [ ] Deployed the widened schema.
- [ ] Authored a `defineMigration`; previewed with `lunora migrate up --dry-run`.
- [ ] Ran `lunora migrate up`; `lunora migrate generate` + deploy for `.global()`
      structural changes.
- [ ] Verified completion with `lunora migrate status`.
- [ ] Narrowed the schema (required / drop old field); removed both-shapes code.
- [ ] Deployed the final schema; schema-drift gate passed.