@nytka/plugin-typesafe
Version:
TypeSafe connector for nytka projects. Asks Jev typed questions (noul, choice, score) about a file or an existing dataset and registers the judgements in datasets/ with the model, the questions hash and the usage.
275 lines (216 loc) • 14 kB
Markdown
# @nytka/plugin-typesafe
TypeSafe connector for nytka projects. Asks [Jev](https://docs.typesafe.ai), TypeSafe's System
One decision model, typed questions about material the project already holds — a choice from
named options, a score against named levels, a yes/no probability — and registers the answers
in `datasets/index.json` with the model that gave them, the questions they answered and what
they cost.
It is the first `@nytka` connector that **judges rather than collects**. The six before it
fetch a dated fact from a system the project uses; this one takes a payload one of them
wrote, or any file of items, and returns a judgement per item with a confidence. No text is
generated. Nothing at TypeSafe is written. Every call spends.
## Install
```bash
npm install @nytka/plugin-typesafe
```
**One dependency: `@nytka/core`**, which has no dependencies of its own and holds the project
plumbing. The HTTP client is Node's built-in `fetch`: one endpoint, one verb, and adding an SDK
for it would put a second dependency in every project that judges anything.
## Setup
About a minute, and there is no key file.
### 1. Get a key
Create an API key in your TypeSafe account at <https://typesafe.ai>. It is a bearer key; it is
the whole credential.
### 2. Configure
One `.env` at the **project root**. Every key for the project lives there and nowhere else.
The names ship with the package, so recovering them later never means re-reading this guide:
```bash
cat node_modules/@nytka/plugin-typesafe/.env.example >> .env
```
```bash
TYPESAFE_API_KEY=ts_…
# Optional. Defaults to jev-latest; --model on any command wins over both:
# TYPESAFE_MODEL=jev-1.13
```
### 3. Verify
```bash
npx nytka-typesafe check
```
One ten-token question. Prints the resolved model, the noul answer, and the usage with its
estimated cost. Left unset, the key is named and the command exits 0 — an unconfigured
project, not a broken one.
## Use
Three commands. Two are the "hand" and cost one call; one is the "memory" and writes a dataset.
```bash
# The auth smoke test
npx nytka-typesafe check
# One state, every question in the file. Answers as JSON on stdout, usage on stderr.
npx nytka-typesafe ask --questions references/questions/intent.json --state-json '{"keyword":"bathroom remodel cost"}'
npx nytka-typesafe ask --questions references/questions/quality.json --state research/sources/passage-12.txt
# The same questions over every item — plan first, then run
npx nytka-typesafe classify --input dataforseo-keyword-ideas-bathroom-remodel-cost-7c1e90aa-us-en-2026-09 \
--questions references/questions/intent.json --field keyword
npx nytka-typesafe classify --input … --questions … --field keyword --yes
```
### The questions file
The API's own `questions` map, verbatim, committed anywhere in the project —
`references/questions/<name>.json` is the suggestion, and the file's name becomes the readable
part of the dataset id:
```json
{
"intent": {
"type": "choice",
"instructions": "What is the searcher most likely trying to do with this keyword?",
"criteria": {
"informational": "wants to learn or understand something",
"commercial": "is comparing or pricing something to buy",
"navigational": "is trying to reach a specific site or brand"
}
},
"is_question": { "type": "noul", "instructions": "Is the keyword phrased as a question?" }
}
```
`choice` needs `criteria` as an object of at least two options; `score` needs `criteria` as an
array of at least two levels; `noul` needs neither. The file is validated before anything is
sent.
### `classify`, and why it plans first
`classify` is the one command whose spend is proportional to a payload you are not supposed
to read. So **without `--yes` it plans and stops**: how many items, which questions, which
model, how many calls, roughly how many tokens and what that costs at the published rate —
and the dataset id it would write. Nothing is sent and no key is needed. Add `--yes` to run.
```
classify intent over dataforseo-keyword-ideas-bathroom-remodel-cost-7c1e90aa-us-en-2026-09
input dataset (rows): 412 items
questions intent:choice, is_question:noul [8b1d4e0f]
model jev-latest
would send 412 calls, about 61800 input tokens
estimated $0.0026 at $0.042/Mtok (rate as of 2026-09-19; output tokens are free)
dataset id typesafe-classify-dataforseo-keyword-ideas-bathroom-remode-7c1e90aa-intent-8b1d4e0f-jev-latest
Nothing was sent. Add --yes to run it.
```
Flags:
| Flag | Does |
|---|---|
| `--input` | a dataset id from `datasets/index.json` — its payload is read by **this connector**, never by you — or a `.json` array / `.jsonl` file of items |
| `--field name` | send one field of each item as the state instead of the whole item. Jev's accuracy falls as the state fills with things unrelated to the question, so this is usually right |
| `--key name` | which field names an item in the rows. Auto-detected from `_id`, `id`, `key`, `keyword`, `url`, `slug`, `name`; the row's `index` is always there |
| `--items key` | which key of a payload holds the items. Auto: `rows`, `documents`, `items`, `results`, `campaigns` |
| `--limit N` | the first N items. `--limit 5 --yes` is the cheap trial before the real run |
| `--concurrency N` | parallel calls, 1–16, default 4. Lower it on a 429 |
| `--model M` | wins over `TYPESAFE_MODEL` and the default |
| `--snapshot` | append today's date to the id, so re-runs accumulate instead of replace |
| `--no-register` | write the payload, leave `datasets/index.json` alone |
With `--yes` the run sends item 0 alone first — a questions file the API rejects costs one
call, not four — then the rest through a pool. Rows land in input order whatever the network
did. **A failure part-way fails the whole run**, and the answers already paid for go with it;
the retry policy, the canary and `--limit` are the mitigations, and a partial-write mode is
deliberately not one of them.
### Reading the result
Query the payload with a script, like any dataset. Each row is:
```json
{ "index": 0, "key": "bathroom remodel cost",
"answers": { "intent": { "type": "choice", "choice": "commercial", "probabilities": { "…": 0.91 }, "confidence": 0.91 },
"is_question": { "type": "noul", "noul": 0.12, "confidence": 0.88 } },
"confidence": 0.88 }
```
`answers` is exactly what the API returned. The row's `confidence` is the smallest across its
answers — the number to filter on for "rows Jev was sure about on every question". The
conclusion goes into `research/`, with the dataset id in `sources` and `confidence: inferred`.
## What it writes
| Path | Committed? |
|---|---|
| `datasets/payloads/<id>.json` | no — the directory ships its own `.gitignore` |
| `datasets/index.json` | yes — one entry, added or replaced |
### The dataset id
```
typesafe-classify-<input>-<sha8>-<questions>-<sha8>-<model>[-<date>]
```
- **The input, keyed on its reference** — the dataset id or the project-relative path — not
its content. The id names the question, never the answer; whether the content changed under
the same reference is `inputHash` in the entry.
- **The questions, keyed on their content.** Editing one criterion is a different question,
and a re-run after the edit must not overwrite the answers to the old one. The file's name
is the readable prefix.
- **The requested model.** The same items under two judges are two answers. The alias is what
was asked for; the resolved version is in the entry.
- **Date-free by default**, so a re-run replaces. `--snapshot` appends the local date and makes
a series — for the project that wants to watch answers drift across model releases.
`--limit` and `--concurrency` are not in the id: they size the run, they do not change the
question. Re-running at a smaller limit replaces a longer payload with a shorter one, and the
entry's `rows` shows it happened.
### The registry entry
Shares `id`, `source`, `operation`, `collectedAt`, `validUntil`, `rawPath`, `rows`, `schema`,
`summary`, `status` and `producedBy` with every collector's entry, so one registry reads as one
file. Its own fields say what was judged, by what:
- **`model`** — the resolved model (`jev-1.13`), the provenance of every answer in the payload.
- **`inputs`** — the reference that was judged. The first connector whose input is another
dataset; lineage is a `PLG-001` item and until the contract names the field, it is this one.
- **`inputHash`**, **`questionsPath`**, **`questionsHash`**, **`questionIds`**,
**`questionTypes`** — enough to know what the payload holds without opening it.
- **`subject`**, not `property`: `"<questions name> over <input ref>"`.
- **`dateRange` is explicitly `null`**, and there is no `period`. A judgement covers no dates;
what makes it stale is the model changing, and `model` shows that.
Cost is **not** in the entry. It is in the payload's `apiMetadata` — `usage` as the API
reported it, `estimatedCost` computed from it, and the `rate` with the date it was checked —
and on stdout. Same as DataForSEO, and the same open `PLG-001` question.
The registry writer preserves the file's existing formatting: adding one dataset produces a
one-entry diff, not a reformat of every entry already there.
## Troubleshooting
| What you see | What it means |
|---|---|
| `TYPESAFE_API_KEY is not set` | Not in `.env` at the project root. Nothing was sent; exit 0 |
| `(HTTP 401)` | The key is set and TypeSafe rejected it — revoked, or pasted with a stray character |
| `(HTTP 422)` | The request did not validate. The questions file or the state, not the key; the body says which |
| `(HTTP 429)` after a wait | Rate-limited, already retried five times with backoff. Lower `--concurrency` |
| `(HTTP 529)` after a wait | TypeSafe overloaded, already retried. Later |
| `asked for a 120s wait` | A `Retry-After` past 30 s. The connector will not wait that long unattended; you decide |
| `not a dataset id … and not a file` | `--input` matched nothing in `datasets/index.json` and no file under the project root or the working directory |
| `points at … which is not on this machine` | The dataset is registered but its payload is gitignored and absent here. Re-run the collector, or classify a file |
| `has no array under any of rows, documents, …` | The payload's items are under another key. Say which with `--items` |
| `item 12 has no "keyword" field` | `--field` names a key not every item carries |
| `REFUSES 3 item(s) too large` | Over ~32k tokens of state each. Use `--field`, or trim the input |
| `no project.yaml found walking up from cwd` | Not inside a nytka project |
## Rules it follows
- **Payloads never enter agent context.** This connector reads one so that you do not have to:
the plan prints counts and an estimate, the run prints counts, usage and paths. Query the
result with a script; write the conclusion to `research/`.
- **A model's answer is a suggestion, never a gate.** A `confidence` cannot accept a task,
approve a publication, or establish a fact. It can route, rank and flag for a human.
- **Spend is visible or it did not happen.** Every call's usage is recorded; every estimate
names its rate and the date the rate was read.
- **Plan before spend.** `classify` shows what it would send and stops without `--yes`.
- The project is found by walking up for `project.yaml`, so it works at any install depth.
- No YAML parsing, no config file, no key file. Secrets from `.env`, everything else from flags.
- Dates are written in the **local** calendar, never UTC-stamped.
- One URL, one verb. There is nothing at TypeSafe this package could write, and a test asserts
the hostname appears exactly once in the source.
## Limits worth knowing
- **Jev reads literally** and answers the question you wrote, not the one you meant. State the
boundary cases in `instructions`; align `criteria` with them.
- **It is not a calculator.** No arithmetic, no counting, no date comparison — those belong in
code, before or after the call. Ask it to *classify* a date's month, not to compare two.
- **Irrelevant state lowers accuracy.** `--field` exists for this. Send the sentence, not the
row.
- **Injected instructions in the state can move the answer.** Text scraped from the web is
adversarial by default; treat judgements over it as a first pass a human reviews.
- **English first.** Other languages, Ukrainian included, are handled but not equally well —
test on your own content with `--limit` before relying on it.
- **`jev-latest` moves.** Pin `TYPESAFE_MODEL` when two runs must be comparable. The resolved
model is in every entry, so a drift is at least visible afterwards.
- **The token estimate is bytes over four.** Rough on purpose. It under-counts dense scripts
and over-counts JSON; the usage the API reports is what gets recorded.
- **The rate is a constant that goes stale.** `$0.042 per million input tokens` was read from
TypeSafe's models page on 2026-09-19. Every printed cost says so.
- 64k tokens per request, 32k for the state plus the longest question. Checked before sending.
- The answers' `probabilities` are the model's, not a calibrated truth; `confidence` is
TypeSafe's own certainty measure and is not a probability. Their docs on
[confidence](https://docs.typesafe.ai/confidence) are worth the five minutes.
## Tests
```bash
npm test # from this package
node --test # same thing
```
Eighty tests, no network, no credential, no client data. The request layer runs against a fake
`fetch`; `classify` runs against a stub client that records what was sent and in what order;
the registry tests append into a fixture written by the gsc, ga4 and dataforseo connectors and
assert a one-entry diff; the CLI tests spawn the real process and check exit codes and the
absence of stack traces.