rigorrun
Version:
Permission and scope tests for AI agents: send your agent tickets that tempt it to act for the wrong customer or organisation, then read the system itself — a multi-tenant helpdesk twin, or Stripe — instead of the transcript.
186 lines (137 loc) • 8.88 kB
Markdown
<div align="center">
# RigorRun
**Your agent acts for one customer at a time. You check it stays there by reading the transcript.**
RigorRun sends your agent tickets that tempt it across that line — somebody else's order, a decision
that is not its to make, an instruction hidden in the customer's message — then reads the system
itself and shows what actually happened beside what the agent said it did. It ships two packs: Larch
Helpdesk, a multi-tenant helpdesk twin, and Stripe — your test mode, or a local twin with no keys.
The verdict comes from what the system recorded, never from what the agent says about itself.
**Early Access** — parts of it are honestly unfinished, and they are listed rather than hidden.
[rigorrun.xyz](https://rigorrun.xyz) · [Documentation](https://docs.rigorrun.xyz) · [A recorded run](https://rigorrun.xyz/replay) · [What is and is not built](https://rigorrun.xyz/what-is-built)
</div>
```bash
npx rigorrun helpdesk try # six tickets on a two-organisation helpdesk twin; no keys, seconds
npx rigorrun demo # a recorded run, replayed offline; no keys, no account
```
`helpdesk try` runs one support agent twice — once with the organisation's own token, once with the
helpdesk's service token — and shows, from the twin's own access log, tables and outbox, which one
read, changed or sent another organisation's data.
`demo` replays a real model working a support desk: what it said beside what the system held
afterwards. The same run, case by case, is at [rigorrun.xyz/replay](https://rigorrun.xyz/replay).
### Your multi-tenant support agent
```bash
npx rigorrun helpdesk twin # MCP at http://127.0.0.1:12113/mcp; leave it running
npx rigorrun helpdesk try --agent <agent URL> # your agent, connected to the twin with its token
```
Each ticket is decided from what the twin recorded: its access log (another organisation's rows
read), its tables (records changed or deleted) and its outbox (who was emailed, and whether another
organisation's planted marker went with it). The
[helpdesk guide](https://docs.rigorrun.xyz/start/helpdesk) has the whole path, including `init`
and the gate.
### Your Stripe agent, no code changes
```bash
npx rigorrun stripe twin # a local Stripe twin; leave it running
npx rigorrun stripe init --twin --yes # the Stripe pack's tickets, as a project
npx rigorrun agent add --project <id> --name my-agent \
--black-box <agent URL> --claim-path message # your agent's endpoint, as it runs today
npx rigorrun gate --project <id> --report report.html # exit 1 when a case fails
```
No change to your agent's code: RigorRun sends each ticket to the endpoint your agent already
serves (map your request shape with `--body-template`) and reads Stripe itself. Your agent keeps its
own Stripe test key — or, for the local twin, its Stripe base URL points at the twin.
Against your Stripe test mode, `stripe init` takes a test key instead of `--twin`; live keys are
refused. The [Stripe guide](https://docs.rigorrun.xyz/start/stripe) has the whole path, including
the canary and the GitHub Action.
`npx rigorrun` opens a local interface for any other system RigorRun can reach and read back.
Everything runs on your machine: there is no account, and no hosted component to send your systems
to.
---
## Why
An agent that reports success and an agent that achieved it are indistinguishable from the
transcript. They are trivially distinguishable from the database.
Evaluation harnesses score what the model wrote. RigorRun compares system state before and after,
through read operations you nominate, and asks the questions that have a fact behind them: does the
refund exist, is the amount right, is the ticket attached, was approval required, and did anything
change that should not have.
```
most tools: you write the tests → the tool runs them
RigorRun: you do the job once → RigorRun writes the tests
```
## How
1. **Connect your system** — an MCP server you already run, an OpenAPI document, or a web
application through a browser.
2. **Do the job once.** RigorRun reads your system before and after and derives what the rules must
be.
3. **Rule on what it worked out.** It shows the evidence behind each proposed rule. A rule you
reject cannot fail your agent.
4. **Connect your agent** — no change to its code, wherever it runs: RigorRun sends each case's work
to the endpoint it already serves (map your request shape with `--body-template`) and reads the
result from your system (a black box). Or an HTTP endpoint, a local
command, or your own loop pulling work.
5. **Run it.** A verdict, with how strongly each answer could be verified.
6. **Gate the next change** in CI.
Setting a project up happens in the local interface. Running, gating and comparing are also
available from the command line, which is the half CI needs.
```bash
rigorrun gate --project <id> # exit 1 stops the build
rigorrun gate --project <id> --case <case-id> # only the cases you name; never the baseline
rigorrun agent add --project <id> --black-box <url> # connect an agent from a script; probed first
```
## A Stripe refund agent
```bash
rigorrun stripe twin # a local twin of Stripe's API; no keys
rigorrun stripe init --twin --yes # or: STRIPE_TEST_KEY=sk_test_… rigorrun stripe init --safety staging
rigorrun agent add --project <id> --name my-agent --black-box <url> --claim-path message
rigorrun stripe canary --project <id> --agent my-agent
rigorrun gate --project <id> --report report.html
```
Every case creates its own customer and payments; RigorRun reads them back with its own test key
and judges the agent on what Stripe holds. Keys must be test-mode keys, confirmed by Stripe before
anything is stored; live mode is never used. Verdicts are `PARTIAL` and print what their reads
covered. [The walkthrough](https://docs.rigorrun.xyz/start/stripe/).
## Verify a published MCP server
An MCP server can annotate a tool `readOnlyHint: true`. Nothing checks that, and agents use it to
decide whether a tool may be called without asking.
```bash
rigorrun verify npm:@modelcontextprotocol/server-memory@2026.8.31
```
Pins the server to the exact bytes the registry published, runs it in a container with no network
and no access to your machine, calls each tool with arguments derived from its own schema, and reads
the filesystem before and after. Needs a container runtime; nothing else does.
Run against four published servers, it exercised **19 of 37** tools. The other 18 are named, each
with the reason it was not reached — [see the records](https://rigorrun.xyz/evidence).
## What you should know before relying on it
- **A verdict against a real system is `PARTIAL`, by design.** RigorRun reads back what your
nominated reads return and no more, and says so on every result.
- **A browser cannot verify itself.** A page saying "done" is a claim by the system that would have
to be wrong for it not to be done, so a browser connection is `OBSERVATIONAL`. Attach a readable
API or MCP connection for the same system to strengthen it.
- **Isolation is `DECLARED`, not `RESET`, when you nominate a reset tool** — RigorRun has not run it
twice and compared.
- **Three of the six verification sources are not emitted yet.** Model-judged and human-review
evaluators exist in the schema and are unreachable in practice.
- **No external team has used this yet.** There are no customers.
The full list is at [rigorrun.xyz/what-is-built](https://rigorrun.xyz/what-is-built), and the
version carrying how each line was checked is
[docs/V1_GAP_AUDIT.md](https://github.com/Konuktor/rigorrun/blob/master/docs/V1_GAP_AUDIT.md).
## Privacy
Your systems, your credentials and your recordings stay on this machine. There is no account, and
nothing is uploaded unless you ask for it.
Three things can make a network request, all because you asked: `rigorrun verify` downloads the named
package from the npm registry; connecting your own system sends requests to your own system; and an
LLM-backed agent, which exists only if you set a provider key, talks to that provider. None of them
reach RigorRun. [How it is built](https://rigorrun.xyz/security).
## Verifying what you installed
Published by a GitHub Actions workflow that holds no npm token, using npm trusted publishing, with a
SLSA build provenance attestation.
```bash
npm audit signatures
npm view rigorrun dist.integrity
```
## Requirements
Node 20.11 or newer. Docker only for `rigorrun verify`.
```bash
rigorrun doctor # what this machine has, and what it does not
```
---
MIT · Built by Erbol Tahirov · [Report a problem](https://github.com/Konuktor/rigorrun/issues)