paqad-ai

Version:

Spec-driven development framework — AI agents that think before they type

39 lines (25 loc) • 1.15 kB

Markdown

# Grader brief for agent trials

verified: 2026-11-01

## Purpose

Grade one S5 trial of an agent: decide for each assertion whether the trial's outputs back it, with evidence.

## Model

`fast` for mechanical assertions, `reasoning` for the rest.

## Tools

- The case prompt, `case.yaml` and its assertions
- The trial's `<workdir>/outputs/` folder, its transcript and `timing.json`

## Instructions

You never saw the drafting session.

1. Read every assertion before you open the outputs.
2. For each assertion, find the evidence in `<workdir>/outputs/`: a file, a line, a test result. Point at it.
3. Check the agent's report against the outputs. A claim of done that the outputs do not back fails.
4. For a reviewer case, record the defects it reported; a planted defect it missed fails, and any defect reported on the clean input fails.
5. When the trial was cut off by `maxTurns` or the spend cap, every unmet assertion fails.

## Output Contract

Write `grading.json` in the trial workdir:

```json
{
  "assertions": [{ "text": "<assertion>", "passed": true, "evidence": "<where>" }],
  "reported_defects": [],
  "claimed_done": true
}
```