relay-runtime/llm-docs/guides/document-comparison.mdx

A core runtime for building GraphQL-driven applications.

---
id: document-comparison
title: Document Comparison
slug: /guides/document-comparison/
description: Relay guide to document comparison
keywords:
  - compare
  - diff
  - ir
---

import DocsRating from '@site/src/core/DocsRating';

:::note
This feature is **experimental** and the API may change in future versions.
:::

The Relay compiler includes an experimental feature for comparing the intermediate representations (IR) of two GraphQL documents. This feature determines whether one document is a subset of another and reports any missing selections.

This feature is particularly useful for comparing documents generated by LLMs where exact match of LLM generated documents and static expected documents is neither attainable or desirable, due to the nature of LLMs. In those cases, you could:
* Specify (via `--left`) an expected document with minimally required selections, and determine if the LLM generated document (via `--right`) contains all required selections, via the subset test that the command performs.
* Optionally, pass exepected document via `--right` and LLM generated document via `--left` to determine whether LLM generated documents contains extra selections than expected, aka. overfetching.

## Usage

```
> relay experimental-compare-document-ir --help
EXPERIMENTAL! Compare intermediate representations (IR) of documents passed via left and right, outputs selections that exists in left but not in right.

Usage: relay experimental-compare-document-ir [OPTIONS] --left <LEFT> --right <RIGHT>

Options:
      --left <LEFT>                    Document in string, must contain exactly 1 operation
      --right <RIGHT>                  Document in string, must contain exactly 1 operation
      --schemaPaths <SCHEMA_PATHS>...  Path(s) to the full schema file(s) to convert documents to intermediate representation (IR) for comparison
  -h, --help                           Print help
```

## Output

Detailed information about missing selection(s) from the right document, including line number and path.

## How It Works

The 2 documents (left and right) are first transformed to their Intermediate Representation (IR), then to **Normalized Tree** form, for comparison. Each node of the tree corresponds to a selection in the original document, and is checked for its existence in the other tree.

### Normalized Trees

Each IR is transformed into a normalized tree structure that:
- **Inlines fragments**: Fragment spreads and inline fragments are inlined, while type conditions specified by the fragment are encoded in the node.
- **De-aliases fields**: Converts aliased fields to their original names (e.g., `my_id: id` → `id`)
- **Deduplicates fields**: Removes duplicate identical fields
- **Tracks type conditions**: Records which concrete object types can reach each scalar field at the leaf level

The normalized tree mirrors the JSON response structure of the query.

#### Tree Node Identity
Consists of
- Path from operation root to the corresponding selection
- Field arguments of the corresponding selection (if applicable)
- Type condition: the type that the enclosing object must be, for the corresponding selection to be included in the response. E.g. with `... on A { id }`, the type condition of selection `id` is `A` if `A` is a concrete type, or a set of types that are `A` if `A` is an abstract type.

#### Tree Node Existence

A selection exists in a tree if:
- An identical selection is found
- A superset of the selection is found. A selection is a superset of another if:
  - It allows the same selection and more, with query variables. E.g. `user(id: $id)` is a superset of `user(id: 100)` because variable `$id` can be set to arbitrary values, including `100`, at runtime.
  - Its type condition is a superset of another.

## Example

**Left document:**
```graphql
query A {
  viewer {
    timezone_estimate {
      display_name
      gmt_offset
    }
  }
}
```

**Right document:**
```graphql
query A {
  viewer {
    timezone_estimate {
      display_name
    }
  }
}
```

**Output**
```text
[INFO] Found 1 missing selection(s):
    4 │       display_name
    5 │       gmt_offset
      │       ^^^^^^^^^^
    6 │     }
* query -> viewer -> timezone_estimate -> gmt_offset
```

Explore [tests](https://github.com/facebook/relay/blob/main/compiler/crates/graphql-ir-diff/tests/ir_diff_test.rs) for more examples.