@adonis-agora/filter
Version:
Server-side query filtering/sorting/pagination for AdonisJS — Spatie-style input, a Lucid adapter, and field allow-listing. Part of the Agora ecosystem.
320 lines (237 loc) • 10.4 kB
Markdown
---
name: filter-querying
description: >
Advanced querying with @adonis-agora/filter beyond basic offset filtering —
keyset/cursor pagination (applyCursor, applyCursorFromRequest, buildCursorPage,
buildKeyset, opaque base64url encodeCursor/decodeCursor, after/before/first/last
params), the three search modes (searchable ILIKE scan vs fullText tsvector
websearch_to_tsquery/ts_rank vs vectorSimilarity pgvector distance metrics
cosine/l2/innerProduct with threshold/topK), DISTINCT projections and their
root-table-column restriction, and the richer request surface (parseSpatieRequest
with include/sparse fieldsets/cursor params, resolveInputFromRequest sources,
normalizeInput key casing). Use when serving stable deep pages, ranking by
relevance or embeddings, projecting distinct column values, or accepting
filters from a POST body.
metadata:
type: core
library: "@adonis-agora/filter"
library_version: "0.7.0"
framework: adonisjs
sources:
- "DavideCarvalho/adonis-filter:docs/pagination/index.mdx"
- "DavideCarvalho/adonis-filter:docs/search/index.mdx"
- "DavideCarvalho/adonis-filter:docs/search/full-text.mdx"
- "DavideCarvalho/adonis-filter:docs/search/vector-similarity.mdx"
- "DavideCarvalho/adonis-filter:docs/guides/lucid.mdx"
- "DavideCarvalho/adonis-filter:docs/definitions/request-input.mdx"
---
# Structured querying: cursors, search modes, distinct
Offset pagination (`applyFilter` → `page`/`size`) drifts as rows are inserted
and gets slow on deep pages. Keyset pagination seeks from a boundary row; text
`search` routes through one of two engines you pick per policy; `distinct`
projects the values that exist under the current filters.
## Setup
```typescript
// app/controllers/users_controller.ts
import type { HttpContext } from '@adonisjs/core/http'
import { applyCursorFromRequest, buildCursorPage } from '@adonis-agora/filter'
import { userFilter } from '#filters/user_filter'
import User from '#models/user'
export default class UsersController {
async index(ctx: HttpContext) {
const query = User.query()
// same spec as offset flow; cursor params instead of page/size
const resolved = applyCursorFromRequest(query, userFilter, ctx, { primaryKey: 'id' })
const rows = await query.exec()
const page = buildCursorPage(rows.map((r) => r.serialize()), resolved)
return {
data: page.items,
nextCursor: page.nextCursor,
prevCursor: page.prevCursor,
hasNext: page.hasNext,
hasPrev: page.hasPrev,
}
}
}
```
Cursor params arrive as `after`/`before`/`first`/`last` (or JSON:API
`page[after]`…). `after` wins if both bounds are sent. Filters, search, and sort
pass through the same allow-list boundary as `applyFilter`.
## Core patterns
### 1. Keyset building blocks
The effective sort plus a primary-key tiebreaker forms the keyset; the query is
limited to `size + 1` so `buildCursorPage` can detect a next page:
```typescript
import { applyCursor, buildCursorPage } from '@adonis-agora/filter'
const resolved = applyCursor(
User.query(),
{ sort: [{ field: 'name', direction: 'asc' }], after: cursor, first: 20 },
{ allowed: ['name'], primaryKey: 'id' },
)
const rows = await query.exec()
const page = buildCursorPage(rows.map((r) => r.serialize()), resolved)
```
A malformed cursor is ignored, not fatal. Backward paging (`before`) reverses
the walk internally and `buildCursorPage` restores your requested order;
`prevCursor` from a forward page is a valid `before`.
Source: `docs/pagination/index.mdx`
### 2. Full-text search (keywords) via fullText
When the policy declares `fullText`, the request `search` term routes through
Postgres tsvector matching instead of ILIKE:
```typescript
import { defineFilter } from '@adonis-agora/filter'
export const articleFilter = defineFilter({
filterable: ['status'],
fullText: {
column: 'search_vector', // stored, GIN-indexed tsvector column
columnKind: 'tsvector',
language: 'english',
rank: true, // ORDER BY ts_rank(...) DESC
},
})
```
`?search=quick brown fox` becomes
`WHERE "search_vector" @@ websearch_to_tsquery('english', ?)` — raw user input
is safe because `websearch_to_tsquery` tolerates stray punctuation and the term
always travels bound. Plain text columns work too (`columnKind: 'text'`),
wrapped in `to_tsvector` at query time. `applyFullTextSearch` is exported for
direct use outside the runner.
Source: `docs/search/full-text.mdx`
### 3. Vector similarity (embeddings) via vectorSimilarity
Rank rows nearest-first by pgvector distance to a query embedding — supplied in
code, never through the URL:
```typescript
export const docFilter = defineFilter({
filterable: ['status'],
vectorSimilarity: { column: 'embedding', metric: 'cosine', threshold: 0.25, topK: 20 },
})
// controller — compute the embedding, then pass it in options:
const embedding = await embeddingService.embed(q)
const { page, size } = applyFilterFromRequest(query, docFilter, ctx, {
vectorSimilarity: embedding,
})
```
Similarity ordering lands before user sort (sort becomes the tiebreaker) and
runs only when both policy and request carry an embedding. The embedding binds
as `?::vector`; NaN/Infinity components throw rather than emitting bad SQL.
`applyVectorSimilarity` is exported for direct use.
Source: `docs/search/vector-similarity.mdx`
### 4. DISTINCT projections come free on the wire format
`?distinct=status` parses into `input.distinct`, is allow-listed like any
field, and applies itself:
```typescript
const input = parseFilterRequest(request.qs())
applyFilter(query, input, { allowed: ['status', 'baseId'] })
const rows = await query.select('status') // DISTINCT already applied
```
### 5. Richer request shapes
```typescript
import {
parseSpatieRequest,
resolveInputFromRequest,
normalizeInput,
} from '@adonis-agora/filter'
// Spatie/JSON:API superset: includes, sparse fieldsets, cursor page params
const input = parseSpatieRequest(request.qs())
// source filters from body (POST /search), auto-merging query+body on writes
const raw = resolveInputFromRequest(request, 'auto')
// snake_case API → camelCase columns (top-level keys only)
normalizeInput({ company_id: 5 }, { normalizer: 'camelCase' })
```
Source: `docs/definitions/request-input.mdx`
## Common mistakes
### [HIGH] Passing model instances to buildCursorPage
Wrong:
```typescript
const rows = await query.exec()
const page = buildCursorPage(rows, resolved) // Lucid model instances
```
Correct:
```typescript
const rows = await query.exec()
const page = buildCursorPage(rows.map((r) => r.serialize()), resolved)
```
`buildCursorPage` reads keyset values off plain objects by field name; model
instances do not expose those plain keys, so boundary cursors encode garbage
and the next page silently starts from the wrong row.
Source: docs/pagination/index.mdx ("Feed buildCursorPage plain rows")
### [HIGH] Projecting distinct on a relation path
Wrong:
```typescript
applyFilter(User.query(), input, { allowed: ['status', 'posts.title'] })
// ?distinct=posts.title → dropped / InvalidColumnFilterError under throwOnInvalid
```
Correct:
```typescript
// distinct only names root-table columns: ?distinct=status
applyFilter(User.query(), input, { allowed: ['status'] })
```
Relation filtering compiles to correlated EXISTS subqueries, so the relation is
never joined and there is no alias to project from — the emitted SQL would be a
missing-FROM error at the database. Adding the path to allow-lists cannot
change this; it is structural, not a policy decision.
Source: docs/guides/lucid.mdx ("distinct takes root-table columns only")
### [HIGH] Expecting include/select to be applied
Wrong:
```typescript
const input = parseSpatieRequest(request.qs())
applyFilter(query, input, { allowed: ['name'] })
return query.paginate(1, 25) // ?include=posts ignored — posts never loaded
```
Correct:
```typescript
const input = parseSpatieRequest(request.qs())
applyFilter(query, input, { allowed: ['name'] })
for (const relation of input.include ?? []) {
if (PRELOADABLE.includes(relation)) query.preload(relation as any)
}
return query.paginate(1, 25)
```
`include` and sparse fieldset `select` are parsed but deliberately applied by no
runner — eager-loading which relations are safe stays your validated decision.
The client builder's `.include(...)` is likewise only a request.
Source: docs/guides/client.mdx ("include is a request, not an action"); docs/definitions/request-input.mdx
### [HIGH] Confusing full-text search with vector similarity
Wrong:
```typescript
defineFilter({ fullText: { column: 'embedding' } }) // tsvector @@ against a vector column
```
Correct:
```typescript
defineFilter({
vectorSimilarity: { column: 'embedding', metric: 'cosine' }, // embeddings rank nearest-first
// fullText: { column: 'search_vector' }, // keywords match lexemes
})
```
Full-text matches words against a tsvector document; vector similarity ranks by
embedding distance. Configuring one while sending the other's input yields
silently wrong search semantics. They are additive — filter by keywords, rank
by embedding.
Source: docs/search/index.mdx ("Full-text search ≠ vector similarity")
### [MEDIUM] Trusting paginate() totals alongside distinct
Wrong:
```typescript
const result = await query.distinct('status').paginate(page, size)
result.totalCount // counts UN-deduped rows
```
Correct:
```typescript
// run your own countDistinct when the total must match the deduped page
```
Lucid's count leg rebuilds the SELECT, so the reported total counts un-deduped
rows while the page itself is deduped.
Source: docs/guides/lucid.mdx (distinct/paginate total callout)
### [MEDIUM] Wrapping large text columns in query-time to_tsvector
Wrong:
```typescript
fullText: { column: ['title', 'body'], columnKind: 'text' } // re-tokenizes every row
```
Correct:
```typescript
fullText: { column: 'search_vector', columnKind: 'tsvector' } // GIN-indexable
```
Query-time `to_tsvector` cannot use an index — fine for small tables, a
sequential scan on anything big.
Source: docs/search/full-text.mdx (text-columns warning)
See also: `../filter-definitions/SKILL.md` (where fullText/vectorSimilarity are
declared), `../filter-safety/SKILL.md` (the allow-list these paths respect).