nodedb-json/README.md

A lightweight JSON-based database for Node.js with TypeScript support, indexing, and complex query capabilities

# nodedb-json

A small JSON-file database for Node.js. It is useful for local tools, examples,
desktop apps, small services, tests, and configuration-like data where a full
database server would be too much.

It stores one JSON document on disk and gives you CRUD helpers, array operations,
indexes, batch writes, and query helpers.

## Install

```bash
npm install nodedb-json
```

## Quick Start

```js
const NodedbJson = require('nodedb-json');

const db = new NodedbJson('data/db.json', {
  defaultValue: { users: [] }
});

db.createIndex('users', { field: 'id', type: 'unique' });

db.push('users', { id: 1, name: 'Bob', age: 31 });
db.update('users', user => user.id === 1, { role: 'admin' });

const user = db.findByField('users', 'id', 1);
console.log(user);
```

TypeScript can type the root document with a schema:

```ts
import NodedbJson, { DuplicateIndexError } from 'nodedb-json';

type User = {
  id: number;
  name: string;
  age?: number;
};

type AppConfig = {
  theme: string;
  language?: string;
};

interface Schema {
  users: User[];
  config: AppConfig;
}

const db = new NodedbJson<Schema>('data/db.json', {
  defaultValue: {
    users: [],
    config: { theme: 'dark' }
  }
});

const users = db.collection('users');

try {
  users.createIndex({ field: 'id', type: 'unique' });
  users.insert({ id: 1, name: 'Bob' });

  const user = users.findOne({ id: 1 }); // User | undefined
  const config = db.get('config'); // AppConfig
} catch (error) {
  if (error instanceof DuplicateIndexError) {
    console.error(error.message);
  }
}
```

## What It Does

- Reads and writes a JSON file.
- Creates missing parent directories automatically.
- Writes through a same-directory temp file and `rename`.
- Keeps a `.bak` backup by default and can restore from it if the JSON file is corrupt.
- Creates a `.lock` file by default to prevent two live writers from opening the same database.
- Supports dot-path keys such as `config.theme`.
- Returns deep copies from read APIs so callers cannot mutate internal state by accident.
- Supports `unique` and `multi` indexes for array fields.
- Persists index definitions to a `.meta.json` file by default.
- Supports `NodedbJson<Schema>` and typed `collection()` helpers in TypeScript.
- Includes Vitest coverage and GitHub Actions for Node 18, 20, and 22.

## Important Limits

This package is still a single-file, synchronous JSON database.

- It is best used by one process / one writer at a time.
- File locking is cooperative and based on a sibling `.lock` file.
- Call `close()` when a long-lived instance is done so the lock is released.
- If `fileLock: false` is used, two live instances can overwrite each other's in-memory snapshots.
- Every save serializes the whole JSON document.
- For many small writes, use `batch()` or `autoSave: false` plus `flush()`.
- The package publishes separate ESM and CommonJS entries through `exports`.

## Constructor Options

```ts
type DbOptions = {
  autoSave?: boolean;          // default: true
  createIfNotExists?: boolean; // default: true
  defaultValue?: Partial<TSchema>;

  enableIndexing?: boolean;    // default: true
  autoIndex?: boolean;         // default: true
  indexValueMode?: 'strict' | 'coerce'; // default: 'strict'
  indexes?: IndexOptions;
  persistIndexes?: boolean;    // default: true
  indexMetaPath?: string;      // default: `${dbFile}.meta.json`

  atomicWrites?: boolean;      // default: true
  backupOnWrite?: boolean;     // default: true
  backupPath?: string;         // default: `${dbFile}.bak`
  tempPath?: string;           // default: `${dbFile}.tmp`

  fileLock?: boolean;          // default: true
  lockPath?: string;           // default: `${dbFile}.lock`
  staleLockMs?: number;        // remove old locks only when the recorded PID is gone
};
```

Example:

```js
const db = new NodedbJson('data/db.json', {
  autoSave: false,
  defaultValue: {
    users: [],
    config: { theme: 'dark' }
  },
  indexes: {
    users: [
      { field: 'id', type: 'unique' },
      { field: 'department', type: 'multi' }
    ]
  }
});
```

## Basic Usage

### Values And Objects

```js
db.set('config.theme', 'dark');
db.set('config.language', 'zh-CN');

console.log(db.get('config.theme')); // dark
console.log(db.has('config.language')); // true

db.update('config', { timezone: 'Asia/Shanghai' });
db.delete('config', ['language']);
```

`get()` returns a deep copy:

```js
const config = db.get('config');
config.theme = 'light';

console.log(db.get('config.theme')); // still "dark"
```

For controlled in-place style changes, use `mutate()`:

```js
db.mutate('config', config => {
  config.theme = 'light';
});
```

`getUnsafeReference()` exists for advanced cases, but mutating its return value
bypasses change tracking, auto-save, and index refreshes.

### Arrays

```js
db.push('users', { id: 1, name: 'Bob', age: 31 });
db.push('users', [
  { id: 2, name: 'Charlie', age: 35 },
  { id: 3, name: 'Dave', age: 40 }
]);

const user = db.find('users', user => user.id === 2);
const adults = db.filter('users', user => user.age >= 18);

db.update('users', user => user.id === 1, { role: 'admin' });
db.delete('users', [2, 3], 'id');
```

### Collections

`collection()` is a small typed wrapper around an array path. It keeps common
array operations readable without changing the stored JSON shape:

```ts
const users = db.collection<User>('users');

users.insert({ id: 1, name: 'Bob' });
users.updateOne({ id: 1 }, { name: 'Robert' });
users.deleteOne({ id: 1 });

const activeUsers = users.findMany(user => user.active === true);
```

When the database has a schema, the item type is inferred:

```ts
const db = new NodedbJson<Schema>('data/db.json');
const users = db.collection('users'); // Collection<User>
```

### Manual Save

`autoSave` is enabled by default. Disable it when you want several changes to be
written once:

```js
const db = new NodedbJson('data/db.json', { autoSave: false });

db.set('config.theme', 'dark');
db.push('logs', { time: new Date().toISOString(), action: 'updated config' });

db.flush();
```

### Lifecycle

```js
const db = new NodedbJson('data/db.json', {
  autoSave: false,
  staleLockMs: 60_000
});

db.push('users', { id: 1 });
db.flush();  // force pending changes to disk
db.reload(); // read the JSON file again and rebuild indexes
db.close();  // flush and release the lock
```

Opening a database creates `db.json.lock` by default. A second instance opening
the same file throws `FileLockError` until the first one calls `close()`.
If `close()` cannot save pending changes, the instance stays open and keeps the
lock so callers can retry `flush()` or `close()`. Use `close({ force: true })`
only when you intentionally want to discard pending in-memory changes and
release the lock.
When `staleLockMs` is set, an old lock is removed only if its recorded process
is no longer alive.

If the main JSON file is corrupt and a valid `.bak` exists, the backup is
restored automatically. If both are invalid, construction throws
`CorruptedFileError`.

### Batch

`batch()` runs allowed write operations with `autoSave` temporarily disabled. If
one operation throws, in-memory data and indexes are rolled back.

```js
db.batch([
  { method: 'set', args: ['config.theme', 'dark'] },
  { method: 'push', args: ['logs', { action: 'config updated' }] }
]);
```

Allowed batch methods are `set`, `update`, `delete`, `push`, and `mutate`.

## Indexes

Indexes are for arrays of objects.

```js
db.createIndex('users', { field: 'id', type: 'unique' });
db.createIndex('users', { field: 'department', type: 'multi' });

const bob = db.findByField('users', 'id', 1);
const devUsers = db.filterByField('users', 'department', ['dev']);

db.dropIndex('users', 'department');
console.log(db.getIndexes());
```

Index types:

- `unique`: one item per field value. Duplicate non-null values throw `DuplicateIndexError`.
- `multi`: many items can share the same field value.

Index definitions created with `createIndex()` are persisted by default. You can
also define them in options:

```js
const db = new NodedbJson('data/db.json', {
  indexes: {
    users: [
      { field: 'id', type: 'unique' },
      { field: 'department', type: 'multi' }
    ]
  }
});
```

Index values are type-strict by default. A numeric `1` and a string `"1"` are
different index values, which keeps indexed lookups consistent with non-indexed
`===` lookups:

```js
db.findByField('users', 'id', 1);   // matches { id: 1 }
db.findByField('users', 'id', '1'); // matches { id: '1' }
```

For compatibility with older releases, set `indexValueMode: 'coerce'` to use the
old string-coercion behavior.

## Query

Use `query()` for filtering, sorting, pagination, projection, and aggregation.
Queries run as an in-memory eager pipeline over the current array.

```js
const result = db.query('users', {
  where: {
    department: 'dev',
    age: { $gte: 18, $lt: 60 },
    status: { $in: ['active', 'pending'] }
  },
  sort: { field: 'age', direction: 'desc' },
  pagination: { page: 1, pageSize: 10 },
  select: ['id', 'name', 'age'],
  aggregation: [
    { type: 'count' },
    { type: 'avg', field: 'age' }
  ]
});

console.log(result.data);
console.log(result.pagination);
console.log(result.aggregations);
console.log(result.stats.usedIndex);
```

Object `where` conditions can use indexes when indexed equality or `$in` fields
are present. Multiple indexed fields are intersected before the remaining
conditions are checked. Function predicates are supported too, but they scan the
collection:

```js
const result = db.query('users', {
  where: user => user.age >= 18 && user.active === true
});
```

Supported object operators:

```js
db.query('users', {
  where: {
    age: { $gte: 18, $lt: 60 },
    status: { $in: ['active', 'pending'] },
    name: { $startsWith: '张' }
  }
});
```

Operators: `$eq`, `$ne`, `$gt`, `$gte`, `$lt`, `$lte`, `$in`, `$nin`,
`$startsWith`, `$endsWith`, `$contains`, and `$regex`.

`skip` must be a non-negative integer. `limit`, `pagination.page`, and
`pagination.pageSize` must be positive integers. When `skip`, `limit`, and
`pagination` are used together, the order is:

```txt
where -> sort -> skip -> limit -> pagination -> select
```

Aggregations run against the filtered data before `sort`, `skip`, `limit`,
`pagination`, and `select` are applied.

In TypeScript, literal `select` fields narrow the returned item type:

```ts
const result = db.query<User>('users', {
  select: ['id'] as const
});

result.data[0].id;   // ok
result.data[0].name; // TypeScript error
```

Dynamic `string[]` selects and nested path selects stay conservatively typed.

Convenience helpers:

```js
db.orderBy('users', { field: 'age', direction: 'desc' }, 5);
db.paginate('users', 1, 20, { department: 'dev' });
db.count('users', { active: true });
db.aggregate('users', [{ type: 'max', field: 'age' }]);
db.distinct('users', 'department');
```

## 1.4.1 Notes

This is a correctness patch for the 1.4.x line.

Fixed:

- Collection object conditions now use the same matcher as root queries.
- Persisted index definitions no longer cause false indexed lookups when
  `autoIndex: false`.
- Nested object field deletion works with dot-path keys.
- `close()` no longer closes the instance or releases the lock if saving fails.
- `staleLockMs` no longer removes a lock owned by a live process.
- Group aggregations and `distinct()` no longer expose internal references.
- `$regex` queries are isolated from stateful `g` / `y` RegExp `lastIndex`.
- `createIndex()` and `dropIndex()` roll back in-memory index state if metadata
  persistence fails.
- TypeScript `select` results now reflect partial selected records.

Usage impact:

- Normal runtime usage is unchanged.
- `close({ force: true })` is available when you explicitly want to discard
  pending in-memory changes after a close/save failure.
- TypeScript users may now see compile errors when accessing fields that were
  not selected by a literal `select` query. The runtime already returned
  partial objects; the type now matches that behavior.

## 1.4.0 Reliability Notes

The reliability work adds:

- Atomic writes, backup recovery, and parent directory creation.
- Persistent index metadata.
- File locking through a sibling `.lock` file.
- `flush()`, `reload()`, and `close()`.
- `DatabaseError`, `CorruptedFileError`, and `FileLockError`.

Remaining limits:

- The database is still synchronous and single-file.
- File locking is cooperative and local-filesystem oriented.
- There is no cross-process transaction isolation.

## 1.3.1 Notes

This release focuses on correctness and test coverage.

Fixed:

- Multi-value indexes now correctly return items stored at array position `0`.
- Indexed batch deletion no longer deletes the wrong items after array indexes shift.
- Unique indexes now reject duplicate values instead of silently overwriting entries.
- Object updates now merge plain objects as documented.
- Indexed object updates now verify all query conditions before modifying an item.
- Read APIs return deep copies by default.
- Tests now fail properly through Vitest and CI.

Also included:

- Atomic JSON writes with temp files and `rename`.
- Backup creation and corrupt JSON recovery from `.bak`.
- Parent directory creation.
- Persistent index metadata.

## Development

```bash
npm run lint
npm run typecheck
npm test
npm run build
npm pack --dry-run
```

## License

MIT

## Support

Issues: <https://github.com/zxbb1190/nodedb-json/issues>

Email: <mailto:douyaj33@gmail.com>