mpackdb/README.md

A simple, local, binary json (using MessagePack) database with binary search index.

# MPackDB

A fast, local, append-only JSON database with [MessagePack](https://msgpack.org/) serialization for Node.js.

## Features

- **🚀 High Performance** - Append-only writes with MessagePack binary serialization
- **📇 Flexible Indexing** - Optional numeric and lexical indexes for fast queries
- **🔒 Concurrent Access** - File-based locking for safe multi-process access
- **💾 Auto-Persistence** - Configurable automatic index persistence
- **🗜️ Auto-Compaction** - Removes deleted records on startup
- **🎯 Simple API** - Intuitive CRUD operations with async/await
- **📦 Zero Dependencies** - Only requires `msgpackr`

## Installation

```bash
npm install @worldapi.org/mpackdb
```

## Quick Start

```javascript
import MPackDB from '@worldapi.org/mpackdb';

// Create a database with auto-increment numeric primary key
const db = new MPackDB('data/users', {
  primaryKey: '*id',  // * prefix = numeric auto-increment
  indexes: ['email', '*age']  // Index email (lexical) and age (numeric)
});

// Insert records
await db.insert({ name: 'Alice', email: 'alice@example.com', age: 30 });
await db.insert({ name: 'Bob', email: 'bob@example.com', age: 25 });

// Find all records
for await (const user of db.find()) {
  console.log(user);
}

// Find by primary key
const users = await db.find(0);  // Returns array with Alice

// Query with function
for await (const user of db.find(r => r.age > 28)) {
  console.log(user.name);  // Alice
}

// Update records
await db.update(0, { age: 31 });
await db.update(r => r.age < 26, { status: 'junior' });

// Delete records
await db.delete(r => r.age < 18);

// Close database (persists all changes)
await db.close();
```

## API Reference

### Constructor

```javascript
new MPackDB(dbFile, options)
```

**Parameters:**
- `dbFile` (string): Path to database file (without extension)
- `options` (object):
  - `primaryKey` (string): Primary key field name with optional prefix:
    - `*field` - Numeric auto-increment (e.g., `*id`)
    - `@field` - UUID (e.g., `@uuid`)
    - `field` - String (e.g., `username`)
  - `primaryKeyType` (PrimaryKeyType): Explicit type override
  - `indexes` (string[]): Fields to index (use prefixes like primary key)
  - `debug` (boolean): Enable debug logging (default: `false`)
  - `indexPersistInterval` (number): Auto-persist interval in ms (default: `60000`, `0` to disable)
  - `indexPersistThreshold` (number): Auto-persist after N changes (default: `1000`)

**Examples:**
functional style:
```javascript
import MPackDB from '@worldapi.org/mpackdb';

const db = new MPackDB('data/products', {
  primaryKey: '*id',
  indexes: ['category', '*price', '@sku'],
  indexPersistThreshold: 100
});
```

OOP style:
```javascript
import { MPackDB, Model, PrimaryKeyType } from '@worldapi.org/mpackdb';

export class Product extends Model {
  id = 0;
  name = '';
  price = 0;
  category = '';
  sku = '';
}

export class Products extends MPackDB {
  _classToUse = Product;
  _primaryKey = 'id';
  _primaryKeyType = PrimaryKeyType.UUID;
  _indexes = ['category', '*price', '@sku'];
}

export default products = new Products('data/products');
```

### insert(record, options)

Insert a new record into the database.

```javascript
await db.insert({ name: 'Alice', age: 30 });
// Returns: { id: 0, name: 'Alice', age: 30 }
```

**Parameters:**
- `record` (object): The record to insert
- `options` (object):
  - `skipPrimaryKey` (boolean): Don't auto-generate primary key

**Returns:** Promise<Object> - The inserted record with primary key

### find(query, options)

Find records in the database. Returns an async iterable Cursor.

```javascript
// Find all
for await (const record of db.find()) {
  console.log(record);
}

// Find by primary key
const users = await db.find(0);

// Find with query function
for await (const user of db.find(r => r.age > 30)) {
  console.log(user.name);
}
```

**Parameters:**
- `query` (undefined|string|Function):
  - `undefined/null` - Returns all records
  - `string` - Primary key value
  - `Function` - Query function `(record) => boolean`
- `options` (object):
  - `mode` (string): Return mode - `'record'`, `'raw'`, or `'mixed'`

**Returns:** Cursor (async iterable)

### update(query, data, options)

Update records matching a query.

```javascript
// Update by primary key
await db.update(0, { age: 31 });

// Update with query function
await db.update(r => r.age > 30, { status: 'senior' });

// Update with callback
await db.update(r => r.age > 30, r => ({ ...r, age: r.age + 1 }));
```

**Parameters:**
- `query` (string|Function): Primary key or query function
- `data` (object|Function): Data to update or callback function
- `options` (object):
  - `upsert` (boolean): Insert if no records match

**Returns:** Promise<Object[]> - Array of updated records

### upsert(query, data)

Update records or insert if not found.

```javascript
await db.upsert(r => r.email === 'alice@example.com', {
  name: 'Alice',
  email: 'alice@example.com',
  age: 30
});
```

### delete(query, callback)

Delete records matching a query.

```javascript
// Delete by primary key
await db.delete(0);

// Delete with query function
await db.delete(r => r.age < 18);

// Delete with callback
await db.delete(r => r.age < 18, async (record) => {
  console.log('Deleting:', record.name);
  return record;
});
```

**Parameters:**
- `query` (string|Function): Primary key or query function
- `callback` (Function): Optional callback for each deleted record

**Returns:** Promise<Object[]> - Array of deleted records

### compact()

Compact the database by removing deleted records. This rewrites the data file without tombstones.

```javascript
await db.compact();
```

**Note:** Compaction happens automatically on database initialization.

### close()

Close the database and persist all pending changes. Should be called before process exit.

```javascript
await db.close();
```

## Primary Key Types

MPackDB supports three primary key types:

### Numeric (Auto-increment)

```javascript
const db = new MPackDB('data/users', {
  primaryKey: '*id'  // * prefix
});

await db.insert({ name: 'Alice' });
// { id: 0, name: 'Alice' }
```

### UUID

```javascript
const db = new MPackDB('data/sessions', {
  primaryKey: '@sessionId'  // @ prefix
});

await db.insert({ data: 'session data' });
// { sessionId: 'a1b2c3d4-...', data: 'session data' }
```

### String

```javascript
const db = new MPackDB('data/users', {
  primaryKey: 'username'  // No prefix
});

await db.insert({ username: 'alice', name: 'Alice' });
// { username: 'alice', name: 'Alice' }
```

## Indexes

Indexes dramatically improve query performance for large datasets.

### Index Types

- **Lexical** (default): String sorting, good for text fields
- **Numeric**: Number sorting, good for integers/floats
- **UUID**: Treated as lexical (string)

### Creating Indexes

```javascript
const db = new MPackDB('data/products', {
  primaryKey: '*id',
  indexes: [
    'category',      // Lexical index
    '*price',        // Numeric index
    '*stock',        // Numeric index
    '@sku'           // UUID index (lexical)
  ]
});
```

### Index Persistence

Indexes are automatically persisted based on:
- **Threshold**: After N changes (default: 1000)
- **Interval**: Every N milliseconds (default: 60000)
- **On close**: When `db.close()` is called

```javascript
const db = new MPackDB('data/users', {
  primaryKey: '*id',
  indexes: ['email'],
  indexPersistThreshold: 100,  // Persist after 100 changes
  indexPersistInterval: 30000  // Persist every 30 seconds
});
```

## File Structure

MPackDB creates the following files:

```
data/
  users.mpack          # Main data file (MessagePack binary)
  users.meta.json       # Metadata (nextId, deleted offsets)
  users.id.txt          # Primary key index
  users.email.txt       # Email field index
  users.age.txt         # Age field index
  users.lock            # Lock file (temporary)
```

## Concurrency

MPackDB uses file-based locking to ensure safe concurrent access:

```javascript
// Process 1
const db1 = new MPackDB('data/users', { primaryKey: '*id' });
await db1.insert({ name: 'Alice' });

// Process 2 (waits for lock)
const db2 = new MPackDB('data/users', { primaryKey: '*id' });
await db2.insert({ name: 'Bob' });
```

## Performance Tips

1. **Use indexes** for frequently queried fields
2. **Adjust persist thresholds** based on your write patterns
3. **Call `compact()`** periodically if you have many deletes
4. **Use numeric indexes** for number fields
5. **Batch operations** when possible

## Examples

### User Management System

```javascript
import MPackDB from '@worldapi.org/mpackdb';

const users = new MPackDB('data/users', {
  primaryKey: '*id',
  indexes: ['email', '*age', 'role']
});

// Register user
await users.insert({
  email: 'alice@example.com',
  name: 'Alice',
  age: 30,
  role: 'admin'
});

// Find by email
for await (const user of users.find(u => u.email === 'alice@example.com')) {
  console.log('Found user:', user.name);
}

// Get all admins
for await (const admin of users.find(u => u.role === 'admin')) {
  console.log('Admin:', admin.name);
}

// Update age
await users.update(u => u.email === 'alice@example.com', { age: 31 });

// Delete inactive users
await users.delete(u => u.lastLogin < Date.now() - 90 * 24 * 60 * 60 * 1000);

await users.close();
```

### Product Catalog

```javascript
const products = new MPackDB('data/products', {
  primaryKey: '*id',
  indexes: ['category', '*price', '@sku']
});

// Add products
await products.insert({ sku: 'ABC-123', name: 'Laptop', category: 'Electronics', price: 999 });
await products.insert({ sku: 'DEF-456', name: 'Mouse', category: 'Electronics', price: 29 });

// Find by category
for await (const product of products.find(p => p.category === 'Electronics')) {
  console.log(product.name, product.price);
}

// Find products under $50
for await (const product of products.find(p => p.price < 50)) {
  console.log('Affordable:', product.name);
}

// Update price
await products.update(p => p.sku === 'ABC-123', { price: 899 });

await products.close();
```

## License

MIT

## Contributing

Contributions are welcome! Please open an issue or submit a pull request.

## Related Projects

- [**BsonDB**](https://bsondb.gitoria.worldapi.org) - Similar database using BSON serialization