deepbase/README.md

⚡ DeepBase - Fastest and simplest way to add persistence to your projects.

# deepbase

DeepBase - Multi-driver persistence system with JSON driver included.

## Installation

```bash
npm install deepbase
# Automatically includes deepbase-json as dependency
```

> **AI Skill**: You can also add DeepBase as a skill for AI agentic development:
> ```bash
> npx skills add https://github.com/clasen/DeepBase --skill deepbase
> ```

## What is DeepBase?

DeepBase is a powerful database abstraction that orchestrates multiple storage drivers. It includes:

- **JSON driver included**: `deepbase-json` comes as a dependency for filesystem storage
- **Multi-driver management**: Use multiple storage backends simultaneously
- **Automatic fallback**: Read from first available driver
- **Replication**: Write to all drivers or just primary
- **Migration**: Built-in data migration between drivers
- **Driver interface**: Base class for creating custom drivers

## Quick Start

### Simple Usage (JSON Driver - Built-in!)

```javascript
import DeepBase from 'deepbase';

// Backward-compatible syntax - uses JSON driver by default
const db = new DeepBase({ path: './data', name: 'mydb' });
await db.connect();

await db.set('users', 'alice', { name: 'Alice' });
const alice = await db.get('users', 'alice');
```

### Explicit Driver Usage

```javascript
import DeepBase, { JsonDriver } from 'deepbase';

const db = new DeepBase(new JsonDriver({ path: './data' }));
await db.connect();

await db.set('users', 'alice', { name: 'Alice' });
const alice = await db.get('users', 'alice');
```

## Additional Drivers

The JSON driver (`deepbase-json`) is included automatically. Install additional drivers as needed:

- [`deepbase-sqlite`](https://www.npmjs.com/package/deepbase-sqlite) - SQLite embedded database
- [`deepbase-indexeddb`](https://www.npmjs.com/package/deepbase-indexeddb) - IndexedDB for browser environments
- [`deepbase-drizzle`](https://www.npmjs.com/package/deepbase-drizzle) - [Drizzle ORM](https://orm.drizzle.team/) — SQLite, PostgreSQL, MySQL, and other dialects via your Drizzle `db` (default table schema is inferred)
- [`deepbase-mongodb`](https://www.npmjs.com/package/deepbase-mongodb) - MongoDB storage
- [`deepbase-redis`](https://www.npmjs.com/package/deepbase-redis) - Redis Stack storage

### Drizzle Example (PostgreSQL / MySQL / Supabase)

```javascript
import DeepBase from 'deepbase';
import DrizzleDriver from 'deepbase-drizzle';
import { drizzle } from 'drizzle-orm/node-postgres';
import { Pool } from 'pg';

const pool = new Pool({ connectionString: process.env.DATABASE_URL });
const db = drizzle({ client: pool });

const store = new DeepBase(new DrizzleDriver({ db, client: pool }));
await store.connect(); // creates table automatically if it does not exist
await store.set('users', 'alice', { name: 'Alice' });
```

Same pattern works with other Drizzle dialects (`mysql2`, SQLite, etc.).  
For Supabase Postgres, use the Supabase connection string in `DATABASE_URL`.  
Optional: set `tableName` if you want a different table than `deepbase_main`.

### Multi-Driver Example

```javascript
import DeepBase, { JsonDriver } from 'deepbase';
import MongoDriver from 'deepbase-mongodb';

const db = new DeepBase([
  new MongoDriver({ url: 'mongodb://localhost:27017' }),
  new JsonDriver({ path: './backup' })
], {
  writeAll: true,           // Write to both drivers
  readFirst: true,          // Read from first available
  failOnPrimaryError: false // Continue if MongoDB fails
});

await db.connect();
```

## Timeout Configuration

Prevent operations from hanging indefinitely with configurable timeouts:

```javascript
import DeepBase, { JsonDriver } from 'deepbase';

// Global timeout for all operations
const db = new DeepBase(new JsonDriver(), {
  timeout: 5000  // 5 seconds
});

// Different timeouts for reads and writes
const db2 = new DeepBase(new JsonDriver(), {
  readTimeout: 3000,   // 3 seconds for reads
  writeTimeout: 10000  // 10 seconds for writes
});

// All operations will timeout if they exceed the limit
try {
  await db.get('some', 'key');
} catch (error) {
  // Error: get() timed out after 5000ms
  console.error(error.message);
}
```

**Timeout Options:**
- `timeout`: Global timeout for all operations (default: `0` = disabled)
- `readTimeout`: Timeout for `get`, `keys`, `values`, `entries` (default: `timeout`)
- `writeTimeout`: Timeout for `set`, `del`, `inc`, `dec`, `add`, `upd` (default: `timeout`)
- `connectTimeout`: Timeout for connection operation (default: `timeout`)

See [TIMEOUT_FEATURE.md](https://github.com/clasen/DeepBase/blob/main/TIMEOUT_FEATURE.md) for detailed documentation.

## API

### Constructor

```javascript
new DeepBase(drivers, options)
```

**Parameters:**
- `drivers`: Single driver or array of drivers
- `options`:
  - `writeAll` (default: `true`): Write to all drivers
  - `readFirst` (default: `true`): Read from first available
  - `failOnPrimaryError` (default: `true`): Throw on primary failure
  - `lazyConnect` (default: `true`): Auto-connect on first operation
  - `timeout` (default: `0`): Global timeout in ms (0 = disabled)
  - `readTimeout` (default: `timeout`): Timeout for read operations in ms
  - `writeTimeout` (default: `timeout`): Timeout for write operations in ms
  - `connectTimeout` (default: `timeout`): Timeout for connection in ms

### Methods

#### Connection
- `await db.connect()` - Connect all drivers
- `await db.disconnect()` - Disconnect all drivers

#### Data Operations
- `await db.get(...path)` - Get value at path
- `await db.set(...path, value)` - Set value at path
- `await db.del(...path)` - Delete value at path
- `await db.inc(...path, amount)` - Increment value
- `await db.dec(...path, amount)` - Decrement value
- `await db.add(...path, value)` - Add with auto-generated ID (consistent across all drivers)
- `await db.upd(...path, fn)` - Update with function

#### Queue / Stack Operations
- `await db.pop(...path)` - Remove and return the last item
- `await db.shift(...path)` - Remove and return the first item


#### Query Operations
- `await db.keys(...path)` - Get keys at path
- `await db.first(...path)` - Get the first key at path (same driver read order as `get()`)
- `await db.last(...path)` - Get the last key at path (same driver read order as `get()`)
- `await db.values(...path)` - Get values at path
- `await db.entries(...path)` - Get entries at path
- `await db.len(...path)` - Count the number of keys at path

#### Migration
- `await db.migrate(fromIndex, toIndex, options)` - Migrate data between drivers
- `await db.syncAll(options)` - Sync primary to all others

**Migration Options:**
- `clear` (default: `true`): Clear target driver before migration
- `batchSize` (default: `100`): Progress callback frequency (items between calls)
- `onProgress`: Callback `({ migrated, errors, current }) => {}` for monitoring

**Returns:** `{ migrated, errors }` with counts of successful and failed items.

```javascript
import DeepBase, { JsonDriver } from 'deepbase';
import SqliteDriver from 'deepbase-sqlite';

const db = new DeepBase([
  new SqliteDriver({ path: './data', name: 'mydb' }),  // index 0
  new JsonDriver({ path: './backup', name: 'mydb' })   // index 1
]);
await db.connect();

// Migrate SQLite → JSON (clears target first)
const result = await db.migrate(0, 1);
console.log(result); // { migrated: 5, errors: 0 }

// Migrate JSON → SQLite (merge, keep existing data)
await db.migrate(1, 0, { clear: false });

// Migrate with progress reporting
await db.migrate(0, 1, {
  batchSize: 10,
  onProgress: ({ migrated, errors, current }) => {
    console.log(`${migrated} migrated | ${errors} errors | current: ${current}`);
  }
});

// Sync primary (index 0) to all other drivers
await db.syncAll();
```

#### Driver Management
- `db.getDriver(index)` - Get specific driver
- `db.getDrivers()` - Get all drivers

## Queue & Stack

Use `add` + `shift` as a **FIFO queue**, or `add` + `pop` as a **LIFO stack**:

```javascript
// FIFO Queue
await db.add('jobs', { task: 'send-email', to: 'alice@example.com' });
await db.add('jobs', { task: 'resize-image', file: 'photo.jpg' });
await db.add('jobs', { task: 'notify', channel: '#general' });

const next = await db.shift('jobs'); // { task: 'send-email', ... }
await db.len('jobs'); // 2

// LIFO Stack
await db.add('undo', { action: 'delete', id: 42 });
await db.add('undo', { action: 'edit', id: 7 });

const last = await db.pop('undo'); // { action: 'edit', id: 7 }
```

In multi-driver mode, `add` generates a single ID shared across all drivers, so `pop` and `shift` stay consistent regardless of the number of backends.

## Creating Custom Drivers

Extend the `DeepBaseDriver` class:

```javascript
import { DeepBaseDriver } from '@deepbase/core';

class MyDriver extends DeepBaseDriver {
  async connect() { /* ... */ }
  async disconnect() { /* ... */ }
  async get(...args) { /* ... */ }
  async set(...args) { /* ... */ }
  async del(...args) { /* ... */ }
  async inc(...args) { /* ... */ }
  async dec(...args) { /* ... */ }
  async add(...args) { /* ... */ }
  async upd(...args) { /* ... */ }
  async first(...args) { /* optional optimization; fallback exists in base class */ }
  async last(...args) { /* optional optimization; fallback exists in base class */ }
}
```

`DeepBase.first()` / `DeepBase.last()` delegate to each driver's `first()` / `last()` and follow the same read policy as `get()` (`readFirst` order or `Promise.any` mode). `shift()` / `pop()` use these methods, and drivers without overrides still work through the base fallback to `keys()`.

## 🔐 Extending DeepBase with Encryption

DeepBase supports custom `stringify`/`parse` functions in its `JsonDriver`, making it easy to add transparent AES encryption:

```javascript
import CryptoJS from 'crypto-js';
import DeepBase from 'deepbase';
import { JsonDriver } from 'deepbase-json';

class DeepbaseSecure extends DeepBase {
    constructor(opts) {
        const encryptionKey = opts.encryptionKey;
        delete opts.encryptionKey;

        const driver = new JsonDriver({
            ...opts,
            stringify: (obj) => {
                const iv = CryptoJS.lib.WordArray.random(128 / 8);
                const encrypted = CryptoJS.AES.encrypt(JSON.stringify(obj), encryptionKey, { iv });
                return iv.toString(CryptoJS.enc.Hex) + ':' + encrypted.toString();
            },
            parse: (encryptedData) => {
                const [ivHex, encrypted] = encryptedData.split(':');
                const iv = CryptoJS.enc.Hex.parse(ivHex);
                const bytes = CryptoJS.AES.decrypt(encrypted, encryptionKey, { iv });
                return JSON.parse(bytes.toString(CryptoJS.enc.Utf8));
            }
        });

        super(driver);
    }
}

const db = new DeepbaseSecure({ name: 'secrets', encryptionKey: 'my-key-123' });
await db.set('token', 'sk-super-secret');
console.log(await db.get('token')); // 'sk-super-secret' (file on disk is encrypted)
```

## License

MIT - Copyright (c) Martin Clasen