@joystick.js/db-canary/README.md

JoystickDB - A minimalist database server for the Joystick framework

# JoystickDB

A high-performance, production-ready TCP-based database server built on LMDB with comprehensive client libraries, multi-database support, replication, and enterprise features for Node.js applications.

## Table of Contents

- [Overview](#overview)
- [Features](#features)
- [Installation](#installation)
- [Quick Start](#quick-start)
- [Server Setup](#server-setup)
- [Client Usage](#client-usage)
- [Multi-Database Operations](#multi-database-operations)
- [Database Operations](#database-operations)
- [Indexing](#indexing)
- [HTTP API](#http-api)
- [Replication](#replication)
- [Administration](#administration)
- [Backup & Restore](#backup--restore)
- [Configuration](#configuration)
- [Production Deployment](#production-deployment)
- [API Reference](#api-reference)

## Overview

JoystickDB is a lightweight, blazing-fast database server that provides MongoDB-like operations over TCP using MessagePack for efficient data serialization. Built on top of LMDB (Lightning Memory-Mapped Database), it offers ACID transactions, high performance, minimal resource usage, and enterprise-grade features like multi-database support, replication, and automatic backups.

**Think of it as**: A simpler, faster alternative to MongoDB that's perfect for applications that need reliable data storage without the complexity of a full database cluster, now with full multi-database support for better data organization.

## Features

### Core Database Features
- **Multi-Database Support**: Organize data across multiple isolated databases with MongoDB-like chaining API
- **High Performance**: Built on LMDB with MessagePack serialization for maximum speed
- **MongoDB-like API**: Familiar CRUD operations and query syntax - easy to learn
- **ACID Transactions**: Your data is always consistent and safe
- **TCP Protocol**: Efficient binary communication over the network
- **Automatic Indexing**: Smart index creation based on your query patterns
- **Secondary Indexes**: Create custom indexes for lightning-fast queries
- **Database Isolation**: Complete data separation between databases with proper key namespacing

### Production & Enterprise Features
- **API Key Authentication**: Secure API key-based authentication system with user management
- **HTTP API**: RESTful HTTP endpoints for user management and external integrations
- **TCP Replication**: Master-slave replication for high availability and read scaling
- **Password Authentication**: Secure bcrypt password hashing with rate limiting
- **Role-Based Access Control**: Read, write, and read_write user roles
- **Clustering**: Multi-process clustering support for better performance
- **Backup & Restore**: Automatic S3-compatible backup system with scheduling
- **Admin Interface**: Comprehensive administrative operations and monitoring
- **Connection Pooling**: Efficient connection management and resource usage
- **Health Monitoring**: Built-in performance metrics and health checks
- **Write Serialization**: Ensures data consistency under high concurrent load

### Developer Experience
- **Fluent API**: Chain methods for intuitive database operations
- **Multi-Database Chaining**: `client.db('database').collection('collection')` API pattern
- **Backward Compatibility**: Existing single-database API still supported
- **Auto-Reconnection**: Client automatically reconnects on network issues
- **TypeScript Support**: Full TypeScript definitions included
- **Comprehensive Logging**: Detailed logs for debugging and monitoring
- **Easy Setup**: Get started in minutes with minimal configuration
- **Clean Code Architecture**: Refactored following Uncle Bob Martin's clean code principles for maintainability
- **Modular Design**: Small, focused functions with clear responsibilities and minimal dependencies

## Installation

```bash
npm install @joystick.js/db
```

## Quick Start

### 1. Start the Server

```bash
# Start with default settings (port 1983)
npm start

# Or with custom port
PORT=3000 npm start

# With clustering (uses all CPU cores)
WORKER_COUNT=4 npm start
```

### 2. Setup Authentication (One-time)

When you first start JoystickDB, it automatically generates a secure API key and displays setup instructions:

```
JoystickDB Setup

To finish setting up your database and enable operations for authenticated users, you will need to create an admin user via the database's admin API using the API key displayed below.

=== STORE THIS KEY SECURELY -- IT WILL NOT BE DISPLAYED AGAIN ===
USGwrwK36RM97xiWs6Df1yFuxPLfo4aY
===

To create a user, send a POST request to the server:

fetch('http://localhost:1984/api/users', {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'x-joystick-db-api-key': 'USGwrwK36RM97xiWs6Df1yFuxPLfo4aY'
  },
  body: JSON.stringify({
    username: 'admin',
    password: 'your_secure_password',
    role: 'read_write'
  })
});
```

**Important**: Save the API key securely - it will not be displayed again and is required for user management operations.

### 3. Create Your First Admin User

Use the API key to create an admin user via HTTP:

```javascript
// Create admin user via HTTP API
const response = await fetch('http://localhost:1983/api/users', {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'x-joystick-db-api-key': 'your-api-key-from-step-2'
  },
  body: JSON.stringify({
    username: 'admin',
    password: 'your_secure_password',
    role: 'read_write'
  })
});

const result = await response.json();
console.log('✅ Admin user created:', result);
```

### 4. Connect and Use Your Database

```javascript
import joystickdb from '@joystick.js/db';

const client = joystickdb.client({
  port: 1983,
  authentication: {
    username: 'admin',
    password: 'your_secure_password'
  }
});

client.on('authenticated', async () => {
  console.log('🎉 Connected to JoystickDB!');
  
  // NEW: Multi-database support with MongoDB-like chaining
  const users_db = client.db('user_management');
  const users = users_db.collection('users');
  
  // Insert a document
  const result = await users.insert_one({
    name: 'John Doe',
    email: 'john@example.com',
    age: 30,
    hobbies: ['reading', 'coding', 'gaming']
  });
  
  console.log('✅ User created with ID:', result.inserted_id);
  
  // Find the document
  const user = await users.find_one({ name: 'John Doe' });
  console.log('👤 Found user:', user);
  
  // Update the document
  await users.update_one(
    { name: 'John Doe' },
    { $set: { age: 31 }, $push: { hobbies: 'photography' } }
  );
  
  console.log('✏️ User updated!');
  
  // Work with different databases
  const inventory_db = client.db('inventory');
  const products = inventory_db.collection('products');
  
  await products.insert_one({
    name: 'Laptop',
    price: 999.99,
    category: 'electronics'
  });
  
  console.log('📦 Product added to inventory database!');
});
```

## Server Setup

### Basic Server Configuration

JoystickDB uses the `JOYSTICK_DB_SETTINGS` environment variable for configuration. Set this to a JSON string containing your settings:

```bash
export JOYSTICK_DB_SETTINGS='{
  "port": 1983,
  "database": {
    "path": "./data",
    "auto_map_size": true
  },
  "authentication": {
    "rate_limit": {
      "max_attempts": 5,
      "window_ms": 300000
    }
  },
  "primary": true,
  "secondary_nodes": [
    { "ip": "192.168.1.100" },
    { "ip": "192.168.1.101" }
  ],
  "secondary_sync_key": "/path/to/sync.key",
  "sync_port": 1985,
  "backup": {
    "enabled": true,
    "schedule": "0 2 * * *",
    "retention_days": 30
  },
  "s3": {
    "region": "us-east-1",
    "bucket": "my-database-backups"
  }
}'
```

### Environment Variables

```bash
# Core server configuration via JSON
JOYSTICK_DB_SETTINGS='{"port": 1983, "database": {"path": "./data"}}'

# Additional server options
WORKER_COUNT=4              # Number of worker processes
NODE_ENV=production         # Environment mode

# AWS S3 for backups (optional)
AWS_REGION=us-east-1
AWS_ACCESS_KEY_ID=your-access-key
AWS_SECRET_ACCESS_KEY=your-secret-key
```

### Programmatic Server Creation

```javascript
import { create_server } from '@joystick.js/db/server';

const server = await create_server();

server.listen(1983, () => {
  console.log('🚀 JoystickDB server running on port 1983');
});

// Graceful shutdown
process.on('SIGTERM', async () => {
  console.log('🛑 Shutting down gracefully...');
  await server.cleanup();
  server.close();
});
```

## Client Usage

### Connection Options

```javascript
const client = joystickdb.client({
  host: 'localhost',              // Server hostname
  port: 1983,                     // Server port
  timeout: 5000,                  // Request timeout (5 seconds)
  reconnect: true,                // Auto-reconnect on disconnect
  max_reconnect_attempts: 10,     // Max reconnection attempts
  reconnect_delay: 1000,          // Initial reconnect delay (1 second)
  authentication: {
    username: 'your-username',    // Your database username
    password: 'your-password'     // Your database password
  },
  auto_connect: true              // Connect automatically when created
});
```

### Event Handling

```javascript
client.on('connect', () => {
  console.log('🔌 Connected to JoystickDB');
});

client.on('authenticated', () => {
  console.log('🔐 Authentication successful - ready to use!');
});

client.on('error', (error) => {
  console.error('❌ Client error:', error.message);
});

client.on('disconnect', () => {
  console.log('📡 Disconnected from server');
});

client.on('reconnecting', ({ attempt, delay }) => {
  console.log(`🔄 Reconnecting... attempt ${attempt}, delay ${delay}ms`);
});
```

## Multi-Database Operations

JoystickDB now supports multiple isolated databases, allowing you to organize your data more effectively. Each database maintains complete isolation with separate collections, indexes, and storage.

### Multi-Database API

JoystickDB uses MongoDB's familiar chaining pattern for multi-database operations:

```javascript
// Connect to different databases
const user_db = client.db('user_management');
const inventory_db = client.db('inventory');
const analytics_db = client.db('analytics');

// Each database has its own collections
const users = user_db.collection('users');
const profiles = user_db.collection('profiles');

const products = inventory_db.collection('products');
const categories = inventory_db.collection('categories');

const events = analytics_db.collection('events');
const reports = analytics_db.collection('reports');

// Operations are completely isolated between databases
await users.insert_one({ name: 'John', role: 'admin' });
await products.insert_one({ name: 'Laptop', price: 999 });
await events.insert_one({ type: 'login', user_id: 'john123' });

// Each database maintains separate indexes and statistics
await users.create_index('email');
await products.create_index('category');
await events.create_index('timestamp');
```

### Database Management Operations

```javascript
// List all databases on the server
const databases = await client.list_databases();
console.log('📚 Available databases:', databases.databases);

// Get database-specific statistics
const user_db = client.db('user_management');
const stats = await user_db.get_stats();
console.log('📊 User DB stats:', stats);

// List collections in a specific database
const collections = await user_db.list_collections();
console.log('📋 Collections in user_management:', collections.collections);

// Create a collection explicitly (optional - collections are created automatically)
await user_db.create_collection('audit_logs', {
  // Collection options can be specified here
});

// Drop an entire database (admin operation - use with caution!)
await user_db.drop_database();
console.log('🗑️ Database dropped');
```

### Database Naming Rules

Database names must follow these rules:

- Must be 1-64 characters long
- Can contain letters, numbers, underscores, and hyphens
- Cannot start with a number
- Cannot be reserved names: `admin`, `config`, `local`

```javascript
// Valid database names
const app_db = client.db('my_app');
const user_data = client.db('user-data');
const analytics2024 = client.db('analytics_2024');

// Invalid database names (will throw errors)
// client.db('admin');        // Reserved name
// client.db('123invalid');   // Starts with number
// client.db('');             // Empty name
// client.db('a'.repeat(65)); // Too long
```


## Database Operations

JoystickDB provides a clean, MongoDB-like chaining API for multi-database operations.

### Multi-Database Chaining API

Work with JoystickDB across multiple databases using the intuitive chaining pattern:

```javascript
// Get database and collection references
const user_db = client.db('user_management');
const inventory_db = client.db('inventory');

const users = user_db.collection('users');
const products = inventory_db.collection('products');

// Insert documents into different databases
const user = await users.insert_one({
  name: 'Alice Smith',
  email: 'alice@example.com',
  age: 28,
  tags: ['developer', 'javascript'],
  created_at: new Date()
});

const product = await products.insert_one({
  name: 'Wireless Mouse',
  price: 29.99,
  category: 'electronics',
  stock: 150
});

console.log('👤 Created user in user_management DB:', user.inserted_id);
console.log('📦 Created product in inventory DB:', product.inserted_id);

// Bulk operations across databases
const user_bulk = await users.bulk_write([
  { insert_one: { document: { name: 'Bob', age: 25, role: 'designer' } } },
  { insert_one: { document: { name: 'Carol', age: 30, role: 'manager' } } },
  { insert_one: { document: { name: 'Dave', age: 35, role: 'developer' } } }
]);

const product_bulk = await products.bulk_write([
  { insert_one: { document: { name: 'Keyboard', price: 79.99, category: 'electronics' } } },
  { insert_one: { document: { name: 'Monitor', price: 299.99, category: 'electronics' } } }
]);

console.log('👥 Created users:', user_bulk.inserted_count);
console.log('📦 Created products:', product_bulk.inserted_count);
```

### Finding Documents Across Databases

```javascript
const user_db = client.db('user_management');
const inventory_db = client.db('inventory');

const users = user_db.collection('users');
const products = inventory_db.collection('products');

// Find in user database
const alice = await users.find_one({ name: 'Alice Smith' });
console.log('Found Alice in user DB:', alice);

// Find in inventory database
const electronics = await products.find({ 
  category: 'electronics',
  price: { $lte: 100 }
});
console.log('Found affordable electronics:', electronics.documents);

// Each database maintains separate query performance
const recent_users = await users.find({}, {
  sort: { created_at: -1 },
  limit: 10
});

const expensive_products = await products.find({
  price: { $gte: 200 }
}, {
  sort: { price: -1 },
  limit: 5
});
```

### Updating and Deleting Across Databases

```javascript
const user_db = client.db('user_management');
const inventory_db = client.db('inventory');

const users = user_db.collection('users');
const products = inventory_db.collection('products');

// Update in user database
await users.update_one(
  { name: 'Alice Smith' },
  { 
    $set: { 
      age: 29, 
      last_updated: new Date(),
      status: 'active'
    },
    $push: { tags: 'senior' }
  }
);

// Update in inventory database
await products.update_one(
  { name: 'Wireless Mouse' },
  { 
    $set: { price: 24.99 },
    $inc: { stock: -5 }
  }
);

// Delete from user database
await users.delete_one({ 
  status: 'inactive',
  last_login: { $lt: new Date(Date.now() - 365*24*60*60*1000) }
});

// Delete from inventory database
await products.delete_one({ 
  stock: { $lte: 0 },
  discontinued: true
});

// Delete multiple documents at once
await users.delete_many({ 
  status: 'inactive',
  last_login: { $lt: new Date(Date.now() - 365*24*60*60*1000) }
});

await products.delete_many({ 
  stock: { $lte: 0 }
}, { 
  limit: 100  // Optional: limit number of deletions
});
```


### Query Operators (MongoDB-style)

All query operators work consistently across all databases:

```javascript
const user_db = client.db('user_management');
const inventory_db = client.db('inventory');

const users = user_db.collection('users');
const products = inventory_db.collection('products');

// Comparison operators work in any database
await users.find({ age: { $gt: 25 } });
await products.find({ price: { $gte: 50, $lte: 200 } });

// Text matching with regular expressions
await users.find({ 
  name: { $regex: '^John' }
});

await products.find({ 
  name: { $regex: 'wireless', $options: 'i' }  // Case insensitive
});

// Array operations
await users.find({ 
  tags: { $in: ['developer', 'designer'] }
});

await products.find({ 
  categories: { $all: ['electronics', 'accessories'] }
});

// Complex queries across databases
await users.find({
  age: { $gte: 25, $lte: 35 },
  tags: { $in: ['developer', 'designer'] },
  status: 'active',
  name: { $regex: 'Smith$' }
});

await products.find({
  price: { $gte: 100 },
  category: 'electronics',
  stock: { $gt: 0 },
  name: { $regex: 'pro', $options: 'i' }
});
```

## Indexing

Indexes work independently within each database, providing optimal performance for database-specific query patterns.

### Automatic Indexing (Per Database)

JoystickDB automatically creates indexes based on query patterns within each database:

```javascript
const user_db = client.db('user_management');
const inventory_db = client.db('inventory');

const users = user_db.collection('users');
const products = inventory_db.collection('products');

// First query in user database - might be slow
const user1 = await users.find_one({ email: 'john@example.com' });

// JoystickDB creates index for user database email queries
const user2 = await users.find_one({ email: 'jane@example.com' }); // Much faster!

// Separate automatic indexing for inventory database
const product1 = await products.find_one({ sku: 'MOUSE001' });
const product2 = await products.find_one({ sku: 'KEYBOARD002' }); // Auto-indexed!

// Each database maintains its own automatic indexing statistics
```

### Manual Index Management (Per Database)

Create and manage indexes independently for each database:

```javascript
const user_db = client.db('user_management');
const inventory_db = client.db('inventory');

const users = user_db.collection('users');
const products = inventory_db.collection('products');

// Create indexes in user database
await users.create_index('email');
await users.create_index('username', { unique: true });
console.log('📇 User database indexes created');

// Create indexes in inventory database
await products.create_index('sku', { unique: true });
await products.create_index('category');
await products.create_index('price');
console.log('📇 Inventory database indexes created');

// List indexes for each database separately
const user_indexes = await users.get_indexes();
const product_indexes = await products.get_indexes();

console.log('📋 User DB indexes:', user_indexes.indexes);
console.log('📋 Inventory DB indexes:', product_indexes.indexes);

// Drop indexes independently
await users.drop_index('email');
await products.drop_index('category');
```

### Cross-Database Index Isolation

Indexes are completely isolated between databases:

```javascript
const db1 = client.db('database1');
const db2 = client.db('database2');

const collection1 = db1.collection('items');
const collection2 = db2.collection('items');

// Create index in database1 only
await collection1.create_index('name');

// This collection in database2 has no indexes yet
const items1 = await collection1.find({ name: 'test' }); // Uses index
const items2 = await collection2.find({ name: 'test' }); // No index, full scan

// Create separate index in database2
await collection2.create_index('name');

// Now both have independent indexes
const items1_fast = await collection1.find({ name: 'test' }); // Uses db1 index
const items2_fast = await collection2.find({ name: 'test' }); // Uses db2 index
```

### Index Performance Benefits

```javascript
const inventory_db = client.db('inventory');
const products = inventory_db.collection('products');

// Without index: Searches through ALL products (slow for large datasets)
const expensive_products = await products.find({ price: { $gte: 1000 } });

// Create index on price
await products.create_index('price');

// With index: Jumps directly to expensive products (super fast!)
const expensive_products_fast = await products.find({ price: { $gte: 1000 } });

// Compound queries benefit from multiple indexes
await products.create_index('category');
const gaming_laptops = await products.find({ 
  category: 'laptops',
  price: { $gte: 800 }
}); // Uses both category and price indexes
```

## HTTP API

JoystickDB provides RESTful HTTP endpoints for user management and external integrations. The HTTP API uses API key authentication for secure access.

### Authentication

All HTTP API requests require an API key in the `x-joystick-db-api-key` header:

```javascript
const headers = {
  'Content-Type': 'application/json',
  'x-joystick-db-api-key': 'your-api-key-here'
};
```

### User Management Endpoints

#### Create User

Create a new user with username, password, and role.

```http
POST /api/users
Content-Type: application/json
x-joystick-db-api-key: your-api-key

{
  "username": "john_doe",
  "password": "secure_password123",
  "role": "read_write"
}
```

**Response:**
```json
{
  "success": true,
  "message": "User created successfully",
  "username": "john_doe",
  "role": "read_write"
}
```

**Roles:**
- `read` - Can only read data from all databases
- `write` - Can only write data to all databases  
- `read_write` - Can read and write data to all databases (admin privileges)

#### Get User

Retrieve user information by username.

```http
GET /api/users?username=john_doe
x-joystick-db-api-key: your-api-key
```

**Response:**
```json
{
  "success": true,
  "user": {
    "username": "john_doe",
    "role": "read_write",
    "created_at": "2024-01-15T10:30:00.000Z"
  }
}
```

#### Update User

Update user password or role.

```http
PUT /api/users
Content-Type: application/json
x-joystick-db-api-key: your-api-key

{
  "username": "john_doe",
  "password": "new_secure_password456",
  "role": "read"
}
```

**Response:**
```json
{
  "success": true,
  "message": "User updated successfully",
  "username": "john_doe",
  "updates": ["password", "role"]
}
```

#### Delete User

Remove a user from the system.

```http
DELETE /api/users?username=john_doe
x-joystick-db-api-key: your-api-key
```

**Response:**
```json
{
  "success": true,
  "message": "User deleted successfully",
  "username": "john_doe"
}
```

### HTTP API Examples

#### JavaScript/Node.js

```javascript
const API_KEY = 'your-api-key-here';
const BASE_URL = 'http://localhost:1983';

// Create user
const create_user = async (username, password, role) => {
  const response = await fetch(`${BASE_URL}/api/users`, {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
      'x-joystick-db-api-key': API_KEY
    },
    body: JSON.stringify({ username, password, role })
  });
  
  return await response.json();
};

// Get user
const get_user = async (username) => {
  const response = await fetch(`${BASE_URL}/api/users?username=${username}`, {
    headers: {
      'x-joystick-db-api-key': API_KEY
    }
  });
  
  return await response.json();
};

// Update user
const update_user = async (username, updates) => {
  const response = await fetch(`${BASE_URL}/api/users`, {
    method: 'PUT',
    headers: {
      'Content-Type': 'application/json',
      'x-joystick-db-api-key': API_KEY
    },
    body: JSON.stringify({ username, ...updates })
  });
  
  return await response.json();
};

// Delete user
const delete_user = async (username) => {
  const response = await fetch(`${BASE_URL}/api/users?username=${username}`, {
    method: 'DELETE',
    headers: {
      'x-joystick-db-api-key': API_KEY
    }
  });
  
  return await response.json();
};

// Usage examples
const demo = async () => {
  // Create a new user
  const create_result = await create_user('alice', 'secure123', 'read_write');
  console.log('User created:', create_result);
  
  // Get user info
  const user_info = await get_user('alice');
  console.log('User info:', user_info);
  
  // Update user role
  const update_result = await update_user('alice', { role: 'read' });
  console.log('User updated:', update_result);
  
  // Delete user
  const delete_result = await delete_user('alice');
  console.log('User deleted:', delete_result);
};
```

#### cURL Examples

```bash
# Create user
curl -X POST http://localhost:1983/api/users \
  -H "Content-Type: application/json" \
  -H "x-joystick-db-api-key: your-api-key-here" \
  -d '{"username": "alice", "password": "secure123", "role": "read_write"}'

# Get user
curl -X GET "http://localhost:1983/api/users?username=alice" \
  -H "x-joystick-db-api-key: your-api-key-here"

# Update user
curl -X PUT http://localhost:1983/api/users \
  -H "Content-Type: application/json" \
  -H "x-joystick-db-api-key: your-api-key-here" \
  -d '{"username": "alice", "role": "read"}'

# Delete user
curl -X DELETE "http://localhost:1983/api/users?username=alice" \
  -H "x-joystick-db-api-key: your-api-key-here"
```

### Error Responses

The HTTP API returns consistent error responses:

```json
{
  "success": false,
  "error": "Invalid API key"
}
```

**Common Error Codes:**
- `400` - Bad Request (invalid data)
- `401` - Unauthorized (invalid API key)
- `404` - Not Found (user doesn't exist)
- `409` - Conflict (user already exists)
- `500` - Internal Server Error

## Replication

JoystickDB features a simplified primary/secondary replication system for high availability and read scaling. The replication system maintains database separation across all nodes and uses API key authentication for secure sync operations.

### How Simplified Replication Works

- **Primary Node**: Main database server that accepts both read and write operations for all databases
- **Secondary Nodes**: Read-only copies that receive synchronized data from primary for all databases  
- **API Key Authentication**: All sync operations between nodes use API key authentication for security
- **Read-only Secondaries**: Secondary nodes block write operations from clients, only accepting authenticated sync from primary
- **Manual Failover**: Admin operations allow promoting a secondary to primary when needed
- **Database Isolation**: Replication maintains complete database separation across all nodes

### Setting Up Simplified Replication

#### 1. Configure Primary Node

```bash
# Primary server configuration
export JOYSTICK_DB_SETTINGS='{
  "port": 1983,
  "primary": true,
  "secondary_nodes": [
    { "ip": "192.168.1.100" },
    { "ip": "192.168.1.101" }
  ],
  "secondary_sync_key": "/path/to/sync.key",
  "sync_port": 1985
}'
```

#### 2. Configure Secondary Nodes

```bash
# Secondary server configuration  
export JOYSTICK_DB_SETTINGS='{
  "port": 1983,
  "primary": false,
  "secondary_sync_key": "/path/to/sync.key",
  "sync_port": 1985
}'
```

#### 3. Create Sync Key File

The sync key file contains the API key for authenticated sync operations:

```bash
# Generate and save sync key (same key for all nodes)
echo "your-secure-sync-api-key" > /path/to/sync.key
chmod 600 /path/to/sync.key
```

### Managing Simplified Replication

```javascript
// Connect to primary node
const client = joystickdb.client({
  port: 1983,
  authentication: { 
    username: 'admin',
    password: 'your-password' 
  }
});

// Check sync system status (admin operation)
const sync_status = await client.admin_operation({
  operation: 'get_sync_status'
});

console.log('🔄 Sync enabled:', sync_status.enabled);
console.log('📡 Connected secondaries:', sync_status.connected_nodes);
console.log('📊 Pending operations:', sync_status.pending_count);

// Promote secondary to primary (manual failover)
const promote_result = await client.admin_operation({
  operation: 'promote_to_primary',
  secondary_ip: '192.168.1.100'
});

console.log('🔄 Failover completed:', promote_result.success);

// Force sync to all secondaries
const sync_result = await client.admin_operation({
  operation: 'force_sync'
});

console.log('🔄 Manual sync completed:', sync_result.synced_nodes);
```

### Using Simplified Replication for Read Scaling

```javascript
// Connect to primary for writes (all databases supported)
const primary = joystickdb.client({
  host: '192.168.1.10',
  port: 1983,
  authentication: { 
    username: 'admin',
    password: 'your-password' 
  }
});

// Connect to secondary for reads only (all databases replicated)
const secondary = joystickdb.client({
  host: '192.168.1.100',
  port: 1983,
  authentication: { 
    username: 'admin',
    password: 'your-password' 
  }
});

// Write to primary (automatically synced to secondaries)
const user_db = primary.db('user_management');
const inventory_db = primary.db('inventory');

await user_db.collection('users').insert_one({
  name: 'New User',
  email: 'new@example.com'
});

await inventory_db.collection('products').insert_one({
  name: 'New Product',
  price: 99.99
});

// Read from secondary (reduces load on primary)
const secondary_user_db = secondary.db('user_management');
const secondary_inventory_db = secondary.db('inventory');

const users = await secondary_user_db.collection('users').find({});
const products = await secondary_inventory_db.collection('products').find({});

console.log('👥 Users from secondary:', users.documents.length);
console.log('📦 Products from secondary:', products.documents.length);

// Note: Write operations to secondary will be rejected
try {
  await secondary_user_db.collection('users').insert_one({ name: 'Test' });
} catch (error) {
  console.log('❌ Secondary is read-only:', error.message);
}
```

## Administration

### Database Statistics and Monitoring

```javascript
// Get comprehensive server statistics
const stats = await client.get_stats();

console.log('🖥️ Server Info:');
console.log('  Uptime:', stats.server.uptime_formatted);
console.log('  Version:', stats.server.version);
console.log('  Node.js:', stats.server.node_version);

console.log('💾 Memory Usage:');
console.log('  Heap Used:', Math.round(stats.memory.heapUsed / 1024 / 1024), 'MB');
console.log('  Heap Total:', Math.round(stats.memory.heapTotal / 1024 / 1024), 'MB');

console.log('🗄️ Database:');
console.log('  Size:', stats.database.used_space_mb, 'MB');
console.log('  Databases:', stats.database.databases);
console.log('  Collections:', stats.database.collections);

console.log('⚡ Performance:');
console.log('  Operations/sec:', stats.performance.ops_per_second);
console.log('  Avg Response Time:', stats.performance.avg_response_time_ms, 'ms');

console.log('🔌 Connections:');
console.log('  Active:', stats.connections.active);
console.log('  Total:', stats.connections.total);
```

### Multi-Database Administration

```javascript
// List all databases on the server
const databases = await client.list_databases();
console.log('📚 Available databases:', databases.databases);

// Get statistics for a specific database
const user_db = client.db('user_management');
const user_db_stats = await user_db.get_stats();
console.log('📊 User management DB stats:', user_db_stats);

// List collections in each database
for (const db_name of databases.databases) {
  const db = client.db(db_name);
  const collections = await db.list_collections();
  console.log(`📋 Collections in ${db_name}:`, collections.collections);
}

// Create collections explicitly in different databases
const analytics_db = client.db('analytics');
await analytics_db.create_collection('events');
await analytics_db.create_collection('reports');

const inventory_db = client.db('inventory');
await inventory_db.create_collection('products');
await inventory_db.create_collection('categories');
```

### Collection and Document Management

```javascript
// List documents across different databases
const user_db = client.db('user_management');
const inventory_db = client.db('inventory');

// List documents in user management database
const user_docs = await client.list_documents('users', {
  database: 'user_management',
  limit: 50,
  skip: 0,
  sort_field: 'created_at',
  sort_order: 'desc'
});

// List documents in inventory database
const product_docs = await client.list_documents('products', {
  database: 'inventory',
  limit: 50,
  skip: 0,
  sort_field: 'price',
  sort_order: 'asc'
});

console.log('📄 User documents:', user_docs.documents.length);
console.log('📄 Product documents:', product_docs.documents.length);

// Get specific documents by ID from different databases
const user_doc = await client.get_document('users', 'user-id-here', 'user_management');
const product_doc = await client.get_document('products', 'product-id-here', 'inventory');

// Advanced document querying across databases
const active_users = await client.query_documents('users', {
  status: 'active',
  last_login: { $gte: new Date(Date.now() - 30*24*60*60*1000) }
}, {
  database: 'user_management',
  limit: 100,
  sort: { last_login: -1 }
});

const expensive_products = await client.query_documents('products', {
  price: { $gte: 500 },
  stock: { $gt: 0 }
}, {
  database: 'inventory',
  limit: 50,
  sort: { price: -1 }
});
```

### Health Checks and Monitoring

```javascript
// Simple health check
const ping_result = await client.ping();
if (ping_result.ok === 1) {
  console.log('✅ Database is healthy');
} else {
  console.log('❌ Database is not responding');
}

// Reload server configuration without restart
const reload_result = await client.reload();
console.log('🔄 Configuration reloaded:', reload_result.message);

// Monitor performance over time
setInterval(async () => {
  const stats = await client.get_stats();
  console.log(`📊 ${new Date().toISOString()}: ${stats.performance.ops_per_second} ops/sec, ${stats.connections.active} connections, ${stats.database.databases} databases`);
}, 30000); // Every 30 seconds
```

## Backup & Restore

JoystickDB includes automatic backup capabilities with S3 integration. Backups include all databases and maintain database isolation during restore operations.

### Automatic Backups

Configure automatic backups via the `JOYSTICK_DB_SETTINGS` environment variable:

```bash
export JOYSTICK_DB_SETTINGS='{
  "backup": {
    "enabled": true,
    "schedule": "0 2 * * *",
    "retention_days": 30,
    "compression": true
  },
  "s3": {
    "region": "us-east-1",
    "bucket": "my-database-backups",
    "access_key_id": "your-access-key",
    "secret_access_key": "your-secret-key"
  }
}'
```

### Manual Backup Operations

```javascript
// Create an immediate backup (includes all databases)
const backup_result = await client.backup_now();
console.log('💾 Backup created:', backup_result.filename);
console.log('📊 Backup size:', backup_result.size_mb, 'MB');
console.log('⏱️ Time taken:', backup_result.duration_ms, 'ms');
console.log('🗄️ Databases backed up:', backup_result.databases_count);

// List all available backups
const backups = await client.list_backups();
console.log('📋 Available backups:');
backups.backups.forEach(backup => {
  console.log(`  📦 ${backup.filename} (${backup.size_mb} MB) - ${backup.created_at}`);
  console.log(`     Databases: ${backup.databases_count}, Collections: ${backup.collections_count}`);
});

// Restore from a specific backup (restores all databases)
const restore_result = await client.restore_backup('backup-2024-01-15-02-00-00.tar.gz');
console.log('🔄 Restore completed in:', restore_result.duration_ms, 'ms');
console.log('🗄️ Restored databases:', restore_result.databases_restored);
console.log('📊 Restored collections:', restore_result.collections_restored);
```

### Backup Best Practices

```javascript
// Test your backups regularly
const test_backup = async () => {
  console.log('🧪 Testing backup system...');
  
  // Create a test backup
  const backup = await client.backup_now();
  console.log('✅ Backup created successfully');
  
  // Verify backup exists and contains all databases
  const backups = await client.list_backups();
  const latest = backups.backups.find(b => b.filename === backup.filename);
  
  if (latest && latest.size_mb > 0 && latest.databases_count > 0) {
    console.log('✅ Backup verification passed');
    console.log(`📊 Backup contains ${latest.databases_count} databases`);
  } else {
    console.log('❌ Backup verification failed');
  }
};

// Run backup test monthly
setInterval(test_backup, 30 * 24 * 60 * 60 * 1000); // 30 days
```

## Configuration

### Complete Configuration Reference

```json
{
  "port": 1983,
  "cluster": true,
  "worker_count": 4,
  
  "database": {
    "path": "./data",
    "auto_map_size": true,
    "map_size": 1073741824,
    "max_dbs": 100
  },
  
  "authentication": {
    "rate_limit": {
      "max_attempts": 5,
      "window_ms": 300000,
      "lockout_duration_ms": 900000
    }
  },
  
  "primary": true,
  "secondary_nodes": [
    { "ip": "192.168.1.100" },
    { "ip": "192.168.1.101" }
  ],
  "secondary_sync_key": "/path/to/sync.key",
  "sync_port": 1985,
  
  "s3": {
    "region": "us-east-1",
    "bucket": "my-backups",
    "access_key_id": "AKIA...",
    "secret_access_key": "...",
    "endpoint": "https://s3.amazonaws.com"
  },
  
  "backup": {
    "enabled": true,
    "schedule": "0 2 * * *",
    "retention_days": 30,
    "compression": true
  },
  
  "logging": {
    "level": "info",
    "file": "./logs/joystickdb.log",
    "max_size": "20m",
    "max_files": "14d"
  },
  
  "performance": {
    "connection_pool_size": 1000,
    "idle_timeout": 600000,
    "request_timeout": 5000,
    "auto_index_threshold": 3
  }
}
```

### Environment Variable Overrides

```bash
# Server settings
JOYSTICKDB_PORT=1983
JOYSTICKDB_CLUSTER=true
JOYSTICKDB_WORKER_COUNT=4

# Database settings
JOYSTICKDB_DATABASE_PATH=./data
JOYSTICKDB_DATABASE_AUTO_MAP_SIZE=true
JOYSTICKDB_DATABASE_MAX_DBS=100

# Simplified Replication settings
JOYSTICKDB_PRIMARY=true
JOYSTICKDB_SECONDARY_SYNC_KEY=/path/to/sync.key
JOYSTICKDB_SYNC_PORT=1985

# S3 settings
JOYSTICKDB_S3_REGION=us-east-1
JOYSTICKDB_S3_BUCKET=my-backups
AWS_ACCESS_KEY_ID=your-access-key
AWS_SECRET_ACCESS_KEY=your-secret-key

# Backup settings
JOYSTICKDB_BACKUP_ENABLED=true
JOYSTICKDB_BACKUP_RETENTION_DAYS=30

# Logging
JOYSTICKDB_LOG_LEVEL=info
```

## Production Deployment

### Docker Deployment

```dockerfile
FROM node:18-alpine

WORKDIR /app

# Install dependencies
COPY package*.json ./
RUN npm ci --only=production

# Copy application code
COPY . .

# Create data and logs directories
RUN mkdir -p data logs

# Expose port
EXPOSE 1983

# Create volumes for persistent data
VOLUME ["/app/data", "/app/logs"]

# Health check
HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
  CMD node -e "
    const client = require('./src/client/index.js').default.client({port: 1983, timeout: 2000, reconnect: false});
    client.ping().then(() => process.exit(0)).catch(() => process.exit(1));
  "

# Start the server
CMD ["npm", "start"]
```

### Docker Compose

```yaml
version: '3.8'

services:
  joystickdb:
    build: .
    ports:
      - "1983:1983"
    volumes:
      - ./data:/app/data
      - ./logs:/app/logs
    environment:
      - NODE_ENV=production
      - WORKER_COUNT=4
      - JOYSTICKDB_LOG_LEVEL=info
      - JOYSTICK_DB_SETTINGS={"port": 1983, "database": {"path": "./data", "max_dbs": 100}}
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "npm", "run", "health-check"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 40s

  # Optional: Secondary node for replication
  joystickdb-secondary:
    build: .
    ports:
      - "1984:1984"
    volumes:
      - ./data-secondary:/app/data
      - ./logs-secondary:/app/logs
    environment:
      - NODE_ENV=production
      - JOYSTICKDB_PORT=1984
      - JOYSTICK_DB_SETTINGS={"port": 1984, "database": {"path": "./data", "max_dbs": 100}}
    restart: unless-stopped
    depends_on:
      - joystickdb
```

### Kubernetes Deployment

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: joystickdb
  labels:
    app: joystickdb
spec:
  replicas: 3
  selector:
    matchLabels:
      app: joystickdb
  template:
    metadata:
      labels:
        app: joystickdb
    spec:
      containers:
      - name: joystickdb
        image: joystickdb:latest
        ports:
        - containerPort: 1983
        env:
        - name: WORKER_COUNT
          value: "2"
        - name: NODE_ENV
          value: "production"
        - name: JOYSTICKDB_LOG_LEVEL
          value: "info"
        - name: JOYSTICK_DB_SETTINGS
          valueFrom:
            configMapKeyRef:
              name: joystickdb-config
              key: settings.json
        resources:
          requests:
            memory: "256Mi"
            cpu: "250m"
          limits:
            memory: "512Mi"
            cpu: "500m"
        volumeMounts:
        - name: data
          mountPath: /app/data
        livenessProbe:
          exec:
            command:
            - npm
            - run
            - health-check
          initialDelaySeconds: 30
          periodSeconds: 30
        readinessProbe:
          exec:
            command:
            - npm
            - run
            - health-check
          initialDelaySeconds: 5
          periodSeconds: 10
      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: joystickdb-data
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: joystickdb-config
data:
  settings.json: |
    {
      "port": 1983,
      "database": {
        "path": "./data",
        "auto_map_size": true,
        "max_dbs": 100
      },
      "authentication": {
        "rate_limit": {
          "max_attempts": 5,
          "window_ms": 300000
        }
      },
      "backup": {
        "enabled": true,
        "schedule": "0 2 * * *",
        "retention_days": 30
      },
      "logging": {
        "level": "info"
      }
    }
---
apiVersion: v1
kind: Service
metadata:
  name: joystickdb-service
spec:
  selector:
    app: joystickdb
  ports:
  - port: 1983
    targetPort: 1983
  type: LoadBalancer
```

### Monitoring and Health Checks

```javascript
// Health check endpoint
const health_check = async () => {
  try {
    const client = joystickdb.client({ 
      port: 1983,
      timeout: 2000,
      reconnect: false
    });
    
    const result = await client.ping();
    client.disconnect();
    
    return result.ok === 1;
  } catch (error) {
    return false;
  }
};

// Performance monitoring
const monitor_performance = async () => {
  const client = joystickdb.client({ 
    port: 1983,
    authentication: { 
      username: 'admin',
      password: process.env.DB_PASSWORD 
    }
  });
  
  const stats = await client.get_stats();
  
  // Log metrics to your monitoring system
  console.log('Metrics:', {
    uptime: stats.server.uptime,
    memory_usage: stats.memory.heapUsed,
    ops_per_second: stats.performance.ops_per_second,
    active_connections: stats.connections.active,
    database_count: stats.database.databases,
    total_collections: stats.database.collections,
    database_size: stats.database.used_space_mb
  });
  
  client.disconnect();
};
```

## API Reference

### Client Methods

#### Connection Management
- `client.connect()` - Establish connection to server
- `client.disconnect()` - Close connection to server
- `client.ping()` - Test server connectivity and responsiveness

#### Authentication & Setup
- `client.setup()` - Initial server setup (generates password) - **DEPRECATED**
- `client.authenticate()` - Manual authentication (usually automatic)

#### Multi-Database Interface
- `client.db(database_name)` - Get database interface for method chaining
- `client.list_databases()` - List all databases on the server

#### Database-Level Operations
- `database.collection(collection_name)` - Get collection interface within database
- `database.list_collections()` - List collections in the database
- `database.get_stats()` - Get database-specific statistics
- `database.drop_database()` - Drop the entire database (admin operation)
- `database.create_collection(collection_name, options)` - Create collection explicitly

#### CRUD Operations (Collection Chaining API)
- `collection.insert_one(document, options)` - Insert single document
- `collection.find_one(filter, options)` - Find single document
- `collection.find(filter, options)` - Find multiple documents
- `collection.update_one(filter, update, options)` - Update single document
- `collection.delete_one(filter, options)` - Delete single document
- `collection.delete_many(filter, options)` - Delete multiple documents
- `collection.bulk_write(operations, options)` - Bulk operations

#### Index Management (Collection Chaining API)
- `collection.create_index(field, options)` - Create index on field
- `collection.upsert_index(field, options)` - Create or update index (upsert)
- `collection.drop_index(field)` - Drop index on field
- `collection.get_indexes()` - List all indexes for collection

#### Auto-Indexing Management
- `client.get_auto_index_stats()` - Get automatic indexing statistics

#### Simplified Replication Management (via Admin Operations)
- `client.admin_operation({ operation: 'get_sync_status' })` - Get sync system status and statistics
- `client.admin_operation({ operation: 'promote_to_primary', secondary_ip: 'ip' })` - Promote secondary to primary (manual failover)
- `client.admin_operation({ operation: 'force_sync' })` - Force synchronization with all secondaries

#### Administration
- `client.get_stats()` - Get comprehensive server statistics
- `client.list_collections()` - List all collections in default database (backward compatible)
- `client.list_documents(collection, options)` - List documents with pagination and sorting
- `client.get_document(collection, document_id, database)` - Get document by ID
- `client.query_documents(collection, filter, options)` - Advanced document query with filtering
- `client.insert_document(collection, document)` - Admin insert operation
- `client.update_document(collection, document_id, update)` - Admin update operation by ID
- `client.delete_document(collection, document_id)` - Admin delete operation by ID

#### Backup & Restore
- `client.backup_now()` - Create immediate backup (includes all databases)
- `client.list_backups()` - List all available backups
- `client.restore_backup(backup_name)` - Restore from backup (restores all databases)

#### Server Management
- `client.reload()` - Reload server configuration

### HTTP API Methods

#### User Management (via HTTP)
- `POST /api/users` - Create new user with username, password, and role
- `GET /api/users?username=<username>` - Get user information by username
- `PUT /api/users` - Update user password or role
- `DELETE /api/users?username=<username>` - Delete user by username

All HTTP API endpoints require the `x-joystick-db-api-key` header for authentication.

### Events

The client emits these events that you can listen to:

- `connect` - Connection established with server
- `authenticated` - Authentication successful, ready to use
- `error` - Error occurred (connection, authentication, etc.)
- `disconnect` - Connection closed
- `reconnecting` - Reconnection attempt in progress
- `response` - Unsolicited server response received

### Query Options

#### Find Options
```javascript
{
  limit: 10,              // Maximum number of documents to return
  skip: 0,                // Number of documents to skip
  sort: { field: 1 },     // Sort order (1 = ascending, -1 = descending)
  projection: { field: 1 } // Fields to include (1) or exclude (0)
}
```

#### Update Options
```javascript
{
  upsert: false          // Create document if it doesn't exist
}
```

#### Index Options
```javascript
{
  unique: false,         // Enforce uniqueness
  sparse: false          // Skip documents missing the indexed field
}
```

### Update Operators

JoystickDB supports these MongoDB-style update operators:

- `$set` - Set field values
- `$unset` - Remove fields
- `$inc` - Increment numeric values
- `$push` - Add elements to arrays
- `$pull` - Remove elements from arrays

### Query Operators

JoystickDB supports these MongoDB-style query operators:

- `$eq` - Equal to
- `$ne` - Not equal to
- `$gt` - Greater than
- `$gte` - Greater than or equal to
- `$lt` - Less than
- `$lte` - Less than or equal to
- `$in` - Value in array
- `$nin` - Value not in array
- `$exists` - Field exists
- `$regex` - Regular expression match

### Error Handling

```javascript
try {
  const user_db = client.db('user_management');
  const users = user_db.collection('users');
  const result = await users.insert_one({ name: 'John' });
} catch (error) {
  if (error.message.includes('Authentication')) {
    // Handle authentication error
    console.error('Please check your username and password');
  } else if (error.message.includes('timeout')) {
    // Handle timeout error
    console.error('Server is not responding');
  } else if (error.message.includes('Connection')) {
    // Handle connection error
    console.error('Cannot connect to server');
  } else if (error.message.includes('Invalid database name')) {
    // Handle database naming error
    console.error('Database name violates naming rules');
  } else {
    // Handle other errors
    console.error('Database error:', error.message);
  }
}
```

### Performance Tips

1. **Use Indexes**: Create indexes on frequently queried fields in each database
2. **Batch Operations**: Use `bulk_write` for multiple operations
3. **Limit Results**: Always use `limit` for large result sets
4. **Use Projections**: Only fetch fields you need
5. **Connection