zeno-db/README.md

A lightweight, offline-first client-side database with automatic sync capabilities

# ZenoDB

<div align="center">
  <img src="https://i.ibb.co/67ktWMm0/zenodb.png" alt="ZenoDB Logo" width="300" style="margin-bottom: 10px;" />

</div>

# Zeno - Offline-First Sync System

A lightweight, offline-first synchronization system that enables seamless data synchronization between client-side IndexedDB and server-side PostgreSQL. Perfect for note-taking applications and other data-heavy web apps that need to work offline.

> ⚠️ **BETA VERSION NOTICE**
> 
> This project is currently in beta stage and under active development. While it's functional, you may encounter:
> - Unexpected behaviors
> - Breaking changes between versions
> - Incomplete features
> - Limited documentation
>
> **Use in production with caution.** We recommend thorough testing in development/staging environments before deploying to production.
> 
> Please report any issues you encounter on our GitHub repository.

## Table of Contents
- [Features](#features)
- [Installation](#installation)
- [Quick Start](#quick-start)
- [How It Works](#how-it-works)
- [Configuration](#configuration)
- [Sync Status & Events](#sync-status--events)
- [Best Practices](#best-practices)
- [Schema Migrations & Conflict Resolution](#schema-migrations--conflict-resolution)
- [Examples](#examples)
- [Troubleshooting](#troubleshooting)
- [License](#license)
- [TODO List & Roadmap](#todo-list--roadmap)

## Features

- 🔄 **Real-time Sync**: Instant synchronization between all connected clients
- 📱 **Offline Support**: Continue working without internet connection
- 🔌 **Auto-Reconnect**: Automatically reconnects when network is available
- 💾 **Persistent Storage**: IndexedDB (client) + PostgreSQL (server)
- ⚡ **WebSocket Protocol**: Fast, real-time updates across clients
- 🔒 **Safe Transactions**: Ensures data consistency
- 📊 **Sync Status**: Real-time sync status indicators and offline mode support
- 🗑️ **Soft Delete**: Recoverable deletions with automatic cleanup

## Installation

```bash
# Install the package
npm install zeno-db

# Or with yarn
yarn add zeno-db
```

## Quick Start

### Server Setup

1. Install dependencies:
```bash
npm install zeno-db
```

2. Create a server.js file:
```javascript
import { startSyncServer } from 'zeno-db/server';

startSyncServer({
  port: 3000, 
  pg: {
    connectionString: 'your-postgresql-connection-string',
    table: 'notes' // Your table name
  }
});
```

### Client Setup

1. Initialize the database in your app:

```javascript
import { ZenoDB } from 'zeno-db';

const db = new ZenoDB({
  storage: {
    type: 'indexeddb',
    name: 'my-app-db'
  },
  sync: {
    type: 'websocket',
    url: 'ws://localhost:3000'
  },
  clientId: `client-${Date.now()}-${Math.random().toString(36).substr(2, 9)}`,
  softDelete: {
    enabled: true,
    fieldName: 'deletedAt',
    permanentDeleteAfter: 30 // Delete after 30 days
  }
});

await db.init();
```

### Usage Example (Notes App)

```javascript
// Create a new note
const noteId = `note-${Date.now()}`;
await db.set(noteId, {
  id: noteId,
  text: 'Your note text',
  createdAt: new Date().toISOString()
});

// Read a note
const note = await db.get(noteId);

// Delete a note
await db.delete(noteId);

// Subscribe to changes
db.subscribe((key, value) => {
  console.log('Note changed:', key, value);
});
```

### Batch Operations Example

You can efficiently get or set multiple records at once using batch operations:

```javascript
// Batch get
const notes = await db.getMany(['note-1', 'note-2', 'note-3']);

// Batch set
await db.setMany([
  { id: 'note-4', text: 'Fourth note', createdAt: new Date().toISOString() },
  { id: 'note-5', text: 'Fifth note', createdAt: new Date().toISOString() }
]);
```

## Examples

### React Example with Sync Status
```jsx
import { useState, useEffect } from 'react';
import { ZenoDB } from 'zeno-db';

function NotesApp() {
  const [db, setDb] = useState(null);
  const [syncStatus, setSyncStatus] = useState({
    isOnline: false,
    isSyncing: false,
    syncProgress: 0,
    pendingChangesCount: 0
  });

  useEffect(() => {
    const initDB = async () => {
      const zenoDB = new ZenoDB({
        storage: { type: 'indexeddb', name: 'notes-db' },
        sync: { type: 'websocket', url: 'ws://localhost:3000' },
        softDelete: {
          enabled: true,
          fieldName: 'deletedAt',
          permanentDeleteAfter: 30 // Delete after 30 days
        }
      });

      await zenoDB.init();
      setDb(zenoDB);

      // Subscribe to sync events
      zenoDB.subscribe((event) => {
        setSyncStatus(event.status);
      });
    };

    initDB();
  }, []);

  return (
    <div>
      <div className={`status ${syncStatus.isOnline ? 'online' : 'offline'}`}>
        {syncStatus.isOnline ? 'Online' : 'Offline'}
      </div>
      {syncStatus.isSyncing && (
        <div className="progress">
          Syncing... {syncStatus.syncProgress}%
        </div>
      )}
    </div>
  );
}
```

### Vue Example
```vue
<template>
  <div>
    <div :class="['status', { online: syncStatus.isOnline }]">
      {{ syncStatus.isOnline ? 'Online' : 'Offline' }}
    </div>
    <div v-if="syncStatus.isSyncing" class="progress">
      Syncing... {{ syncStatus.syncProgress }}%
    </div>
  </div>
</template>

<script>
import { ZenoDB } from 'zeno-db';

export default {
  data() {
    return {
      db: null,
      syncStatus: {
        isOnline: false,
        isSyncing: false,
        syncProgress: 0,
        pendingChangesCount: 0
      }
    };
  },
  async mounted() {
    this.db = new ZenoDB({
      storage: { type: 'indexeddb', name: 'notes-db' },
      sync: { type: 'websocket', url: 'ws://localhost:3000' },
      softDelete: {
        enabled: true,
        fieldName: 'deletedAt',
        permanentDeleteAfter: 30 // Delete after 30 days
      }
    });

    await this.db.init();
    
    this.db.subscribe((event) => {
      this.syncStatus = event.status;
    });
  }
};
</script>
```

## How It Works

### Client Side
- Uses IndexedDB for local storage
- Queues changes while offline
- Auto-syncs when connection restores
- Real-time updates via WebSocket
- Sync status indicators and event handling

### Server Side
- PostgreSQL database for persistent storage
- WebSocket server for real-time communication
- Handles concurrent updates
- Broadcasts changes to all connected clients

## Configuration

### Server Config
```typescript
interface ServerConfig {
  port: number;
  pg: {
    connectionString: string;  // PostgreSQL connection string
    table: string;            // Table name for your data
  };
  softDelete?: {
    enabled: boolean;         // Enable soft delete functionality
    fieldName?: string;       // Field to store deletion timestamp (default: 'deletedAt')
    permanentDeleteAfter?: number; // Days to keep soft-deleted items
  };
}
```

### Client Config
```typescript
interface ClientConfig {
  storage: {
    type: 'indexeddb';
    name: string;        // IndexedDB database name
  };
  sync: {
    type: 'websocket';
    url: string;        // WebSocket server URL
  };
  clientId: string;     // Unique client identifier
  softDelete?: {
    enabled: boolean;   // Enable soft delete functionality
    fieldName?: string; // Field to store deletion timestamp (default: 'deletedAt')
    permanentDeleteAfter?: number; // Days to keep soft-deleted items
  };
}
```

### Usage Example with Soft Delete

```javascript
// Initialize with soft delete enabled
const db = new ZenoDB({
  storage: {
    type: 'indexeddb',
    name: 'my-app-db'
  },
  sync: {
    type: 'websocket',
    url: 'ws://localhost:3000'
  },
  clientId: 'client-1',
  softDelete: {
    enabled: true,
    fieldName: 'deletedAt',
    permanentDeleteAfter: 30 // Delete after 30 days
  }
});

// Soft delete an item
await db.delete('note-1');

// Restore a soft-deleted item
await db.restore('note-1');

// Get all items including soft-deleted ones
const allItems = await db.getAll(true);

// Permanently delete an item
await db.delete('note-1', true);

// Purge old soft-deleted items
await db.purgeDeleted();
```

## Sync Status & Events

ZenoDB provides real-time sync status updates and event handling to help you build responsive UIs that reflect the current sync state of your application.

```typescript
import { ZenoDB, SyncEvent, SyncStatus } from 'zeno-db';

// Initialize your database
const db = new ZenoDB({
  storage: {
    type: 'indexeddb',
    name: 'my-app-db'
  },
  sync: {
    type: 'websocket',
    url: 'ws://localhost:3000'
  },
  softDelete: {
    enabled: true,
    fieldName: 'deletedAt',
    permanentDeleteAfter: 30 // Delete after 30 days
  }
});

// Subscribe to sync events
db.subscribe((event: SyncEvent) => {
  const { type, status } = event;
  
  // status contains:
  // - isOnline: boolean (connection status)
  // - isSyncing: boolean (whether sync is in progress)
  // - syncProgress: number (0-100 percentage)
  // - pendingChangesCount: number (changes waiting to sync)
  
  switch(type) {
    case 'connection_change':
      updateConnectionUI(status.isOnline);
      break;
    case 'sync_started':
      showSyncStarted();
      break;
    case 'sync_progress':
      updateProgressBar(status.syncProgress);
      break;
    case 'sync_completed':
      showSyncComplete();
      break;
  }
});

// Get current status at any time
const currentStatus: SyncStatus = db.getSyncStatus();
```

#### Sync Event Types
- `connection_change`: Fired when connection status changes (online/offline)
- `sync_started`: Fired when sync operation begins
- `sync_progress`: Fired during sync with progress updates
- `sync_completed`: Fired when sync operation completes

#### Sync Status Properties
- `isOnline`: Current connection status
- `isSyncing`: Whether a sync operation is in progress
- `syncProgress`: Progress percentage (0-100)
- `pendingChangesCount`: Number of changes waiting to sync

## Best Practices

1. Always initialize before use:
```javascript
await db.init();
```

2. Handle offline/online transitions gracefully:
```javascript
window.addEventListener('online', () => {
  console.log('Back online, syncing...');
});
```

3. Use try-catch for error handling:
```javascript
try {
  await db.set('key', value);
} catch (error) {
  console.error('Error saving data:', error);
}
```

4. Monitor sync status:
```javascript
db.subscribe((event) => {
  const { type, status } = event;
  // Update UI based on sync status.
});
```

5. Use soft delete for important data:
```javascript
// Instead of permanent deletion
await db.delete('important-data');

// Later, if needed
await db.restore('important-data');

// Clean up old deleted items periodically
await db.purgeDeleted();
```

## Schema Migrations & Conflict Resolution

### Client-side Migration Example

When evolving your data schema, you may need to migrate existing records. For example, if you add a new field to your notes, you can write a migration function that updates all records after initializing the database:

```js
// Example: Migrating notes to add a "tags" field if missing
async function migrateNotes(db) {
  const allNotes = await db.getAll();
  for (const note of allNotes) {
    if (!note.tags) {
      note.tags = [];
      await db.set(note.id, note); // Save migrated note
    }
  }
}

// Run migration after db.init()
await db.init();
await migrateNotes(db);
```
*Tip: Store a schema version in each record or in IndexedDB metadata, and only run migrations when needed.*

### Application-level Merging Example

For collaborative or complex data structures, you may need to merge changes from multiple sources. Here is a simple example for merging two versions of a collaborative list:

```js
// Example: Merging two versions of a collaborative list
function mergeLists(localList, remoteList) {
  const merged = [...localList];
  for (const item of remoteList) {
    if (!merged.find(i => i.id === item.id)) {
      merged.push(item); // Add new items from remote
    }
    // Optionally, resolve conflicts for items with the same id
  }
  return merged;
}

// Usage in sync event handler
// (Assuming your sync system emits a 'sync_conflict' event)
db.subscribe((event) => {
  if (event.type === 'sync_conflict') {
    const merged = mergeLists(event.local, event.remote);
    db.set(event.key, merged);
  }
});
```
*Tip: For more complex data, consider using libraries like Automerge or Yjs for CRDT-based merging.*

## PostgreSQL Table Structure

Your PostgreSQL table should have this structure:
```sql
CREATE TABLE notes (
  id TEXT PRIMARY KEY,
  data JSONB NOT NULL,
  deleted_at TIMESTAMP WITH TIME ZONE,
  created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
  updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP
);

-- Index for soft delete queries
CREATE INDEX idx_notes_deleted_at ON notes(deleted_at);
```

## License

MIT License - see LICENSE file for details.

## TODO List & Roadmap

### Authentication & Security 🔐
- [ ] User authentication system integration
- [ ] JWT-based authentication
- [ ] Role-based access control (RBAC)
- [ ] API rate limiting
- [ ] Data encryption at rest
- [ ] Session management
- [ ] OAuth2 provider integration (Google, GitHub)

### Data Management 📊
- [ ] Data versioning
- [ ] Soft delete functionality
- [ ] Data validation middleware
- [ ] Advanced conflict resolution strategies
- [ ] Bulk import/export functionality

### Performance Optimizations ⚡
- [ ] Data compression
- [ ] Caching layer
- [ ] Request batching
- [ ] Lazy loading support
- [ ] Connection pooling
- [ ] Query optimization

### Developer Experience 🛠️
- [ ] CLI tool for database management
- [ ] Better error handling and logging
- [ ] Development environment tooling
- [ ] TypeScript type definitions
- [ ] API documentation with Swagger/OpenAPI
- [ ] Integration tests
- [ ] E2E testing suite

### Monitoring & Debugging 📈
- [ ] Telemetry integration
- [ ] Performance metrics dashboard
- [ ] Debug logging
- [ ] Error tracking integration
- [ ] Health check endpoints
- [ ] Audit logging

### Infrastructure 🏗️
- [ ] Docker containerization
- [ ] CI/CD pipeline setup
- [ ] Automated deployment scripts
- [ ] Database migration tools
- [ ] Backup and restore functionality
- [ ] High availability setup

## Troubleshooting

### Common Issues

1. **Connection Issues**
   ```javascript
   // Check if WebSocket server is running
   const ws = new WebSocket('ws://localhost:3000');
   ws.onerror = (error) => {
     console.error('WebSocket connection error:', error);
   };
   ```

2. **Database Initialization Failures**
   ```javascript
   try {
     await db.init();
   } catch (error) {
     console.error('Database initialization failed:', error);
     // Check if IndexedDB is supported
     if (!window.indexedDB) {
       console.error('IndexedDB is not supported in this browser');
     }
   }
   ```

3. **Sync Status Not Updating**
   - Ensure you're properly subscribing to sync events
   - Check network connectivity
   - Verify WebSocket connection is active

### Debugging Tips

1. Enable verbose logging:
   ```javascript
   const db = new ZenoDB({
     // ... config
     debug: true
   });
   ```

2. Monitor sync events:
   ```javascript
   db.subscribe((event) => {
     console.log('Sync event:', event);
   });
   ```