dblacerta/readme.md

LacertaDB ? Javascript IndexedDB Database for Web Browsers. Simple, Fast, Secure.

# LacertaDB Documentation

## Table of Contents

1. [Introduction](#introduction)
2. [Architecture Overview](#architecture-overview)
3. [Installation](#installation)
4. [Core Concepts](#core-concepts)
5. [Getting Started](#getting-started)
6. [API Reference](#api-reference)
7. [Advanced Features](#advanced-features)
8. [Performance Optimization](#performance-optimization)
9. [Security Considerations](#security-considerations)
10. [Best Practices](#best-practices)
11. [Troubleshooting](#troubleshooting)
12. [Migration Guide](#migration-guide)

---

## Introduction

**LacertaDB** represents a sophisticated abstraction layer over IndexedDB, architecting a comprehensive document-oriented database system directly within the browser environment. This library synthesizes multiple advanced capabilities including cryptographic operations, data compression, attachment management through the Origin Private File System (OPFS), and intelligent storage lifecycle management.

### Key Differentiators

- **Cryptographic Security**: Military-grade AES-GCM encryption with PBKDF2 key derivation
- **Compression Architecture**: Native browser compression streams for optimal storage utilization
- **Attachment Management**: Seamless binary data handling via OPFS
- **Automatic Space Management**: Intelligent garbage collection with configurable retention policies
- **Metadata Synchronization**: Dual-layer metadata persistence for rapid access patterns
- **Transaction Atomicity**: ACID-compliant operations with rollback capabilities

### Architectural Philosophy

LacertaDB embodies a multi-tiered storage strategy, leveraging IndexedDB for structured document storage while utilizing localStorage for metadata persistence and OPFS for binary attachment management. This tripartite architecture ensures optimal performance characteristics across diverse usage patterns.

---

## Architecture Overview

### Storage Layer Topology

```
┌─────────────────────────────────────────┐
│         Application Layer               │
├─────────────────────────────────────────┤
│            LacertaDB API                │
├─────────────────────────────────────────┤
│   ┌─────────────┬──────────┬──────────┐ │
│   │  Database   │Collection│ Document │ │
│   │  Manager    │ Handler  │ Processor│ │
│   └─────────────┴──────────┴──────────┘ │
├─────────────────────────────────────────┤
│   ┌─────────────┬──────────┬──────────┐ │
│   │  IndexedDB  │LocalStore│   OPFS   │ │
│   │  (Documents)│(Metadata)│(Attachm.)│ │
│   └─────────────┴──────────┴──────────┘ │
└─────────────────────────────────────────┘
```

### Component Interactions

The system orchestrates through several interconnected subsystems:

1. **Database Layer**: Manages database lifecycle and collection orchestration
2. **Collection Layer**: Handles document CRUD operations and indexing
3. **Document Layer**: Processes packing, encryption, and compression
4. **Metadata Layer**: Maintains synchronized metadata across storage boundaries
5. **Utility Layer**: Provides cryptographic, compression, and file system operations

---

## Installation

### Module Import

```javascript
// ES6 Module Import
import { Database, Document, Collection } from './lacertadb.js';
import JOYSON from 'joyson';  // Required dependency
```

### Prerequisites

- Modern browser with IndexedDB support
- OPFS API availability (Chrome 86+, Edge 86+, Safari 15.2+)
- Native Compression Streams API
- Web Crypto API

### Browser Compatibility Matrix

| Feature | Chrome | Firefox | Safari | Edge |
|---------|--------|---------|--------|------|
| IndexedDB | ✓ | ✓ | ✓ | ✓ |
| OPFS | 86+ | 111+ | 15.2+ | 86+ |
| Compression Streams | 80+ | 113+ | 16.4+ | 80+ |
| Web Crypto | 37+ | 34+ | 7+ | 12+ |

---

## Core Concepts

### Document Model

Documents represent the atomic unit of storage, encapsulating both structured data and metadata:

```javascript
{
  _id: "uuid-string",           // Unique identifier
  _created: 1234567890,          // Creation timestamp
  _modified: 1234567890,         // Modification timestamp
  _permanent: false,             // Deletion protection flag
  _encrypted: false,             // Encryption status
  _compressed: false,            // Compression status
  attachments: [],               // Binary attachment references
  data: {                        // User-defined payload
    // Application data
  }
}
```

### Collection Paradigm

Collections function as logical containers for documents, providing:
- Document isolation and namespacing
- Independent size limits and retention policies
- Transaction boundaries
- Index management capabilities

### Metadata Architecture

The metadata subsystem maintains dual-layer persistence:
- **Database Metadata**: Global statistics and collection registry
- **Collection Metadata**: Document inventory and size tracking

---

## Getting Started

### Basic Database Operations

```javascript
// Initialize database with configuration
const db = new Database('myApplication', {
  sizeLimitKB: 50000,        // 50MB limit
  bufferLimitKB: -10000,     // 10MB buffer before cleanup
  freeSpaceEvery: 60000      // Cleanup check every 60 seconds
});

// Initialize database connection
await db.init();

// Create a collection
const users = await db.createCollection('users');

// Add a document
const isNew = await users.addDocument({
  data: {
    name: 'Alice Johnson',
    email: 'alice@example.com',
    preferences: {
      theme: 'dark',
      notifications: true
    }
  },
  _permanent: true  // Protect from automatic cleanup
});

// Retrieve document
const doc = await users.getDocument('document-id');
console.log(doc.data);

// Update document
await users.addDocument({
  _id: doc._id,
  data: {
    ...doc.data,
    lastLogin: Date.now()
  }
});

// Delete document
await users.deleteDocument(doc._id, true); // Force delete even if permanent
```

### Working with Attachments

```javascript
// Create document with attachments
const fileInput = document.getElementById('fileInput');
const files = Array.from(fileInput.files);

await users.addDocument({
  data: {
    title: 'Project Documentation',
    description: 'Q4 Analysis Report'
  },
  attachments: files.map(file => ({ data: file }))
});

// Retrieve document with attachments
const docWithFiles = await users.getDocument(
  'doc-id', 
  null,     // No encryption key
  true      // Include attachment data
);

// Access attachment blobs
for (const attachment of docWithFiles.attachments) {
  const blob = attachment.data;
  const url = URL.createObjectURL(blob);
  // Use the blob URL
}
```

### Encryption Implementation

```javascript
// Store encrypted document
const secretKey = 'user-provided-password-123';

await users.addDocument({
  data: {
    ssn: '123-45-6789',
    creditCard: '4111-1111-1111-1111',
    medicalRecords: { /* sensitive data */ }
  },
  _encrypted: true
}, secretKey);

// Retrieve and decrypt
const encryptedDoc = await users.getDocument('doc-id', secretKey);
// Returns false if wrong key provided
```

---

## API Reference

### Database Class

#### Constructor
```javascript
new Database(dbName: string, settings?: object)
```

**Parameters:**
- `dbName`: Unique database identifier
- `settings`: Configuration object
  - `sizeLimitKB`: Maximum size in kilobytes (default: Infinity)
  - `bufferLimitKB`: Buffer before cleanup triggers (default: -20% of sizeLimitKB)
  - `freeSpaceEvery`: Cleanup interval in milliseconds (default: 10000)

#### Methods

##### `async init()`
Initializes database connection and loads existing collections.

```javascript
const db = new Database('myApp');
await db.init();
```

##### `async createCollection(collectionName: string)`
Creates a new collection within the database.

**Returns:** `Collection` instance

```javascript
const products = await db.createCollection('products');
```

##### `async deleteCollection(collectionName: string)`
Removes a collection and all associated documents.

```javascript
await db.deleteCollection('deprecated_data');
```

##### `async getCollection(collectionName: string)`
Retrieves existing collection instance.

**Returns:** `Collection` instance

```javascript
const orders = await db.getCollection('orders');
```

##### `async close()`
Properly closes database connections and cleans up resources.

```javascript
await db.close();
```

##### `async deleteDatabase()`
Completely removes database and all associated data.

```javascript
await db.deleteDatabase();
```

#### Properties

- `name`: Database identifier
- `totalSizeKB`: Total storage consumption in KB
- `totalLength`: Total document count across collections
- `modifiedAt`: Last modification timestamp
- `collections`: Map of active collection instances

### Collection Class

#### Methods

##### `async addDocument(documentData: object, encryptionKey?: string)`
Inserts or updates a document in the collection.

**Parameters:**
- `documentData`: Document object with data and metadata
- `encryptionKey`: Optional encryption password

**Returns:** `boolean` - true if newly created, false if updated

```javascript
const isNew = await collection.addDocument({
  data: { name: 'Item 1' },
  _compressed: true,
  _permanent: false
});
```

##### `async getDocument(docId: string, encryptionKey?: string, includeAttachments?: boolean)`
Retrieves a single document by ID.

**Parameters:**
- `docId`: Document identifier
- `encryptionKey`: Decryption key if encrypted
- `includeAttachments`: Load attachment data

**Returns:** Document object or `false` if not found

```javascript
const doc = await collection.getDocument('abc-123', 'password', true);
```

##### `async getDocuments(ids: string[], encryptionKey?: string, withAttachments?: boolean)`
Batch retrieval of multiple documents.

**Returns:** Array of document objects

```javascript
const docs = await collection.getDocuments(['id1', 'id2', 'id3']);
```

##### `async deleteDocument(docId: string, force?: boolean)`
Removes a document from the collection.

**Parameters:**
- `docId`: Document identifier
- `force`: Override permanent flag

**Returns:** `boolean` - success status

```javascript
await collection.deleteDocument('obsolete-doc', true);
```

##### `async query(filter?: object, options?: object)`
Performs filtered queries on the collection.

**Parameters:**
- `filter`: Key-value pairs for matching
- `options`: Query configuration
  - `limit`: Maximum results
  - `offset`: Skip count
  - `orderBy`: Sort direction ('asc' or 'desc')
  - `index`: Index name to use
  - `encryptionKey`: Decryption key

**Returns:** Array of matching documents

```javascript
const results = await collection.query(
  { 'status': 'active' },
  { 
    limit: 50, 
    offset: 100,
    orderBy: 'desc'
  }
);
```

##### `async freeSpace(size: number)`
Manually triggers space reclamation.

**Parameters:**
- `size`: Positive for target size, negative for amount to free

**Returns:** Amount of space freed in KB

```javascript
// Keep collection under 10MB
const freed = await collection.freeSpace(10000);

// Free 5MB of space
const freed = await collection.freeSpace(-5000);
```

##### `async createIndex(fieldPath: string, options?: object)`
Creates an index for optimized queries.

**Parameters:**
- `fieldPath`: Dot-notation path to field
- `options`: Index configuration
  - `unique`: Enforce uniqueness
  - `multiEntry`: Index array values

```javascript
await collection.createIndex('data.email', { unique: true });
await collection.createIndex('data.tags', { multiEntry: true });
```

#### Properties

- `name`: Collection identifier
- `sizeKB`: Total size in kilobytes
- `length`: Document count
- `keys`: Array of document IDs
- `documentsMetadata`: Metadata for all documents
- `observer`: Event observer instance

### Document Class

#### Constructor
```javascript
new Document(data: object, encryptionKey?: string)
```

#### Static Methods

##### `static hasAttachments(documentData: object)`
Checks for attachment presence.

```javascript
if (Document.hasAttachments(doc)) {
  // Process attachments
}
```

##### `static isEncrypted(documentData: object)`
Verifies encryption status.

```javascript
if (Document.isEncrypted(doc)) {
  // Request decryption key
}
```

##### `static async decryptDocument(documentData: object, encryptionKey: string)`
Decrypts an encrypted document.

```javascript
const decrypted = await Document.decryptDocument(encDoc, 'password');
```

### QuickStore Class

Provides synchronous localStorage-based storage for small documents.

```javascript
const quickStore = db.quickStore;

// Synchronous operations
quickStore.setDocumentSync({ 
  _id: 'quick-1',
  data: { temp: true } 
});

const doc = quickStore.getDocumentSync('quick-1');
quickStore.deleteDocumentSync('quick-1');

const allKeys = quickStore.getAllKeys();
```

---

## Advanced Features

### Event System

The observer pattern enables reactive programming paradigms:

```javascript
const collection = await db.getCollection('events');

// Register event handlers
collection.observer.on('beforeAdd', (doc) => {
  console.log('Document being added:', doc._id);
  // Validation logic
});

collection.observer.on('afterAdd', (doc) => {
  console.log('Document added successfully:', doc._id);
  // Trigger side effects
});

collection.observer.on('beforeDelete', (docId) => {
  console.log('Preparing to delete:', docId);
  // Cleanup logic
});

// Remove handler
const handler = (doc) => console.log(doc);
collection.observer.on('afterGet', handler);
collection.observer.off('afterGet', handler);
```

### Transaction Management

LacertaDB ensures atomicity through transaction management:

```javascript
// Batch operations execute atomically
const documents = [
  { data: { id: 1, value: 'A' }},
  { data: { id: 2, value: 'B' }},
  { data: { id: 3, value: 'C' }}
];

try {
  for (const doc of documents) {
    await collection.addDocument(doc);
  }
  // All succeed or all fail
} catch (error) {
  console.error('Transaction failed:', error);
  // Automatic rollback
}
```

### Compression Strategies

```javascript
// Enable compression for large documents
await collection.addDocument({
  data: {
    largeText: 'Lorem ipsum...'.repeat(10000),
    matrix: new Array(1000).fill(new Array(1000).fill(0))
  },
  _compressed: true  // Reduces storage footprint
});
```

### Metadata Analysis

```javascript
// Access collection metadata
const metadata = collection.documentsMetadata;
console.table(metadata);

// Analyze storage patterns
const analysis = metadata.reduce((acc, doc) => {
  acc.totalSize += doc.size;
  acc.avgSize = acc.totalSize / metadata.length;
  acc.permanent += doc.permanent ? 1 : 0;
  acc.withAttachments += doc.attachment > 0 ? 1 : 0;
  return acc;
}, { totalSize: 0, avgSize: 0, permanent: 0, withAttachments: 0 });

console.log('Storage Analysis:', analysis);
```

### Custom Indexing Strategies

```javascript
// Create compound indexes
await collection.createIndex('data.category');
await collection.createIndex('data.price');

// Query using indexes
const results = await collection.query(
  { 'data.category': 'electronics' },
  { 
    index: 'data_category',
    orderBy: 'asc',
    limit: 100 
  }
);
```

---

## Performance Optimization

### Storage Strategy Guidelines

1. **Document Size Optimization**
   - Keep documents under 1MB for optimal performance
   - Use compression for documents > 100KB
   - Store large binaries as attachments

2. **Indexing Best Practices**
   - Create indexes before bulk inserts
   - Index frequently queried fields
   - Avoid over-indexing (max 5-7 per collection)

3. **Batch Operations**
   ```javascript
   // Efficient batch insert
   const batch = Array.from({ length: 1000 }, (_, i) => ({
     data: { index: i, value: Math.random() }
   }));
   
   for (const doc of batch) {
     await collection.addDocument(doc);
   }
   ```

4. **Memory Management**
   ```javascript
   // Configure aggressive cleanup
   const db = new Database('app', {
     sizeLimitKB: 20000,      // 20MB limit
     bufferLimitKB: -2000,    // 2MB buffer
     freeSpaceEvery: 30000    // Check every 30s
   });
   ```

### Query Optimization

```javascript
// Use indexes for complex queries
await collection.createIndex('data.timestamp');

// Efficient pagination
async function* paginate(collection, pageSize = 100) {
  let offset = 0;
  let hasMore = true;
  
  while (hasMore) {
    const results = await collection.query({}, {
      limit: pageSize,
      offset: offset,
      orderBy: 'desc'
    });
    
    if (results.length < pageSize) {
      hasMore = false;
    }
    
    yield results;
    offset += pageSize;
  }
}

// Usage
for await (const page of paginate(collection)) {
  processDocuments(page);
}
```

---

## Security Considerations

### Encryption Architecture

LacertaDB implements a multi-layered security model:

1. **Key Derivation**: PBKDF2 with 600,000 iterations
2. **Encryption**: AES-GCM with 256-bit keys
3. **Integrity**: SHA-256 checksums
4. **Salt Generation**: Cryptographically secure random values

### Security Best Practices

```javascript
// Generate strong encryption keys
function generateSecureKey() {
  const array = new Uint8Array(32);
  crypto.getRandomValues(array);
  return btoa(String.fromCharCode(...array));
}

// Implement key rotation
async function rotateEncryption(collection, oldKey, newKey) {
  const docs = await collection.query({});
  
  for (const doc of docs) {
    if (doc._encrypted) {
      // Decrypt with old key
      const decrypted = await collection.getDocument(doc._id, oldKey);
      
      // Re-encrypt with new key
      await collection.addDocument({
        ...decrypted,
        _encrypted: true
      }, newKey);
    }
  }
}
```

### Data Sanitization

```javascript
// Sanitize before storage
function sanitizeDocument(doc) {
  // Remove sensitive fields
  delete doc.data.password;
  delete doc.data.creditCard;
  
  // Mask personal information
  if (doc.data.ssn) {
    doc.data.ssn = '***-**-' + doc.data.ssn.slice(-4);
  }
  
  return doc;
}

// Apply sanitization
await collection.addDocument(sanitizeDocument(userDoc));
```

---

## Best Practices

### 1. Schema Design

```javascript
// Define clear document structures
const UserSchema = {
  _id: null,  // Auto-generated
  data: {
    profile: {
      name: String,
      email: String,
      avatar: String
    },
    settings: {
      theme: String,
      notifications: Boolean
    },
    metadata: {
      createdAt: Number,
      updatedAt: Number,
      version: Number
    }
  },
  _permanent: false,
  _encrypted: false
};
```

### 2. Error Handling

```javascript
class DatabaseService {
  async safeOperation(operation) {
    try {
      return await operation();
    } catch (error) {
      if (error.code === 'QUOTA_EXCEEDED') {
        await this.handleQuotaExceeded();
      } else if (error.code === 'TRANSACTION_FAILED') {
        await this.retryOperation(operation);
      } else {
        this.logError(error);
        throw error;
      }
    }
  }
  
  async handleQuotaExceeded() {
    const collection = await this.db.getCollection('cache');
    await collection.freeSpace(5000); // Free 5MB
  }
}
```

### 3. Migration Strategies

```javascript
async function migrateDatabase(db, fromVersion, toVersion) {
  const migrations = {
    '1.0': async () => {
      // Version 1.0 -> 1.1 migration
      const users = await db.getCollection('users');
      const docs = await users.query({});
      
      for (const doc of docs) {
        if (!doc.data.version) {
          doc.data.version = '1.1';
          await users.addDocument(doc);
        }
      }
    },
    '1.1': async () => {
      // Version 1.1 -> 1.2 migration
      await db.createCollection('analytics');
    }
  };
  
  // Execute migrations sequentially
  const versions = Object.keys(migrations).sort();
  for (const version of versions) {
    if (version > fromVersion && version <= toVersion) {
      await migrations[version]();
    }
  }
}
```

### 4. Testing Patterns

```javascript
// Unit testing example
describe('LacertaDB Operations', () => {
  let db, collection;
  
  beforeEach(async () => {
    db = new Database('test-db');
    await db.init();
    collection = await db.createCollection('test');
  });
  
  afterEach(async () => {
    await db.deleteDatabase();
  });
  
  test('Document CRUD operations', async () => {
    const doc = {
      data: { test: true },
      _permanent: false
    };
    
    // Create
    const isNew = await collection.addDocument(doc);
    expect(isNew).toBe(true);
    
    // Read
    const retrieved = await collection.getDocument(doc._id);
    expect(retrieved.data.test).toBe(true);
    
    // Update
    doc.data.test = false;
    const updated = await collection.addDocument(doc);
    expect(updated).toBe(false);
    
    // Delete
    const deleted = await collection.deleteDocument(doc._id);
    expect(deleted).toBe(true);
  });
});
```

---

## Troubleshooting

### Common Issues and Solutions

#### 1. Quota Exceeded Errors

**Symptom:** `DOMException: Quota exceeded`

**Solution:**
```javascript
// Implement automatic cleanup
db.settings.set('sizeLimitKB', 10000);
db.settings.set('freeSpaceEvery', 5000);

// Manual cleanup
const collection = await db.getCollection('cache');
await collection.freeSpace(2000); // Keep under 2MB
```

#### 2. Encryption Key Loss

**Symptom:** Documents return `false` when retrieved

**Solution:**
```javascript
// Implement key recovery mechanism
const recoveryQuestions = {
  q1: 'First pet name?',
  q2: 'Birth city?'
};

function deriveKeyFromAnswers(answers) {
  const combined = Object.values(answers).join('|');
  return crypto.subtle.digest('SHA-256', 
    new TextEncoder().encode(combined)
  );
}
```

#### 3. Performance Degradation

**Symptom:** Slow query responses

**Solution:**
```javascript
// Optimize with indexes
await collection.createIndex('data.timestamp');
await collection.createIndex('data.status');

// Use pagination
const results = await collection.query(
  { 'data.status': 'active' },
  { limit: 50, offset: 0 }
);
```

#### 4. Transaction Failures

**Symptom:** Intermittent save failures

**Solution:**
```javascript
// Implement retry logic
async function reliableSave(collection, doc, maxRetries = 3) {
  for (let i = 0; i < maxRetries; i++) {
    try {
      return await collection.addDocument(doc);
    } catch (error) {
      if (i === maxRetries - 1) throw error;
      await new Promise(r => setTimeout(r, 100 * Math.pow(2, i)));
    }
  }
}
```

---

## Migration Guide

### From LocalStorage

```javascript
// Migrate localStorage data to LacertaDB
async function migrateFromLocalStorage() {
  const db = new Database('migrated');
  await db.init();
  const collection = await db.createCollection('legacy');
  
  for (let i = 0; i < localStorage.length; i++) {
    const key = localStorage.key(i);
    const value = localStorage.getItem(key);
    
    try {
      const data = JSON.parse(value);
      await collection.addDocument({
        _id: key,
        data: data
      });
    } catch (e) {
      // Handle non-JSON values
      await collection.addDocument({
        _id: key,
        data: { value: value }
      });
    }
  }
  
  // Optionally clear localStorage
  localStorage.clear();
}
```

### From IndexedDB (Raw)

```javascript
// Migrate from raw IndexedDB to LacertaDB
async function migrateFromIndexedDB(oldDbName, storeName) {
  // Open old database
  const oldDb = await new Promise((resolve, reject) => {
    const request = indexedDB.open(oldDbName);
    request.onsuccess = () => resolve(request.result);
    request.onerror = () => reject(request.error);
  });
  
  // Initialize LacertaDB
  const newDb = new Database('migrated');
  await newDb.init();
  const collection = await newDb.createCollection(storeName);
  
  // Transfer data
  const tx = oldDb.transaction([storeName], 'readonly');
  const store = tx.objectStore(storeName);
  const request = store.getAll();
  
  request.onsuccess = async () => {
    const records = request.result;
    for (const record of records) {
      await collection.addDocument({
        data: record
      });
    }
    
    oldDb.close();
    // Optionally delete old database
    indexedDB.deleteDatabase(oldDbName);
  };
}
```

---

## License

MIT License - Copyright (c) 2024 Matias Affolter

---

## Support and Contribution

For issues, feature requests, or contributions, please refer to the project repository. The LacertaDB ecosystem welcomes community involvement in advancing browser-based data persistence paradigms.

### Development Roadmap

- **v2.0**: WebAssembly-accelerated cryptographic operations
- **v2.1**: Distributed synchronization protocols
- **v2.2**: Machine learning-driven compression algorithms
- **v2.3**: GraphQL query interface implementation

---

*Documentation Version: 1.0.0 | Last Updated: 2024*