yq-store/README.md

A high-performance, persistent and in-memory key-value store with TTL support, built for Node.js applications.

> A blazingly fast, zero-dependency key-value store that just works. Built for developers who need reliable data persistence without the complexity.

## Table of Contents

- [🚀 Why yq-store?](#-why-yq-store) - *Discover what makes yq-store special*
- [⚡ Quick Start](#-quick-start) - *Get up and running in 30 seconds*
- [✨ Key Features](#-key-features) - *What you get out of the box*
- [🏗️ How It Works](#️-how-it-works) - *The architecture that powers performance*
- [📦 Installation](#-installation) - *Simple npm install*
- [📚 Usage Guide](#-usage-guide) - *Learn by example*
  - [Basic Operations](#basic-operations) - *Store, retrieve, and delete data*
  - [Time-To-Live (TTL)](#time-to-live-ttl) - *Automatic data expiration*
  - [Event Monitoring](#event-monitoring) - *React to store changes*
  - [Batch Operations](#batch-operations) - *Process multiple operations efficiently*
  - [Transactions](#transactions) - *ACID-compliant data operations*
  - [Querying & Filtering](#querying--filtering) - *Find data with ease*
  - [Storage Modes](#storage-modes) - *Memory vs persistent storage*
- [🔧 Configuration](#-configuration) - *Customize for your needs*
- [📖 API Reference](#-api-reference) - *Complete method documentation*
- [⚡ Performance](#-performance) - *Benchmarks and optimization tips*
- [🎯 Use Cases](#-use-cases) - *Real-world applications*
- [🤝 Contributing](#-contributing) - *Join the community*
- [📄 License](#-license) - *MIT licensed*

## 🚀 Why yq-store?

Building applications shouldn't mean wrestling with complex databases or sacrificing performance for simplicity. **yq-store** gives you the best of both worlds:

- **Zero Setup Complexity**: No database servers, no configuration files, no headaches
- **Production Ready**: Built on SQLite's rock-solid foundation with 20+ years of battle-testing
- **Developer Friendly**: TypeScript-first with intuitive APIs that feel natural
- **Performance Focused**: Optimized for real-world workloads, not just benchmarks

## ⚡ Quick Start

Get started in less than a minute:

```typescript
import { YqStore } from 'yq-store';

// Create your store
const store = await YqStore.create();

// Store some data
await store.set('user:123', {
  name: 'Sarah Chen',
  email: 'sarah@example.com',
  preferences: { theme: 'dark', notifications: true }
});

// Retrieve it later
const user = await store.get('user:123');
console.log(`Welcome back, ${user.name}!`);

// Clean up when done
await store.close();
```

That's it! Your data is automatically persisted to disk and will survive application restarts.

## ✨ Key Features

### 🏎️ **Blazing Fast Performance**
- 20,000+ reads per second
- Optimized SQLite backend with WAL mode
- Intelligent indexing and caching

### 🛡️ **Rock Solid Reliability**
- ACID transactions
- Automatic data recovery
- Battle-tested SQLite foundation

### ⏰ **Smart Data Management**
- Built-in TTL (Time-To-Live) support
- Automatic cleanup and compaction
- Configurable memory limits

### 🎯 **Developer Experience**
- Full TypeScript support
- Zero external dependencies
- Intuitive, promise-based API

### 🔍 **Powerful Querying**
- Prefix and suffix filtering
- Pagination support
- Batch operations

## 🏗️ How It Works

**yq-store** is built on a modern, performance-optimized architecture:

```
┌─────────────────┐    ┌──────────────────┐    ┌─────────────────┐
│   Your App     │───▶│    yq-store      │───▶│   SQLite DB     │
│                │    │                  │    │                 │
│ TypeScript API │    │ • Event System   │    │ • WAL Mode      │
│ Promise-based  │    │ • TTL Management │    │ • Transactions  │
│ Type-safe      │    │ • Auto Cleanup   │    │ • Indexing      │
└─────────────────┘    └──────────────────┘    └─────────────────┘
```

- **SQLite Backend**: Leverages SQLite's proven reliability and performance
- **WAL Mode**: Write-Ahead Logging enables concurrent reads during writes
- **Smart Indexing**: Optimized B-tree indexes for lightning-fast lookups
- **Memory Management**: LRU eviction and configurable limits prevent memory bloat
- **Event System**: Real-time notifications for all store operations

## 📦 Installation

```bash
npm install yq-store
```

No additional setup required! yq-store works out of the box with zero configuration.

## 📚 Usage Guide

### Basic Operations

The core operations you'll use every day:

```typescript
import { YqStore } from 'yq-store';

// Create your store
const store = await YqStore.create();

// Store any JSON-serializable data
await store.set('product:laptop-123', {
  name: 'MacBook Pro',
  price: 3999,
  specs: {
    cpu: 'M3 Pro',
    memory: '48GB',
    storage: '1TB SSD'
  },
  inStock: true
});

// Retrieve with full type safety
interface Product {
  name: string;
  price: number;
  specs: Record<string, string>;
  inStock: boolean;
}

const laptop = await store.get<Product>('product:laptop-123');
if (laptop) {
  console.log(`${laptop.name} - $${laptop.price}`);
  console.log(`In stock: ${laptop.inStock ? 'Yes' : 'No'}`);
}

// Check existence without retrieving
const exists = await store.has('product:laptop-123');
console.log(`Product exists: ${exists}`);

// Remove when no longer needed
const wasDeleted = await store.delete('product:laptop-123');
console.log(`Deletion successful: ${wasDeleted}`);

// Get store statistics
const activeCount = await store.getSize('active');
const totalCount = await store.getSize('all');
console.log(`Active entries: ${activeCount}, Total: ${totalCount}`);
```

### Time-To-Live (TTL)

Perfect for sessions, caches, and temporary data:

```typescript
// Cache API responses for 5 minutes
await store.set('api:weather:london', {
  temperature: 18,
  condition: 'partly-cloudy',
  humidity: 65,
  timestamp: Date.now()
}, 300); // 300 seconds = 5 minutes

// Store user sessions that expire in 24 hours
await store.set('session:user-456', {
  userId: '456',
  loginTime: new Date().toISOString(),
  permissions: ['read', 'write'],
  csrfToken: 'abc123xyz'
}, 86400); // 24 hours

// The data automatically disappears when it expires
setTimeout(async () => {
  const weather = await store.get('api:weather:london');
  console.log(weather); // null - expired and cleaned up
}, 301000); // Check after expiration
```

### Event Monitoring

Stay informed about what's happening in your store:

```typescript
// Monitor store lifecycle
store.on('ready', () => {
  console.log('🟢 Store is ready for operations');
});

store.on('close', () => {
  console.log('🔴 Store has been closed');
});

// Track data operations
store.on('set', (key, value) => {
  console.log(`📝 Stored: ${key}`);
  // Log to analytics, update UI, etc.
});

store.on('delete', (key) => {
  console.log(`🗑️ Deleted: ${key}`);
});

store.on('expire', (key) => {
  console.log(`⏰ Expired: ${key}`);
});

// Monitor performance and maintenance
store.on('compact:start', () => {
  console.log('🔧 Starting database optimization...');
});

store.on('compact:end', (stats) => {
  console.log(`✅ Optimization complete. Processed ${stats.keysProcessed} keys`);
});

// Handle errors gracefully
store.on('error', (error) => {
  console.error('❌ Store error:', error.message);
  // Send to error tracking service
});
```

### Batch Operations

Process multiple operations efficiently and atomically:

```typescript
// Import user data in a single atomic operation
await store.batch([
  {
    type: 'put',
    key: 'user:alice',
    value: { name: 'Alice Johnson', role: 'admin' }
  },
  {
    type: 'put',
    key: 'user:bob',
    value: { name: 'Bob Smith', role: 'user' },
    ttlSeconds: 3600 // Bob's account expires in 1 hour
  },
  {
    type: 'put',
    key: 'user:charlie',
    value: { name: 'Charlie Brown', role: 'moderator' }
  },
  {
    type: 'del',
    key: 'user:old-account' // Remove old account
  }
]);

console.log('✅ All users imported successfully!');
```

### Transactions

Ensure data consistency with ACID transactions:

```typescript
// Transfer credits between user accounts
const tx = store.createTransaction();

try {
  // Get current balances
  const sender = await store.get<{credits: number}>('user:alice:credits');
  const receiver = await store.get<{credits: number}>('user:bob:credits');
  
  if (!sender || sender.credits < 100) {
    throw new Error('Insufficient credits');
  }
  
  // Prepare the transfer
  tx.put('user:alice:credits', { credits: sender.credits - 100 });
  tx.put('user:bob:credits', { credits: (receiver?.credits || 0) + 100 });
  tx.put('transaction:log', {
    from: 'alice',
    to: 'bob',
    amount: 100,
    timestamp: Date.now()
  });
  
  // Execute atomically
  await tx.commit();
  console.log('💰 Transfer completed successfully!');
  
} catch (error) {
  // Rollback on any error
  tx.rollback();
  console.error('❌ Transfer failed:', error.message);
}
```

### Querying & Filtering

Find and process data efficiently:

```typescript
// Find all user records
const userKeys = await store.listKeys({ prefix: 'user:' });
console.log(`Found ${userKeys.length} users`);

// Get paginated results
const firstPage = await store.list<User>({
  prefix: 'user:',
  limit: 10,
  page: 1
});

firstPage.forEach(({ key, value }) => {
  console.log(`${key}: ${value.name} (${value.role})`);
});

// Process all session data
await store.forEach<Session>((key, session) => {
  if (session.lastActivity < Date.now() - 86400000) {
    console.log(`Inactive session: ${key}`);
    // Could mark for cleanup
  }
}, {
  prefix: 'session:',
  limit: 1000 // Process in batches
});

// Find configuration files
const configKeys = await store.listKeys({ suffix: '.config' });
configKeys.forEach(key => {
  console.log(`Config file: ${key}`);
});
```

### Storage Modes

Choose the right storage mode for your use case:

```typescript
// In-memory store (perfect for testing or temporary data)
const memoryStore = await YqStore.create({
  storage: {
    type: 'memory',
    maxEntries: 50000,
    maxMemory: 100 * 1024 * 1024 // 100MB limit
  }
});

// Persistent store with custom location
const persistentStore = await YqStore.create({
  storage: {
    type: 'persistence',
    persistence: {
      dbDir: './data/stores',
      dbFileName: 'my-application-store',
      sqlite: {
        journalMode: 'WAL',
        synchronous: 'NORMAL',
        cacheSize: -50000 // 50MB cache
      }
    }
  },
  compactionInterval: 1800000, // Compact every 30 minutes
  softDelete: true // Enable soft deletes for data recovery
});
```

## 🔧 Configuration

Customize yq-store to fit your specific needs:

```typescript
const store = await YqStore.create({
  // Automatic maintenance
  compactionInterval: 3600000, // Compact every hour (0 = disabled)
  ttl: 0, // Default TTL in seconds (0 = no expiration)
  softDelete: true, // Enable soft deletes for data recovery
  
  // Storage configuration
  storage: {
    type: 'persistence', // 'memory' or 'persistence'
    maxEntries: 1000000, // Maximum number of entries
    maxMemory: 500 * 1024 * 1024, // 500MB memory limit
    
    persistence: {
      dbDir: './data', // Where to store the database
      dbFileName: 'app-store', // Custom database name
      
      sqlite: {
        journalMode: 'WAL', // WAL mode for better concurrency
        synchronous: 'NORMAL', // Balance safety and performance
        cacheSize: -20000, // 20MB cache (negative = KB)
        tempStore: 'memory', // Use memory for temp operations
        busyTimeout: 30000 // 30 second timeout for locks
      },
      
      vacuum: {
        enabled: true,
        mode: 'incremental' // Gradual cleanup
      }
    }
  },
  
  // Development and debugging
  debug: {
    enabled: false, // Enable debug mode
    timing: false, // Log operation timings
    sqlLogging: false // Log SQL queries
  },
  
  // Logging configuration
  logging: {
    enabled: false,
    logDir: './logs'
  }
});
```

## 📖 API Reference

### Core Methods

#### `YqStore.create(options?: KvStoreOptions): Promise<YqStore>`
Creates and initializes a new store instance.

#### `has(key: string): Promise<boolean>`
Checks if a key exists and hasn't expired.

#### `get<T>(key: string): Promise<T | null>`
Retrieves a value by key with full type safety.

#### `set<T>(key: string, value: T, ttlSeconds?: number): Promise<void>`
Stores a value with optional TTL expiration.

#### `delete(key: string): Promise<boolean>`
Deletes a key and returns whether it existed.

#### `getSize(): Promise<number>`
Gets the total count of all entries in the store.

#### `getSize(type: 'active' | 'deleted' | 'expired' | 'all'): Promise<number>`
Gets the count of entries by type.

#### `getSize(options: {prefix: string}): Promise<number>`
Gets the count of entries with keys that start with the specified prefix.

#### `getSize(options: {suffix: string}): Promise<number>`
Gets the count of entries with keys that end with the specified suffix.

#### `getSize(type: 'active' | 'deleted' | 'expired' | 'all', options: {prefix: string}): Promise<number>`
Gets the count of entries by type with keys that start with the specified prefix.

#### `getSize(type: 'active' | 'deleted' | 'expired' | 'all', options: {suffix: string}): Promise<number>`
Gets the count of entries by type with keys that end with the specified suffix.

### Query Methods

#### `listKeys(): Promise<string[]>`
Lists all keys in the store.

#### `listKeys(options: {prefix: string}): Promise<string[]>`
Lists all keys that start with the specified prefix.

#### `listKeys(options: {suffix: string}): Promise<string[]>`
Lists all keys that end with the specified suffix.

#### `list<T>(options?: ListOptions): Promise<Array<{key: string, value: T}>>`
Retrieves key-value pairs with pagination support.

#### `forEach<T>(callback: (key: string, value: T) => void, options?: ForEachOptions): Promise<void>`
Iterates over key-value pairs matching criteria.

### Batch Operations

#### `batch(operations: BatchOperation[]): Promise<void>`
Executes multiple operations atomically.

#### `createTransaction(): Transaction`
Creates a new transaction for ACID operations.

### Maintenance

#### `compact(): Promise<void>`
Manually triggers store compaction and optimization.

#### `clearSoftDeletedEntries(): Promise<number>`
Permanently removes soft-deleted entries.

#### `close(): Promise<void>`
Closes the store and releases all resources.

### Events

#### `on<E>(event: E, listener: KvStoreEvents[E]): this`
Registers an event listener.

#### `off<E>(event: E, listener: KvStoreEvents[E]): this`
Removes an event listener.

### Available Events

- `ready` - Store is initialized and ready
- `db:ready` - Database connection established
- `set` - Key-value pair was stored
- `delete` - Key was deleted
- `expire` - Key expired due to TTL
- `evict` - Entries evicted due to memory limits
- `compact:start` - Compaction started
- `compact:end` - Compaction completed
- `error` - Error occurred
- `close` - Store was closed

## ⚡ Performance

**yq-store** is optimized for real-world performance:

### Benchmark Results

*Tested on MacBook Pro M3 with 48GB RAM*

| Operation | Throughput | Notes |
|-----------|------------|-------|
| Sequential Writes | ~4,400 ops/sec | Single key operations |
| Batch Writes | ~8,600 ops/sec | 100 operations per batch |
| Sequential Reads | ~30,000 ops/sec | Cache-friendly access |
| Random Reads | ~32,000 ops/sec | Real-world mixed access |

### Performance Tips

1. **Use Batch Operations**: Group multiple writes for better throughput
2. **Configure Cache Size**: Increase SQLite cache for read-heavy workloads
3. **Choose Storage Mode**: Use memory mode for temporary data
4. **Monitor Events**: Track performance with timing events
5. **Tune Compaction**: Adjust interval based on write patterns

### Memory Usage

- **Base overhead**: ~2-5MB for the store instance
- **Per entry**: ~100-200 bytes depending on key/value size
- **Cache**: Configurable SQLite cache (default: 10MB)
- **Automatic cleanup**: LRU eviction prevents memory bloat

## 🎯 Use Cases

**yq-store** excels in these scenarios:

### 🌐 **Web Applications**
```typescript
// Session management
await store.set(`session:${sessionId}`, userSession, 3600);

// Feature flags
await store.set('feature:new-ui', { enabled: true, rollout: 0.1 });

// User preferences
await store.set(`prefs:${userId}`, { theme: 'dark', lang: 'en' });
```

### 🚀 **Microservices**
```typescript
// Service configuration
await store.set('config:api-keys', { stripe: 'sk_...', sendgrid: 'SG...' });

// Circuit breaker state
await store.set('circuit:payment-service', { state: 'closed', failures: 0 });

// Rate limiting
await store.set(`rate:${clientId}`, { requests: 1, window: Date.now() }, 60);
```

### 📱 **Desktop Applications**
```typescript
// Application state
await store.set('app:window-state', { x: 100, y: 100, width: 800, height: 600 });

// User data
await store.set('user:recent-files', ['/path/to/file1.txt', '/path/to/file2.txt']);

// Plugin configuration
await store.set('plugins:enabled', ['spell-check', 'auto-save', 'git-integration']);
```

### 🧪 **Development & Testing**
```typescript
// Test fixtures
const testStore = await YqStore.create({ storage: { type: 'memory' } });
await testStore.set('test:user', mockUser);

// Development cache
await store.set('dev:api-response', cachedResponse, 300);
```

## 🤝 Contributing

We love contributions! Here's how you can help:

1. **🐛 Report Bugs**: Open an issue with reproduction steps
2. **💡 Suggest Features**: Share your ideas for improvements
3. **📝 Improve Docs**: Help make our documentation even better
4. **🔧 Submit PRs**: Fix bugs or add features

Before contributing, please read our [Contributing Guide](CONTRIBUTING.md).

## 📄 License

MIT License - see the [LICENSE](LICENSE) file for details.

---

**Built with ❤️ by Yuniq Solutions Tech**

*Need help? Have questions? [Open an issue](https://github.com/yuniqsolutions/yq-store/issues) - we're here to help!*