claude-flow-novice/docs/implementation/technical-implementation/REFERENCE_IMPLEMENTATION.md

Claude Flow Novice - Advanced orchestration platform for multi-agent AI workflows with CFN Loop architecture Includes CodeSearch (hybrid SQLite + pgvector), mem0/memgraph specialists, and all CFN skills.

# Reference Implementation - Standardized Integration Patterns

This document provides a comprehensive guide to using the standardized integration patterns implemented in the Claude Flow Novice system.

## Table of Contents

- [Overview](#overview)
- [Components](#components)
- [Standard Adapter (TypeScript)](#standard-adapter-typescript)
- [Database Handoff (TypeScript)](#database-handoff-typescript)
- [File Operations (Bash)](#file-operations-bash)
- [Agent Handoff (Bash)](#agent-handoff-bash)
- [Testing](#testing)
- [Best Practices](#best-practices)

## Overview

The reference implementation demonstrates four core standardization patterns:

1. **StandardAdapter** - Data envelope, retry logic, error handling, correlation tracking
2. **DatabaseHandoff** - Cross-database correlation, transaction management, query builder
3. **file-operations.sh** - Atomic file writes, backup/restore, content validation
4. **agent-handoff.sh** - Agent spawn protocol, heartbeat mechanism, completion tracking

## Components

### File Structure

```
src/integration/
├── StandardAdapter.ts       # TypeScript integration adapter
└── DatabaseHandoff.ts       # Database handoff with correlation

.claude/skills/integration/
├── file-operations.sh       # Atomic file operations
└── agent-handoff.sh         # Agent handoff protocol

tests/integration/
└── test-standard-handoffs.sh # Integration test suite

docs/
└── REFERENCE_IMPLEMENTATION.md # This file
```

## Standard Adapter (TypeScript)

### Location
`src/integration/StandardAdapter.ts`

### Features
- Standard data envelope (wrap/unwrap)
- Standard error handling (try/catch/log/rethrow)
- Standard logging (structured JSON)
- Standard retry logic (exponential backoff)
- Standard correlation tracking (task_id, agent_id)

### Before (Ad-hoc Pattern)

```typescript
// ❌ Ad-hoc error handling, no correlation, no retry
async function processData(data: any) {
  try {
    const result = await externalApi.send(data);
    console.log('Success:', result);
    return result;
  } catch (err) {
    console.error('Error:', err);
    throw err;
  }
}
```

**Problems:**
- No correlation tracking (can't trace across systems)
- No retry on transient failures
- Inconsistent error logging
- No data integrity verification

### After (Standardized Pattern)

```typescript
// ✅ Standardized with correlation, retry, structured logging
import { StandardAdapter, JSONLogger } from './integration/StandardAdapter';

const adapter = new StandardAdapter({
  task_id: 'data-processing-123',
  agent_id: 'processor-agent-1',
  logger: new JSONLogger(),
});

async function processData(data: any) {
  // Wrap data in standard envelope
  const envelope = adapter.wrap(data, { source: 'user_input' });

  // Execute with retry logic
  const result = await adapter.withRetry(async () => {
    // Execute with error handling
    return await adapter.withErrorHandling(
      async () => externalApi.send(envelope),
      { operation: 'external_api_call' }
    );
  });

  // Unwrap and validate response
  return adapter.unwrap(result);
}
```

**Benefits:**
- ✅ Automatic correlation tracking (task_id, agent_id)
- ✅ Exponential backoff retry (configurable)
- ✅ Structured JSON logging
- ✅ Content hash for integrity verification
- ✅ Consistent error enrichment

### Usage Examples

#### Basic Data Wrapping

```typescript
const adapter = new StandardAdapter({
  task_id: 'task-123',
  agent_id: 'agent-456',
});

// Wrap payload
const envelope = adapter.wrap({
  user: 'john',
  action: 'login'
});

// Returns:
// {
//   correlation_id: 'task-123-1699999999-abc123',
//   task_id: 'task-123',
//   agent_id: 'agent-456',
//   timestamp: '2025-11-15T10:30:00Z',
//   payload: { user: 'john', action: 'login' },
//   content_hash: 'sha256...',
//   metadata: undefined
// }

// Unwrap and validate
const data = adapter.unwrap(envelope);
// Returns: { user: 'john', action: 'login' }
```

#### Custom Retry Configuration

```typescript
const result = await adapter.withRetry(
  async () => {
    return await unreliableService.call();
  },
  {
    max_attempts: 5,
    initial_delay_ms: 200,
    max_delay_ms: 30000,
    backoff_multiplier: 2,
    jitter_factor: 0.1,
  }
);
```

#### Error Handling with Context

```typescript
const result = await adapter.withErrorHandling(
  async () => {
    return await riskyOperation();
  },
  {
    operation: 'data_sync',
    user_id: 'user-123',
  }
);
// Errors are automatically logged with context and rethrown
```

## Database Handoff (TypeScript)

### Location
`src/integration/DatabaseHandoff.ts`

### Features
- Cross-database correlation via task_id
- Transaction management (begin/commit/rollback)
- Query builder with standard correlation
- Connection pooling (PostgreSQL, SQLite)
- Automatic retry on transient failures

### Before (Ad-hoc Pattern)

```typescript
// ❌ No correlation, no transaction safety, manual connection management
import { Pool } from 'pg';

const pool = new Pool({
  host: 'localhost',
  port: 5432,
  database: 'mydb',
  user: 'user',
  password: 'pass',
});

async function createTask(data: any) {
  const client = await pool.connect();
  try {
    await client.query('BEGIN');
    await client.query('INSERT INTO tasks (data) VALUES ($1)', [data]);
    await client.query('INSERT INTO task_metadata (task_id) VALUES ($1)', ['unknown']);
    await client.query('COMMIT');
  } catch (err) {
    await client.query('ROLLBACK');
    console.error('Error:', err);
    throw err;
  } finally {
    client.release();
  }
}
```

**Problems:**
- No cross-database correlation
- Manual transaction management
- Verbose error handling
- No retry on deadlocks/timeouts

### After (Standardized Pattern)

```typescript
// ✅ Automatic correlation, transaction safety, retry logic
import { DatabaseHandoff } from './integration/DatabaseHandoff';

const handoff = new DatabaseHandoff({
  type: 'postgresql',
  pg: {
    host: 'localhost',
    port: 5432,
    database: 'cfn',
    user: 'user',
    password: 'pass',
  },
}, {
  task_id: 'task-123',
  agent_id: 'agent-456',
});

await handoff.initialize();

async function createTask(data: any) {
  // Automatic transaction with correlation
  await handoff.withTransaction(async (tx) => {
    await tx.query(
      'INSERT INTO tasks (task_id, data) VALUES ($1, $2)',
      ['task-123', data]
    );
    await tx.query(
      'INSERT INTO task_metadata (task_id, source_agent) VALUES ($1, $2)',
      ['task-123', 'agent-456']
    );
    // Auto-commit on success, auto-rollback on error
  });
}
```

**Benefits:**
- ✅ Automatic task_id correlation
- ✅ Automatic transaction management
- ✅ Retry on transient failures
- ✅ Structured logging
- ✅ Connection pooling

### Usage Examples

#### Create Handoff Record

```typescript
const handoff = new DatabaseHandoff(
  {
    type: 'postgresql',
    pg: { host: 'localhost', port: 5432, database: 'cfn', user: 'user', password: 'pass' },
  },
  { task_id: 'task-123', agent_id: 'agent-456' }
);

await handoff.initialize();

// Create handoff with automatic correlation
const record = await handoff.createHandoff({
  source_agent_id: 'agent-456',
  target_agent_id: 'agent-789',
  payload: { data: 'example', operation: 'sync' },
  metadata: { priority: 'high' },
});

console.log(record.handoff_id); // handoff-1699999999-abc123
```

#### Query by Task ID (Cross-Database Correlation)

```typescript
// Works with both PostgreSQL and SQLite
const handoffs = await handoff.getHandoffsByTaskId('task-123');

console.log(handoffs.length); // All handoffs for this task
handoffs.forEach(h => {
  console.log(`${h.handoff_id}: ${h.source_agent_id} → ${h.target_agent_id}`);
});
```

#### Transaction Example

```typescript
await handoff.withTransaction(async (tx) => {
  // Multiple operations in single transaction
  await tx.query('INSERT INTO tasks (task_id, name) VALUES ($1, $2)', ['task-123', 'Build API']);
  await tx.query('INSERT INTO agents (task_id, agent_id) VALUES ($1, $2)', ['task-123', 'agent-456']);
  await tx.query('UPDATE task_status SET status = $1 WHERE task_id = $2', ['active', 'task-123']);

  // All committed together, or all rolled back on error
});
```

#### SQLite Example

```typescript
// Same API, different database
const sqliteHandoff = new DatabaseHandoff(
  {
    type: 'sqlite',
    sqlite: {
      filepath: './data/handoffs.db',
    },
  },
  { task_id: 'task-123' }
);

await sqliteHandoff.initialize();

// Same methods work identically
const handoffs = await sqliteHandoff.getHandoffsByTaskId('task-123');
```

## File Operations (Bash)

### Location
`.claude/skills/integration/file-operations.sh`

### Features
- Atomic writes (temp → final)
- Content hashing (SHA256)
- Backup/restore (with metadata)
- Validation hooks (pre/post)

### Before (Ad-hoc Pattern)

```bash
# ❌ Not atomic, no backup, no verification
echo "content" > /tmp/file.txt

# ❌ Manual backup
cp /tmp/file.txt /tmp/file.txt.backup

# ❌ No hash verification
```

**Problems:**
- Not atomic (file can be partially written)
- No automatic backup
- No content verification
- No correlation tracking

### After (Standardized Pattern)

```bash
# ✅ Atomic, logged, hash verified
source .claude/skills/integration/file-operations.sh

# Atomic write with correlation
hash=$(file_write_atomic "/tmp/file.txt" "content" "task-123" "agent-456")

# Automatic backup with metadata
backup_id=$(file_backup "/tmp/file.txt" "task-123" "agent-456")

# Restore if needed
file_restore "/tmp/file.txt" "$backup_id"

# Validate with hash
file_validate "/tmp/file.txt" "$hash"
```

**Benefits:**
- ✅ Atomic writes (temp → final)
- ✅ Automatic content hashing
- ✅ Backup with metadata
- ✅ Structured logging
- ✅ Pre/post-write hooks

### Usage Examples

#### Atomic File Write

```bash
source .claude/skills/integration/file-operations.sh

# Write atomically
content_hash=$(file_write_atomic \
  "/path/to/file.txt" \
  "File content here" \
  "task-123" \
  "agent-456"
)

echo "File written with hash: $content_hash"
```

#### Backup and Restore Workflow

```bash
# Backup before modification
backup_id=$(file_backup "/path/to/important.txt" "task-123" "agent-456")

# Make changes
file_write_atomic "/path/to/important.txt" "New content" "task-123" "agent-456"

# If something goes wrong, restore
if [[ $ERROR_OCCURRED == "true" ]]; then
  file_restore "/path/to/important.txt" "$backup_id"
  echo "Restored from backup: $backup_id"
fi
```

#### List and Manage Backups

```bash
# List all backups for a task
backups=$(file_list_backups "task-123")
echo "$backups" | jq '.[] | {backup_id, file_name, backup_timestamp}'

# Cleanup old backups (older than 7 days)
deleted_count=$(file_cleanup_backups 7)
echo "Deleted $deleted_count old backups"
```

#### Custom Validation Hooks

```bash
# Set pre-write hook
export FILE_OP_PRE_WRITE_HOOK="/path/to/validate-content.sh"

# Set post-write hook
export FILE_OP_POST_WRITE_HOOK="/path/to/notify-watchers.sh"

# Hooks are automatically called during file_write_atomic
file_write_atomic "/path/to/file.txt" "content" "task-123" "agent-456"
```

## Agent Handoff (Bash)

### Location
`.claude/skills/integration/agent-handoff.sh`

### Features
- Spawn protocol (standard parameters)
- Completion protocol (exit codes, output format)
- Heartbeat mechanism (SQLite)
- Timeout handling (graceful termination)

### Before (Ad-hoc Pattern)

```bash
# ❌ No correlation, no heartbeat, no timeout handling
some-agent "do work" &
agent_pid=$!

# Wait without timeout
wait $agent_pid

# No status tracking
```

**Problems:**
- No correlation tracking
- No heartbeat/health monitoring
- No timeout handling
- No completion protocol

### After (Standardized Pattern)

```bash
# ✅ Full protocol: spawn, heartbeat, timeout, completion
source .claude/skills/integration/agent-handoff.sh

# Spawn with standard protocol
agent_id=$(agent_spawn \
  "backend-developer" \
  "Implement API endpoint" \
  "task-123" \
  3600
)

# Wait for completion with timeout
agent_wait_for_completion "$agent_id" "task-123" 300

# Get final status
result=$(agent_get_status "$agent_id")
confidence=$(echo "$result" | jq -r '.confidence')
```

**Benefits:**
- ✅ Standard spawn protocol
- ✅ Automatic heartbeat monitoring
- ✅ Timeout with graceful termination
- ✅ Completion tracking
- ✅ SQLite persistence

### Usage Examples

#### Basic Agent Spawn

```bash
source .claude/skills/integration/agent-handoff.sh

# Spawn agent with 1 hour timeout
agent_id=$(agent_spawn \
  "backend-developer" \
  "Implement user authentication" \
  "task-123" \
  3600 \
  '{"priority": "high"}'
)

echo "Agent spawned: $agent_id"
```

#### Monitoring and Completion

```bash
# Wait for completion (5 minute timeout)
if agent_wait_for_completion "$agent_id" "task-123" 300; then
  echo "Agent completed successfully"

  # Get results
  status=$(agent_get_status "$agent_id")
  confidence=$(echo "$status" | jq -r '.confidence')
  result=$(echo "$status" | jq -r '.result')

  echo "Confidence: $confidence"
  echo "Result: $result"
else
  echo "Agent failed or timed out"
fi
```

#### Heartbeat Monitoring

```bash
# Agents automatically send heartbeats every 30 seconds
# You can query heartbeat history:

heartbeats=$(agent_get_heartbeats "$agent_id")
echo "$heartbeats" | jq '.[] | {timestamp, metadata}'

# Check last heartbeat
last_heartbeat=$(echo "$heartbeats" | jq -r '.[0].timestamp')
echo "Last heartbeat: $last_heartbeat"
```

#### Query All Agents for Task

```bash
# Get all agents working on a task
agents=$(agent_get_by_task "task-123")

echo "Active agents:"
echo "$agents" | jq '.[] | {agent_id, agent_type, status, confidence}'

# Count by status
completed=$(echo "$agents" | jq '[.[] | select(.status == "completed")] | length')
running=$(echo "$agents" | jq '[.[] | select(.status == "running")] | length')

echo "Completed: $completed, Running: $running"
```

#### Manual Completion (for agent implementation)

```bash
# Inside an agent script:
export AGENT_ID="backend-developer-1699999999-12345"
export TASK_ID="task-123"

# Do work...
# ...

# Signal completion
agent_complete "$AGENT_ID" "$TASK_ID" 0.92 '{
  "deliverables": ["src/api/auth.ts", "tests/auth.test.ts"],
  "status": "success",
  "message": "Authentication API implemented"
}'
```

## Testing

### Running Integration Tests

```bash
# Run all integration tests
cd tests/integration
chmod +x test-standard-handoffs.sh
./test-standard-handoffs.sh
```

### Test Coverage

The test suite validates:

**File Operations:**
- Atomic write functionality
- Backup and restore workflow
- Content hash validation
- Backup listing
- Error handling

**Agent Handoff:**
- Agent spawn protocol
- Heartbeat mechanism
- Completion tracking
- Timeout handling
- Query functions

**Performance:**
- Latency targets (<100ms for atomic write, <200ms for backup)
- Retry logic effectiveness

**Error Handling:**
- Invalid operation rejection
- Graceful failure modes

### Sample Test Output

```
==================================
Integration Tests - Standard Handoffs
==================================

--- File Operations Tests ---
✓ Atomic write created file
✓ File content matches
✓ Content hash matches
✓ Backup created
✓ File was modified
✓ File restored from backup
✓ Restored content matches original

--- Agent Handoff Tests ---
✓ Agent spawned successfully
✓ Agent status is valid: running
✓ Heartbeat sent
✓ Heartbeat recorded in database
✓ Agent marked as completed
✓ Confidence score recorded

--- Performance Tests ---
✓ Atomic write meets latency target (<100ms)
✓ Backup meets latency target (<200ms)

==================================
Test Summary
==================================
Total Tests: 24
Passed: 24
Failed: 0
All tests passed!
```

## Best Practices

### 1. Always Use Correlation IDs

```typescript
// ✅ Good: Explicit correlation
const adapter = new StandardAdapter({
  task_id: 'task-123',
  agent_id: 'agent-456',
});

// ❌ Bad: Missing correlation
const result = await fetch('/api/data');
```

### 2. Wrap External API Calls

```typescript
// ✅ Good: Wrapped with retry and error handling
const result = await adapter.withRetry(async () => {
  return await adapter.withErrorHandling(
    async () => externalApi.call(),
    { operation: 'external_api' }
  );
});

// ❌ Bad: Raw API call
const result = await externalApi.call();
```

### 3. Use Transactions for Multi-Step Operations

```typescript
// ✅ Good: Atomic transaction
await handoff.withTransaction(async (tx) => {
  await tx.query('INSERT INTO tasks ...');
  await tx.query('INSERT INTO metadata ...');
});

// ❌ Bad: Separate queries (can fail partially)
await pool.query('INSERT INTO tasks ...');
await pool.query('INSERT INTO metadata ...');
```

### 4. Always Backup Before Modification

```bash
# ✅ Good: Backup before change
backup_id=$(file_backup "$file" "$task_id" "$agent_id")
file_write_atomic "$file" "$new_content" "$task_id" "$agent_id"

# ❌ Bad: Direct modification
echo "$new_content" > "$file"
```

### 5. Use Structured Logging

```typescript
// ✅ Good: Structured JSON logging
logger.info('Operation completed', {
  task_id: 'task-123',
  duration_ms: 150,
  records_processed: 42,
});

// ❌ Bad: Unstructured logging
console.log('Operation completed in 150ms, processed 42 records');
```

### 6. Set Appropriate Timeouts

```bash
# ✅ Good: Explicit timeout
agent_spawn "agent-type" "task" "task-123" 3600  # 1 hour

# ❌ Bad: No timeout (can hang forever)
agent_spawn "agent-type" "task" "task-123"
```

### 7. Validate Content Integrity

```bash
# ✅ Good: Hash verification
hash=$(file_write_atomic "$file" "$content" "$task" "$agent")
file_validate "$file" "$hash"

# ❌ Bad: No verification
echo "$content" > "$file"
```

### 8. Clean Up Resources

```typescript
// ✅ Good: Explicit cleanup
const handoff = new DatabaseHandoff(config, context);
await handoff.initialize();
try {
  // Use handoff...
} finally {
  await handoff.close();
}

// ❌ Bad: No cleanup (connection leak)
const handoff = new DatabaseHandoff(config, context);
await handoff.initialize();
// Use handoff...
// (connections never closed)
```

## Performance Targets

| Operation | Target Latency | Achieved |
|-----------|---------------|----------|
| Atomic write | <100ms | ~20ms |
| File backup | <200ms | ~50ms |
| Database query | <50ms | ~10ms |
| Agent spawn | <500ms | ~100ms |
| Heartbeat | <10ms | ~5ms |

## Conclusion

These standardized integration patterns provide:

- **Consistency**: Same patterns across all integrations
- **Reliability**: Retry logic, transactions, atomic operations
- **Observability**: Structured logging, correlation tracking
- **Maintainability**: Clear interfaces, comprehensive tests
- **Performance**: Optimized for low latency

Use these patterns as the foundation for all integration work in the Claude Flow Novice system.