@tehreet/conduit/docs/conduit-synapse-scaling-plan.md

LLM API gateway with intelligent routing, robust process management, and health monitoring

# Conduit-Synapse Scaling Implementation Plan

## Executive Summary

This document outlines the comprehensive plan to transform Conduit from a CLI-focused tool into a scalable Node.js library that can be seamlessly integrated with Synapse while maintaining backward compatibility with existing CLI usage.

## Current State Analysis

### Conduit Architecture Overview

Conduit currently operates as a sophisticated CLI tool with the following components:

```
conduit/
├── src/
│   ├── cli.ts                    # CLI entry point
│   ├── index.ts                  # Library exports (basic)
│   ├── server.ts                 # Basic server
│   ├── serverWrapper.ts          # Enhanced server with health checks
│   ├── core/                     # Core functionality
│   │   ├── claude-wrapper.ts     # Claude CLI wrapper with routing
│   │   ├── metadata-handler.ts   # Stream metadata injection
│   │   └── index.ts              # Core exports
│   ├── features/                 # Feature modules
│   │   ├── synapse-features.ts   # Synapse-specific features
│   │   └── index.ts              # Feature exports
│   ├── utils/                    # Utility modules
│   │   ├── config-presets.ts     # Configuration presets
│   │   ├── context-extractor.ts  # Context extraction
│   │   ├── token-counter.ts      # Token counting
│   │   ├── router.ts             # Model routing logic
│   │   └── pidManager.ts         # Process management
│   └── plugins/                  # Plugin system
│       ├── plugin-interface.ts   # Plugin interface
│       ├── plugin-manager.ts     # Plugin management
│       └── synapse-plugin.ts     # Synapse plugin
├── dist/
│   ├── cli.js                    # Bundled CLI executable
│   └── tiktoken_bg.wasm          # WebAssembly token counter
└── config/
    └── synapse-preset.json       # Synapse configuration preset
```

### Current Limitations

1. **CLI-Only Build**: Only builds bundled CLI executable
2. **No Library API**: No clean programmatic interface
3. **No TypeScript Definitions**: No `.d.ts` files for library consumers
4. **Process Management**: Tied to CLI commands and file system
5. **Configuration Loading**: Expects JSON files on disk
6. **Logging**: Hardcoded to file system

## Implementation Plan

### Phase 1: Core Library Infrastructure (Week 1)

#### 1.1 Dual Build System Setup

**Update Build Configuration**
```json
// package.json
{
  "main": "./dist/lib/index.js",
  "types": "./dist/lib/index.d.ts",
  "bin": {
    "conduit": "./dist/cli.js"
  },
  "exports": {
    ".": {
      "import": "./dist/lib/index.js",
      "require": "./dist/lib/index.js",
      "types": "./dist/lib/index.d.ts"
    },
    "./server": {
      "import": "./dist/lib/server.js",
      "require": "./dist/lib/server.js",
      "types": "./dist/lib/server.d.ts"
    },
    "./router": {
      "import": "./dist/lib/router.js",
      "require": "./dist/lib/router.js",
      "types": "./dist/lib/router.d.ts"
    },
    "./plugins": {
      "import": "./dist/lib/plugins.js",
      "require": "./dist/lib/plugins.js",
      "types": "./dist/lib/plugins.d.ts"
    }
  },
  "scripts": {
    "build": "npm run build:lib && npm run build:cli",
    "build:lib": "tsc -p tsconfig.lib.json",
    "build:cli": "esbuild src/cli.ts --bundle --platform=node --outfile=dist/cli.js && shx cp node_modules/tiktoken/tiktoken_bg.wasm dist/tiktoken_bg.wasm",
    "build:watch": "npm run build:lib -- --watch",
    "dev": "npm run build:watch"
  }
}
```

**Create Library-Specific TypeScript Config**
```json
// tsconfig.lib.json
{
  "extends": "./tsconfig.json",
  "compilerOptions": {
    "outDir": "./dist/lib",
    "declaration": true,
    "declarationMap": true,
    "module": "CommonJS",
    "target": "ES2020",
    "moduleResolution": "node",
    "esModuleInterop": true,
    "allowSyntheticDefaultImports": true
  },
  "include": ["src/**/*.ts"],
  "exclude": ["src/cli.ts", "**/*.test.ts"]
}
```

#### 1.2 Enhanced Library Exports

**Create Main Library Entry Point**
```typescript
// src/lib.ts
export * from './core';
export * from './features';
export * from './utils/config-presets';
export * from './utils/context-extractor';
export * from './utils/token-counter';
export * from './utils/router';
export * from './plugins';

// Enhanced server exports
export { 
  createConduitServer, 
  ConduitServer,
  type EnhancedServerConfig 
} from './serverWrapper';

// Main run function
export { run } from './index';

// Convenience factory functions
export const createConduitClient = (config: ConduitConfig) => {
  return new ConduitServer({ initialConfig: config });
};

export const createConduitRouter = (config: ConduitConfig) => {
  return new ConduitRouter(config);
};
```

**Update Index File**
```typescript
// src/index.ts (updated)
export * from './lib';

// Keep existing run function for backward compatibility
export { run } from './server';
```

#### 1.3 Core Type Definitions

**Create Comprehensive Type Definitions**
```typescript
// src/types/index.ts
export interface ConduitConfig {
  preset?: string;
  presetConfig?: ConfigPreset;
  Providers?: ProviderConfig[];
  Router?: RouterConfig;
  plugins?: PluginConfig[];
  server?: ServerConfig;
}

export interface ServerConfig {
  port?: number;
  host?: string;
  cors?: boolean;
  healthCheck?: boolean;
  gracefulShutdown?: boolean;
  timeout?: number;
}

export interface ProviderConfig {
  name: string;
  api_base_url: string;
  api_key: string;
  models: string[];
  transformer?: {
    use: string[];
  };
}

export interface RouterConfig {
  default?: string;
  longContext?: string;
  background?: string;
  think?: string;
  customRules?: RoutingRule[];
}

export interface RoutingRule {
  id: string;
  name: string;
  conditions: RuleCondition[];
  target: string;
  priority: number;
  enabled: boolean;
}

export interface RuleCondition {
  field: string;
  operator: 'gt' | 'lt' | 'eq' | 'contains' | 'matches';
  value: any;
}

export interface RoutingContext {
  tokenCount: number;
  model: string;
  message: string;
  metadata: Record<string, any>;
}

export interface RoutingDecision {
  model: string;
  provider: string;
  reason: string;
  confidence: number;
  metadata: Record<string, any>;
}
```

### Phase 2: Enhanced API Design (Week 2)

#### 2.1 Programmatic Server Management

**Create Enhanced Server Class**
```typescript
// src/server/ConduitServer.ts
export class ConduitServer {
  private server: any;
  private config: ConduitConfig;
  private plugins: PluginManager;
  private router: ConduitRouter;
  private healthMonitor: HealthMonitor;

  constructor(config: ConduitConfig) {
    this.config = config;
    this.plugins = new PluginManager();
    this.router = new ConduitRouter(config);
    this.healthMonitor = new HealthMonitor();
  }

  async start(): Promise<void> {
    // Load plugins
    await this.plugins.loadPlugins(this.config.plugins || []);
    
    // Initialize router
    await this.router.initialize();
    
    // Start server
    this.server = await this.createServer();
    
    // Start health monitoring
    this.healthMonitor.start();
    
    console.log(`Conduit server started on port ${this.config.server?.port || 3456}`);
  }

  async stop(): Promise<void> {
    this.healthMonitor.stop();
    
    if (this.server) {
      await this.server.close();
    }
    
    await this.plugins.cleanup();
  }

  async route(context: RoutingContext): Promise<RoutingDecision> {
    return this.router.route(context);
  }

  getHealth(): HealthStatus {
    return this.healthMonitor.getStatus();
  }

  async updateConfig(config: Partial<ConduitConfig>): Promise<void> {
    this.config = { ...this.config, ...config };
    await this.router.updateConfig(this.config);
  }
}
```

#### 2.2 Standalone Router Class

**Create Standalone Router**
```typescript
// src/router/ConduitRouter.ts
export class ConduitRouter {
  private config: ConduitConfig;
  private tokenCounter: TokenCounter;
  private plugins: PluginManager;
  private providers: Map<string, ProviderConfig>;

  constructor(config: ConduitConfig) {
    this.config = config;
    this.tokenCounter = new TokenCounter();
    this.plugins = new PluginManager();
    this.providers = new Map();
  }

  async initialize(): Promise<void> {
    // Initialize providers
    this.config.Providers?.forEach(provider => {
      this.providers.set(provider.name, provider);
    });

    // Load plugins
    await this.plugins.loadPlugins(this.config.plugins || []);
  }

  async route(context: RoutingContext): Promise<RoutingDecision> {
    // Apply pre-routing plugins
    const processedContext = await this.plugins.beforeRouting(context);
    
    // Count tokens
    const tokenCount = await this.tokenCounter.count(processedContext.message);
    
    // Make routing decision
    const decision = await this.makeRoutingDecision({
      ...processedContext,
      tokenCount
    });
    
    // Apply post-routing plugins
    const finalDecision = await this.plugins.afterRouting(decision);
    
    return finalDecision;
  }

  private async makeRoutingDecision(context: RoutingContext): Promise<RoutingDecision> {
    // Custom routing rules
    const customDecision = await this.evaluateCustomRules(context);
    if (customDecision) {
      return customDecision;
    }

    // Token-based routing
    if (context.tokenCount > 60000) {
      return this.createDecision(this.config.Router?.longContext || 'default', 'token-based');
    }

    // Default routing
    return this.createDecision(this.config.Router?.default || 'default', 'default');
  }

  async updateConfig(config: ConduitConfig): Promise<void> {
    this.config = config;
    await this.initialize();
  }
}
```

#### 2.3 Enhanced Plugin System

**Improved Plugin Interface**
```typescript
// src/plugins/PluginInterface.ts
export interface ConduitPlugin {
  name: string;
  version: string;
  description?: string;
  
  // Lifecycle hooks
  onLoad?(): Promise<void>;
  onUnload?(): Promise<void>;
  
  // Routing hooks
  beforeRouting?(context: RoutingContext): Promise<RoutingContext>;
  afterRouting?(decision: RoutingDecision): Promise<RoutingDecision>;
  
  // Custom routing
  customRouter?(context: RoutingContext): Promise<RoutingDecision | null>;
  
  // Health monitoring
  getHealth?(): HealthStatus;
  
  // Configuration
  validateConfig?(config: any): boolean;
}

export interface PluginConfig {
  name: string;
  enabled: boolean;
  config?: any;
  source?: 'file' | 'npm' | 'inline';
  path?: string;
  module?: ConduitPlugin;
}
```

**Enhanced Plugin Manager**
```typescript
// src/plugins/PluginManager.ts
export class PluginManager {
  private plugins: Map<string, ConduitPlugin> = new Map();
  private pluginConfigs: Map<string, PluginConfig> = new Map();

  async loadPlugins(configs: PluginConfig[]): Promise<void> {
    for (const config of configs) {
      if (!config.enabled) continue;
      
      try {
        const plugin = await this.loadPlugin(config);
        this.plugins.set(config.name, plugin);
        this.pluginConfigs.set(config.name, config);
        
        if (plugin.onLoad) {
          await plugin.onLoad();
        }
      } catch (error) {
        console.error(`Failed to load plugin ${config.name}:`, error);
      }
    }
  }

  private async loadPlugin(config: PluginConfig): Promise<ConduitPlugin> {
    switch (config.source) {
      case 'inline':
        return config.module!;
      case 'npm':
        return require(config.name);
      case 'file':
      default:
        return require(config.path || config.name);
    }
  }

  async beforeRouting(context: RoutingContext): Promise<RoutingContext> {
    let processedContext = context;
    
    for (const plugin of this.plugins.values()) {
      if (plugin.beforeRouting) {
        processedContext = await plugin.beforeRouting(processedContext);
      }
    }
    
    return processedContext;
  }

  async afterRouting(decision: RoutingDecision): Promise<RoutingDecision> {
    let processedDecision = decision;
    
    for (const plugin of this.plugins.values()) {
      if (plugin.afterRouting) {
        processedDecision = await plugin.afterRouting(processedDecision);
      }
    }
    
    return processedDecision;
  }

  async customRouting(context: RoutingContext): Promise<RoutingDecision | null> {
    for (const plugin of this.plugins.values()) {
      if (plugin.customRouter) {
        const decision = await plugin.customRouter(context);
        if (decision) {
          return decision;
        }
      }
    }
    
    return null;
  }

  async cleanup(): Promise<void> {
    for (const plugin of this.plugins.values()) {
      if (plugin.onUnload) {
        await plugin.onUnload();
      }
    }
    
    this.plugins.clear();
    this.pluginConfigs.clear();
  }
}
```

### Phase 3: Synapse Integration Layer (Week 3)

#### 3.1 Synapse-Specific Features

**Create Synapse Integration Module**
```typescript
// src/integrations/synapse.ts
export class SynapseIntegration {
  private router: ConduitRouter;
  private usageTracker: UsageTracker;
  private contextExtractor: ContextExtractor;

  constructor(config: ConduitConfig) {
    this.router = new ConduitRouter(config);
    this.usageTracker = new UsageTracker();
    this.contextExtractor = new ContextExtractor();
  }

  async routeClaudeRequest(
    args: string[],
    env: NodeJS.ProcessEnv
  ): Promise<{
    model: string;
    provider: string;
    metadata: any;
  }> {
    // Extract Synapse context
    const context = this.contextExtractor.extractFromEnvironment(env);
    
    // Create routing context
    const routingContext: RoutingContext = {
      tokenCount: await this.estimateTokenCount(args),
      model: this.extractModelFromArgs(args),
      message: this.extractMessageFromArgs(args),
      metadata: {
        projectId: context.projectId,
        agentId: context.agentId,
        agentType: context.agentType,
        ...context.projectModelConfig,
        ...context.agentModelConfig
      }
    };

    // Route request
    const decision = await this.router.route(routingContext);
    
    // Track usage
    this.usageTracker.trackRequest(routingContext, decision);
    
    return {
      model: decision.model,
      provider: decision.provider,
      metadata: decision.metadata
    };
  }

  async trackUsage(usage: UsageData): Promise<void> {
    await this.usageTracker.track(usage);
  }

  getUsageStats(projectId: string): Promise<UsageStats> {
    return this.usageTracker.getStats(projectId);
  }
}
```

#### 3.2 Configuration Management

**Enhanced Configuration System**
```typescript
// src/config/ConfigManager.ts
export class ConfigManager {
  private config: ConduitConfig;
  private presetManager: ConfigPresetManager;

  constructor(initialConfig?: ConduitConfig) {
    this.config = initialConfig || {};
    this.presetManager = new ConfigPresetManager();
  }

  async loadPreset(presetName: string): Promise<void> {
    const preset = await this.presetManager.loadBuiltinPreset(presetName);
    this.config = this.mergeConfigs(this.config, preset);
  }

  updateConfig(updates: Partial<ConduitConfig>): void {
    this.config = this.mergeConfigs(this.config, updates);
  }

  getConfig(): ConduitConfig {
    return { ...this.config };
  }

  validateConfig(): ValidationResult {
    return this.presetManager.validateConfig(this.config);
  }

  private mergeConfigs(base: ConduitConfig, updates: Partial<ConduitConfig>): ConduitConfig {
    // Deep merge logic
    return {
      ...base,
      ...updates,
      Router: {
        ...base.Router,
        ...updates.Router
      },
      server: {
        ...base.server,
        ...updates.server
      }
    };
  }
}
```

### Phase 4: Usage and Monitoring (Week 4)

#### 4.1 Enhanced Usage Tracking

**Usage Tracking System**
```typescript
// src/monitoring/UsageTracker.ts
export class UsageTracker {
  private storage: UsageStorage;
  private metrics: MetricsCollector;

  constructor(config: UsageConfig) {
    this.storage = new UsageStorage(config.storage);
    this.metrics = new MetricsCollector(config.metrics);
  }

  async trackRequest(context: RoutingContext, decision: RoutingDecision): Promise<void> {
    const usage: UsageRecord = {
      id: generateId(),
      timestamp: new Date(),
      projectId: context.metadata.projectId,
      agentId: context.metadata.agentId,
      model: decision.model,
      provider: decision.provider,
      tokenCount: context.tokenCount,
      cost: this.calculateCost(context.tokenCount, decision.model),
      routingReason: decision.reason,
      metadata: {
        ...context.metadata,
        ...decision.metadata
      }
    };

    await this.storage.store(usage);
    this.metrics.record(usage);
  }

  async getStats(projectId: string, timeRange?: TimeRange): Promise<UsageStats> {
    return this.storage.getStats(projectId, timeRange);
  }

  async getCostBreakdown(projectId: string): Promise<CostBreakdown> {
    return this.storage.getCostBreakdown(projectId);
  }
}
```

#### 4.2 Health Monitoring

**Health Monitoring System**
```typescript
// src/monitoring/HealthMonitor.ts
export class HealthMonitor {
  private checks: Map<string, HealthCheck> = new Map();
  private status: HealthStatus = {
    healthy: true,
    checks: {},
    uptime: 0,
    startTime: new Date()
  };

  start(): void {
    // Register default health checks
    this.registerCheck('server', () => this.checkServer());
    this.registerCheck('router', () => this.checkRouter());
    this.registerCheck('plugins', () => this.checkPlugins());
    this.registerCheck('storage', () => this.checkStorage());

    // Start monitoring
    setInterval(() => this.runHealthChecks(), 30000);
  }

  registerCheck(name: string, check: HealthCheck): void {
    this.checks.set(name, check);
  }

  async runHealthChecks(): Promise<void> {
    for (const [name, check] of this.checks) {
      try {
        const result = await check();
        this.status.checks[name] = result;
      } catch (error) {
        this.status.checks[name] = {
          healthy: false,
          error: error.message,
          timestamp: new Date()
        };
      }
    }

    this.status.healthy = Object.values(this.status.checks)
      .every(check => check.healthy);
    this.status.uptime = Date.now() - this.status.startTime.getTime();
  }

  getStatus(): HealthStatus {
    return { ...this.status };
  }
}
```

### Phase 5: Documentation and Testing (Week 5)

#### 5.1 Comprehensive Documentation

**Create Usage Examples**
```typescript
// docs/examples/synapse-integration.ts
import { ConduitServer, SynapseIntegration } from '@tehreet/conduit';

// Example 1: Basic Synapse integration
const conduit = new ConduitServer({
  preset: 'synapse',
  server: {
    port: 3456,
    healthCheck: true
  },
  Router: {
    default: 'anthropic,claude-3-5-sonnet-20241022',
    longContext: 'anthropic,claude-3-5-sonnet-20241022',
    background: 'anthropic,claude-3-5-haiku-20241022'
  }
});

await conduit.start();

// Example 2: Programmatic routing
const integration = new SynapseIntegration(config);
const routingDecision = await integration.routeClaudeRequest(
  ['--message', 'Hello world'],
  process.env
);

// Example 3: Usage tracking
const stats = await integration.getUsageStats('project-123');
console.log('Total cost:', stats.totalCost);
console.log('Request count:', stats.requestCount);
```

#### 5.2 Testing Strategy

**Unit Tests**
```typescript
// tests/unit/ConduitServer.test.ts
describe('ConduitServer', () => {
  let server: ConduitServer;
  
  beforeEach(() => {
    server = new ConduitServer(testConfig);
  });

  it('should start and stop cleanly', async () => {
    await server.start();
    expect(server.getHealth().healthy).toBe(true);
    
    await server.stop();
    expect(server.getHealth().healthy).toBe(false);
  });

  it('should route requests correctly', async () => {
    const decision = await server.route({
      tokenCount: 1000,
      model: 'claude-3-5-sonnet-20241022',
      message: 'Test message',
      metadata: {}
    });
    
    expect(decision.model).toBe('claude-3-5-sonnet-20241022');
    expect(decision.provider).toBe('anthropic');
  });
});
```

## Migration Strategy

### Backward Compatibility

1. **CLI Interface**: Maintain existing CLI commands and functionality
2. **Configuration**: Support existing configuration file formats
3. **Plugin System**: Ensure existing plugins continue to work
4. **API Endpoints**: Maintain existing HTTP API endpoints

### Migration Path for Existing Users

1. **Phase 1**: Update package, CLI continues to work as before
2. **Phase 2**: Introduce library usage alongside CLI
3. **Phase 3**: Migrate to library usage gradually
4. **Phase 4**: Deprecate CLI-only features (with long deprecation period)

## Success Metrics

1. **Performance**: Library usage should be 10x faster than CLI spawning
2. **Memory Usage**: Memory footprint should be 50% lower than CLI spawning
3. **Error Handling**: 99.9% of errors should be properly captured and handled
4. **Compatibility**: 100% backward compatibility with existing CLI usage
5. **Adoption**: 80% of new integrations should use library interface

## Timeline

- **Week 1**: Core library infrastructure and build system
- **Week 2**: Enhanced API design and programmatic interfaces
- **Week 3**: Synapse integration layer and configuration management
- **Week 4**: Usage tracking and monitoring systems
- **Week 5**: Documentation, testing, and migration tools

## Risk Mitigation

1. **Breaking Changes**: Comprehensive testing and gradual rollout
2. **Performance**: Benchmarking and optimization at each phase
3. **Compatibility**: Extensive testing with existing configurations
4. **Security**: Security audit of all new APIs and interfaces
5. **Documentation**: Clear migration guides and examples

This plan transforms Conduit into a scalable, library-first solution while maintaining full backward compatibility and providing a smooth migration path for existing users.