@tehreet/conduit/docs/test-first-phases.md

LLM API gateway with intelligent routing, robust process management, and health monitoring

# Conduit Synapse Integration - Test Plan for Phases 1-4

This document provides a comprehensive test plan to verify that all implemented Synapse integration features (Sections 1-4) work correctly.

## Test Environment Setup

### Prerequisites
- Node.js >= 16.0.0
- npm/pnpm package manager
- Git repository with `conduit-synapse-integration-features` branch

### Initial Setup
```bash
# Switch to feature branch
git checkout conduit-synapse-integration-features

# Install dependencies and build
npm install
npm run build

# Verify CLI is working
node dist/cli.js --help
```

---

## Phase 1: Context-Aware Routing via Environment Variables

### Test 1.1: Basic Context Extraction
**Purpose:** Verify Synapse environment variables are properly extracted

**Setup:**
```bash
# Set Synapse environment variables
export SYNAPSE_PROJECT_ID="test-project-123"
export SYNAPSE_AGENT_ID="test-agent-456"
export SYNAPSE_AGENT_TYPE="coding-assistant"
export SYNAPSE_PROJECT_MODEL_CONFIG='{"enabled":true,"defaultModel":"anthropic/claude-3-5-haiku-20241022"}'
export SYNAPSE_AGENT_MODEL_CONFIG='{"useProjectDefaults":false,"overrideModel":"anthropic/claude-3-opus-20240229"}'
```

**Test Commands:**
```bash
# Test routing with agent override (should use opus)
node dist/cli.js test-routing --message "Hello world" --verbose

# Test routing with project config only
unset SYNAPSE_AGENT_MODEL_CONFIG
node dist/cli.js test-routing --message "Hello world" --verbose

# Test routing without Synapse context
unset SYNAPSE_PROJECT_MODEL_CONFIG SYNAPSE_PROJECT_ID SYNAPSE_AGENT_ID
node dist/cli.js test-routing --message "Hello world" --verbose
```

**Expected Results:**
1. With agent override: `Selected Model: claude-3-opus-20240229`, `Routing Source: synapse-context`
2. With project config: `Selected Model: claude-3-5-haiku-20241022`, `Routing Source: synapse-context`
3. Without Synapse: Falls back to standard routing

### Test 1.2: Model ID Mapping
**Purpose:** Verify Synapse model IDs are correctly mapped to Claude CLI format

**Test Commands:**
```bash
export SYNAPSE_PROJECT_MODEL_CONFIG='{"enabled":true,"defaultModel":"anthropic/claude-3-5-sonnet-20241022"}'
node dist/cli.js test-routing --message "Test mapping"
```

**Expected Results:**
- Synapse format `anthropic/claude-3-5-sonnet-20241022` → Claude format `claude-3-5-sonnet-20241022`

### Test 1.3: Error Handling
**Purpose:** Verify malformed JSON configurations don't break routing

**Test Commands:**
```bash
# Test with malformed JSON
export SYNAPSE_PROJECT_MODEL_CONFIG='{"enabled":true,"defaultModel"'
node dist/cli.js test-routing --message "Test error handling"

# Test with invalid JSON structure
export SYNAPSE_PROJECT_MODEL_CONFIG='{"invalid":"structure"}'
node dist/cli.js test-routing --message "Test error handling"
```

**Expected Results:**
- Should fall back to standard routing without errors
- Logs should show JSON parsing errors

---

## Phase 2: Plugin Architecture for Custom Routing

### Test 2.1: Plugin System Initialization
**Purpose:** Verify plugin system loads without errors

**Test Commands:**
```bash
# Start Conduit to test plugin loading
node dist/cli.js start &
sleep 3
node dist/cli.js status
node dist/cli.js stop
```

**Expected Results:**
- Service starts without plugin-related errors
- Logs show plugin directory scanning

### Test 2.2: Plugin Interface Validation
**Purpose:** Create and test a simple custom plugin

**Setup:**
```bash
# Create plugins directory
mkdir -p ~/.conduit/plugins

# Create a test plugin
cat > ~/.conduit/plugins/test-plugin.js << 'EOF'
class TestPlugin {
  constructor() {
    this.name = 'test-plugin';
    this.version = '1.0.0';
  }
  
  async customRouter(context) {
    console.log('TestPlugin: Custom router called');
    if (context.request.body.messages && 
        context.request.body.messages[0]?.content?.includes('test-plugin')) {
      return 'claude-3-5-haiku-20241022';
    }
    return null;
  }
  
  async onModelSelected(model, context) {
    console.log(`TestPlugin: Model ${model} selected`);
  }
}

module.exports = TestPlugin;
EOF
```

**Test Commands:**
```bash
# Test plugin loading and execution
node dist/cli.js start &
sleep 3

# Test with plugin trigger message
node dist/cli.js test-routing --message "Please use test-plugin for this" --verbose

# Test without plugin trigger
node dist/cli.js test-routing --message "Regular message" --verbose

node dist/cli.js stop
```

**Expected Results:**
1. Plugin loads successfully on startup
2. Message with "test-plugin" → `Selected Model: claude-3-5-haiku-20241022`, `Routing Source: plugins`
3. Regular message → Falls back to other routing methods

### Test 2.3: Plugin Error Isolation
**Purpose:** Verify plugin errors don't break core routing

**Setup:**
```bash
# Create a broken plugin
cat > ~/.conduit/plugins/broken-plugin.js << 'EOF'
class BrokenPlugin {
  constructor() {
    this.name = 'broken-plugin';
    this.version = '1.0.0';
  }
  
  async customRouter(context) {
    throw new Error('Plugin intentionally broken');
  }
}

module.exports = BrokenPlugin;
EOF
```

**Test Commands:**
```bash
node dist/cli.js start &
sleep 3
node dist/cli.js test-routing --message "Test with broken plugin" --verbose
node dist/cli.js stop
```

**Expected Results:**
- Service starts despite broken plugin
- Routing continues to work with fallback methods
- Error logs show plugin failure but system remains stable

### Cleanup
```bash
# Remove test plugins
rm -rf ~/.conduit/plugins/test-plugin.js ~/.conduit/plugins/broken-plugin.js
```

---

## Phase 3: Extended CLI Options for Synapse

### Test 3.1: Configuration Validation
**Purpose:** Verify config validation works for various scenarios

**Setup:**
```bash
# Create test configurations
cat > valid-config.json << 'EOF'
{
  "name": "test-config",
  "providers": [
    {
      "name": "test-provider",
      "type": "claude-cli",
      "models": [
        {"id": "claude-3-5-sonnet-20241022"}
      ]
    }
  ]
}
EOF

cat > invalid-config.json << 'EOF'
{
  "name": "test-config",
  "providers": [
    {
      "name": "test-provider"
    }
  ]
}
EOF
```

**Test Commands:**
```bash
# Test valid configuration
node dist/cli.js validate-config valid-config.json

# Test invalid configuration
node dist/cli.js validate-config invalid-config.json

# Test non-existent file
node dist/cli.js validate-config non-existent.json

# Test malformed JSON
echo '{"invalid":json}' > malformed.json
node dist/cli.js validate-config malformed.json
```

**Expected Results:**
1. Valid config: `✅ Configuration is valid` + available models list
2. Invalid config: `❌ Configuration errors` + specific error messages
3. Non-existent: `❌ Configuration file not found`
4. Malformed: JSON parsing error

### Test 3.2: Routing Testing Command
**Purpose:** Verify routing test scenarios work correctly

**Test Commands:**
```bash
# Test with project configuration
node dist/cli.js test-routing \
  --project-config '{"enabled":true,"defaultModel":"claude-3-5-haiku-20241022"}' \
  --message "Hello world"

# Test with agent configuration
node dist/cli.js test-routing \
  --agent-config '{"useProjectDefaults":false,"overrideModel":"claude-3-opus-20240229"}' \
  --message "Hello world"

# Test with token count
node dist/cli.js test-routing --token-count 70000

# Test with thinking flag
node dist/cli.js test-routing --thinking --message "Complex reasoning task"

# Test verbose mode
node dist/cli.js test-routing --message "Test" --verbose
```

**Expected Results:**
- Each test should show appropriate routing decisions
- Verbose mode should show plugin results and fallback chain
- Token count should trigger appropriate model selection

### Test 3.3: Token Counting Accuracy
**Purpose:** Verify token counting is working correctly

**Test Commands:**
```bash
# Test short message
node dist/cli.js test-routing --message "Hi"

# Test long message
node dist/cli.js test-routing --message "$(printf 'word %.0s' {1..1000})"

# Test with manual token count override
node dist/cli.js test-routing --token-count 1000 --message "Short message"
```

**Expected Results:**
- Short message: Low token count (< 10)
- Long message: High token count (> 1000)
- Override: Uses specified token count regardless of message

### Cleanup
```bash
rm -f valid-config.json invalid-config.json malformed.json
```

---

## Phase 4: Configuration File Support for Synapse

### Test 4.1: Preset Management Commands
**Purpose:** Verify all preset CLI commands work correctly

**Test Commands:**
```bash
# List available presets
node dist/cli.js list-presets

# Show built-in preset details
node dist/cli.js show-preset config/synapse-preset.json

# Validate preset file
node dist/cli.js validate-preset config/synapse-preset.json

# Test preset routing rules
node dist/cli.js test-preset config/synapse-preset.json --token-count 2000
node dist/cli.js test-preset config/synapse-preset.json --token-count 70000

# Test with non-existent preset
node dist/cli.js show-preset non-existent-preset
```

**Expected Results:**
1. List: Shows available presets with descriptions
2. Show: Displays providers, models, routing rules, and Synapse integration
3. Validate: `✅ Preset configuration is valid` + model list
4. Test (2000): Triggers small-context rule → haiku
5. Test (70000): Triggers large-context rule → sonnet
6. Non-existent: Error message with file not found

### Test 4.2: Preset Integration with Routing
**Purpose:** Verify presets are properly integrated into the routing system

**Setup:**
```bash
# Create a test config that uses a preset
cat > test-preset-config.json << 'EOF'
{
  "preset": "config/synapse-preset.json",
  "LOG": true
}
EOF
```

**Test Commands:**
```bash
# Test preset routing integration
export CONDUIT_CONFIG_PATH="./test-preset-config.json"

# Test small context (should use haiku via preset rule)
node dist/cli.js test-routing --message "Short message"

# Test large context (should use sonnet via preset rule)  
node dist/cli.js test-routing --token-count 70000

# Test thinking mode (should use sonnet via preset rule)
node dist/cli.js test-routing --thinking --message "Complex task"

unset CONDUIT_CONFIG_PATH
```

**Expected Results:**
- Small context: `claude-3-5-haiku-20241022` via preset rule
- Large context: `claude-3-5-sonnet-20241022` via preset rule
- Thinking mode: `claude-3-5-sonnet-20241022` via preset rule

### Test 4.3: Preset Rule Evaluation
**Purpose:** Verify routing rule conditions are evaluated correctly

**Setup:**
```bash
# Create custom preset with specific rules
cat > custom-preset.json << 'EOF'
{
  "name": "test-preset",
  "providers": [
    {
      "name": "test-provider",
      "type": "claude-cli",
      "models": [
        {"id": "claude-3-5-haiku-20241022"},
        {"id": "claude-3-5-sonnet-20241022"}
      ]
    }
  ],
  "defaultRouting": {
    "rules": [
      {
        "name": "tiny-context",
        "condition": "tokenCount < 100",
        "model": "claude-3-5-haiku-20241022"
      },
      {
        "name": "large-context",
        "condition": "tokenCount >= 1000",
        "model": "claude-3-5-sonnet-20241022"
      }
    ],
    "fallback": "claude-3-5-sonnet-20241022"
  }
}
EOF
```

**Test Commands:**
```bash
# Test rule conditions
node dist/cli.js test-preset custom-preset.json --token-count 50    # Should match tiny-context
node dist/cli.js test-preset custom-preset.json --token-count 500   # Should use fallback
node dist/cli.js test-preset custom-preset.json --token-count 1500  # Should match large-context

# Validate custom preset
node dist/cli.js validate-preset custom-preset.json
```

**Expected Results:**
- 50 tokens: `✅ tiny-context` rule matched
- 500 tokens: Uses fallback (no rule matched)
- 1500 tokens: `✅ large-context` rule matched

### Cleanup
```bash
rm -f test-preset-config.json custom-preset.json
```

---

## Integration Tests

### Test I.1: End-to-End Routing Priority
**Purpose:** Verify the complete routing priority chain works correctly

**Setup:**
```bash
# Set up all routing methods
export SYNAPSE_PROJECT_ID="test-project"
export SYNAPSE_PROJECT_MODEL_CONFIG='{"enabled":true,"defaultModel":"anthropic/claude-3-5-haiku-20241022"}'

# Create test plugin
mkdir -p ~/.conduit/plugins
cat > ~/.conduit/plugins/priority-test.js << 'EOF'
class PriorityTestPlugin {
  constructor() {
    this.name = 'priority-test';
    this.version = '1.0.0';
  }
  
  async customRouter(context) {
    if (context.request.body.messages && 
        context.request.body.messages[0]?.content?.includes('plugin-priority')) {
      return 'claude-3-opus-20240229';
    }
    return null;
  }
}

module.exports = PriorityTestPlugin;
EOF

# Create config with preset
cat > priority-test-config.json << 'EOF'
{
  "preset": "config/synapse-preset.json",
  "Router": {
    "default": "claude-3-5-sonnet-20241022"
  }
}
EOF
```

**Test Commands:**
```bash
export CONDUIT_CONFIG_PATH="./priority-test-config.json"

# Test plugin priority (highest)
node dist/cli.js test-routing --message "Use plugin-priority routing" --verbose

# Test preset rules priority (second)
node dist/cli.js test-routing --token-count 2000 --verbose

# Test Synapse context priority (third)
node dist/cli.js test-routing --message "Regular message" --verbose

# Test legacy routing (lowest)
unset SYNAPSE_PROJECT_MODEL_CONFIG SYNAPSE_PROJECT_ID
node dist/cli.js test-routing --token-count 70000 --verbose

unset CONDUIT_CONFIG_PATH
```

**Expected Results:**
1. Plugin message: `claude-3-opus-20240229` via plugin
2. Small token count: `claude-3-5-haiku-20241022` via preset rule
3. Regular message: `claude-3-5-haiku-20241022` via Synapse context
4. Large token count (no Synapse): `claude-3-5-sonnet-20241022` via legacy routing

### Test I.2: Service Integration
**Purpose:** Verify features work with running Conduit service

**Test Commands:**
```bash
# Start service with all features enabled
export SYNAPSE_PROJECT_ID="test-integration"
export SYNAPSE_PROJECT_MODEL_CONFIG='{"enabled":true,"defaultModel":"anthropic/claude-3-5-haiku-20241022"}'

node dist/cli.js start &
sleep 5

# Verify service is running
node dist/cli.js status

# Test that routing works through the service
# (This would require actual Claude CLI integration to fully test)

node dist/cli.js stop
```

**Expected Results:**
- Service starts successfully with all features loaded
- Status shows service running
- No errors in service logs related to new features

### Cleanup
```bash
rm -rf ~/.conduit/plugins/priority-test.js
rm -f priority-test-config.json
unset SYNAPSE_PROJECT_ID SYNAPSE_PROJECT_MODEL_CONFIG SYNAPSE_AGENT_ID SYNAPSE_AGENT_MODEL_CONFIG
```

---

## Error Handling and Edge Cases

### Test E.1: Malformed Environment Variables
```bash
# Test with malformed JSON in environment variables
export SYNAPSE_PROJECT_MODEL_CONFIG='invalid json'
node dist/cli.js test-routing --message "Test malformed env"

# Test with empty environment variables
export SYNAPSE_PROJECT_MODEL_CONFIG=''
export SYNAPSE_PROJECT_ID=''
node dist/cli.js test-routing --message "Test empty env"
```

### Test E.2: Missing Files and Permissions
```bash
# Test with non-existent preset file
node dist/cli.js show-preset /non/existent/preset.json

# Test with permission denied (if applicable)
# sudo chmod 000 config/synapse-preset.json
# node dist/cli.js validate-preset config/synapse-preset.json
# sudo chmod 644 config/synapse-preset.json
```

### Test E.3: Resource Limits
```bash
# Test with extremely large token counts
node dist/cli.js test-routing --token-count 999999999

# Test with very long messages
node dist/cli.js test-routing --message "$(printf 'word %.0s' {1..10000})"
```

---

## Success Criteria

✅ **Phase 1 Complete** if:
- Synapse environment variables are correctly extracted and used for routing
- Model ID mapping works correctly
- Error handling gracefully falls back to standard routing

✅ **Phase 2 Complete** if:
- Plugin system loads and executes custom routing logic
- Plugin errors are isolated and don't break core functionality
- Routing priority respects plugin decisions

✅ **Phase 3 Complete** if:
- Configuration validation catches errors and reports them clearly
- Routing testing provides accurate simulation of routing decisions
- Token counting and CLI argument parsing work correctly

✅ **Phase 4 Complete** if:
- Configuration presets load and apply correctly
- Preset routing rules are evaluated accurately
- CLI preset commands provide useful functionality

✅ **Integration Complete** if:
- Routing priority follows: Plugins → Preset rules → Synapse context → Legacy routing
- All features work together without conflicts
- Service starts and runs stably with all features enabled

---

## Troubleshooting

### Common Issues

1. **"Plugin directory not found"**
   - Create `~/.conduit/plugins` directory
   - Ensure proper permissions

2. **"Preset file not found"**
   - Use full paths for custom presets
   - Check file exists and is readable

3. **"Environment variables not working"**
   - Verify exports are in current shell
   - Check JSON formatting in environment variables

4. **"Service won't start"**
   - Check port 3456 is not in use
   - Verify build completed successfully

### Debug Commands
```bash
# Enable verbose logging
export LOG=true

# Check service logs
node dist/cli.js start
# (logs will show routing decisions and plugin loading)

# Test individual components
node dist/cli.js test-routing --verbose
node dist/cli.js validate-preset config/synapse-preset.json
```

---

## Notes

- All tests should be run from the project root directory
- Some tests may require actual Claude CLI installation for full integration testing
- Environment variables should be unset between test sections to avoid conflicts
- Build the project (`npm run build`) after any code changes during testing