contaigents/CONVERSATION_MESSAGE_ARCHITECTURE.md

Modular AI Content Ecosystem with Audio Generation

# Conversation Message Architecture Design

## Overview

This document defines the hybrid message handling system that supports both modern conversation APIs (OpenAI, Anthropic, DeepSeek, OpenRouter) and simple prompt-based providers (HuggingFace, Ollama, Replicate) while maintaining backward compatibility.

## Current Problems

### 1. Single Prompt Concatenation Issue
```typescript
// ❌ Current problematic approach
context += `System: ${session.systemPrompt}\n\n`;
context += fileTreeContext;
context += toolInstructions;
context += conversationHistory;
context += `Human: ${currentMessage.content}\n\nAssistant:`;

// Sent as single user message - loses conversation structure
messages: [{ role: "user", content: context }]
```

### 2. Tool Response Integration Issues
- XML tool responses are injected as user messages
- Breaks conversation flow for conversation-capable providers
- Tool responses appear in UI when they should be internal

### 3. Provider Capability Mismatch
- All providers treated the same despite different capabilities
- No fallback strategy for simple prompt providers
- Google Gemini has custom format requirements

## New Message Architecture

### 1. Enhanced Message Interface

```typescript
export interface ConversationMessage {
  id: string;
  role: 'system' | 'user' | 'assistant';
  content: string;
  timestamp: number;
  metadata?: {
    isToolResponse?: boolean;
    toolCallId?: string;
    provider?: string;
    internal?: boolean; // Hide from UI
  };
}

export interface ChatSession {
  id: string;
  messages: ConversationMessage[];
  systemPrompt?: string;
  createdAt: number;
  updatedAt: number;
  providerCapabilities?: ProviderCapabilities;
}
```

### 2. Provider Capability Detection

```typescript
export interface ProviderCapabilities {
  supportsConversation: boolean;
  supportsSystemMessages: boolean;
  supportsToolCalls: boolean;
  customFormat?: 'gemini' | 'standard';
  maxContextLength?: number;
}

export const PROVIDER_CAPABILITIES: Record<string, ProviderCapabilities> = {
  OpenAI: {
    supportsConversation: true,
    supportsSystemMessages: true,
    supportsToolCalls: true,
    customFormat: 'standard'
  },
  Anthropic: {
    supportsConversation: true,
    supportsSystemMessages: true,
    supportsToolCalls: true,
    customFormat: 'standard'
  },
  DeepSeek: {
    supportsConversation: true,
    supportsSystemMessages: true,
    supportsToolCalls: true,
    customFormat: 'standard'
  },
  OpenRouter: {
    supportsConversation: true,
    supportsSystemMessages: true,
    supportsToolCalls: true,
    customFormat: 'standard'
  },
  Google: {
    supportsConversation: true,
    supportsSystemMessages: false, // Uses different format
    supportsToolCalls: true,
    customFormat: 'gemini'
  },
  HuggingFace: {
    supportsConversation: false,
    supportsSystemMessages: false,
    supportsToolCalls: false,
    customFormat: 'standard'
  },
  Ollama: {
    supportsConversation: false,
    supportsSystemMessages: false,
    supportsToolCalls: false,
    customFormat: 'standard'
  },
  Replicate: {
    supportsConversation: false,
    supportsSystemMessages: false,
    supportsToolCalls: false,
    customFormat: 'standard'
  }
};
```

### 3. System Prompt Composition Strategy

```typescript
export interface SystemPromptComponents {
  base: string;           // Core agent instructions
  fileTree: string;       // Current project structure
  tools: string;          // Available tools and usage
  context: string;        // Additional context
}

export class SystemPromptComposer {
  static compose(components: SystemPromptComponents): string {
    return [
      components.base,
      components.fileTree ? `\n\n## Current Project Structure\n\n${components.fileTree}` : '',
      components.tools ? `\n\n${components.tools}` : '',
      components.context ? `\n\n## Additional Context\n\n${components.context}` : ''
    ].filter(Boolean).join('');
  }
}
```

### 4. Message Format Strategies

#### A. Conversation-Capable Providers (OpenAI, Anthropic, DeepSeek, OpenRouter)

```typescript
// ✅ Proper conversation format
const messages = [
  {
    role: "system",
    content: SystemPromptComposer.compose({
      base: session.systemPrompt,
      fileTree: await toolManager.generateFileTreeContext(),
      tools: toolManager.generateToolInstructions(),
      context: ""
    })
  },
  { role: "user", content: "First user message" },
  { role: "assistant", content: "First assistant response" },
  { role: "user", content: "Tool response XML (internal)", metadata: { internal: true } },
  { role: "assistant", content: "Assistant response with tool results" },
  { role: "user", content: "Current user message" }
];
```

#### B. Simple Prompt Providers (HuggingFace, Ollama, Replicate)

```typescript
// ✅ Fallback to concatenated prompt format
const prompt = [
  `System: ${systemPrompt}`,
  `\n\n## Conversation History\n`,
  ...conversationHistory.map(msg => `${msg.role}: ${msg.content}`),
  `\n\nHuman: ${currentMessage}\n\nAssistant:`
].join('');
```

#### C. Google Gemini Custom Format

```typescript
// ✅ Gemini-specific format
const parts = [
  { text: systemPrompt },
  ...conversationHistory.map(msg => ({
    role: msg.role === 'assistant' ? 'model' : 'user',
    parts: [{ text: msg.content }]
  }))
];
```

### 5. Tool Response Integration Strategy

#### For Conversation-Capable Providers:
```typescript
// Tool responses as internal user messages
const toolResponseMessage: ConversationMessage = {
  id: generateId(),
  role: 'user',
  content: xmlToolResponses,
  timestamp: Date.now(),
  metadata: {
    isToolResponse: true,
    internal: true, // Hidden from UI
    toolCallId: toolCall.id
  }
};
```

#### For Simple Prompt Providers:
```typescript
// Tool responses integrated into prompt context
const promptWithToolResults = [
  basePrompt,
  '\n\n## Tool Execution Results\n',
  xmlToolResponses,
  '\n\nBased on the tool results above, please provide your response:'
].join('');
```

## Implementation Plan

### Phase 1: Interface Updates
1. Update `ConversationMessage` interface with metadata
2. Add `ProviderCapabilities` interface and constants
3. Create `SystemPromptComposer` utility class

### Phase 2: Provider Detection
1. Add capability detection to each provider
2. Update `LLMInterface` with capability methods
3. Implement provider-specific message formatting

### Phase 3: Message Handling Refactor
1. Replace `buildConversationContext()` with `buildMessageArray()`
2. Implement hybrid message formatting logic
3. Update tool response integration

### Phase 4: UI Integration
1. Filter internal messages from UI display
2. Update message rendering logic
3. Add provider capability indicators

## Benefits

1. **Proper Conversation Structure**: Maintains role separation for conversation-capable providers
2. **Backward Compatibility**: Fallback strategy for simple prompt providers
3. **Tool Integration**: Clean separation of tool responses from user conversation
4. **Provider Optimization**: Each provider uses its optimal format
5. **UI Clarity**: Internal messages hidden from user interface
6. **Extensibility**: Easy to add new providers with different capabilities

## Migration Strategy

1. **Gradual Rollout**: Implement provider by provider
2. **Feature Flags**: Toggle between old and new systems
3. **Validation**: Extensive testing with each provider type
4. **Monitoring**: Track conversation quality improvements

## Detailed Implementation Specifications

### 1. Updated LLM Interface

```typescript
export interface LLMInterface {
  configure(config: Record<string, any>): void;
  executePrompt(prompt: string, options?: PromptOptions): Promise<PromptResponse>;
  executeConversation?(messages: ConversationMessage[], options?: PromptOptions): Promise<PromptResponse>;
  executePrompts(prompts: string[], options?: PromptOptions): Promise<PromptResponse[]>;
  imagine?(prompt: string, options?: ImageOptions): Promise<ImageResponse>;
  getConfig(): Record<string, any>;
  isConfigured(): Promise<boolean>;
  getCapabilities(): ProviderCapabilities;
  supportsConversation(): boolean;
}
```

### 2. Message Builder Classes

```typescript
export class ConversationMessageBuilder {
  static buildForProvider(
    session: ChatSession,
    currentMessage: string,
    provider: string,
    toolManager: ToolManager
  ): Promise<ConversationMessage[] | string> {
    const capabilities = PROVIDER_CAPABILITIES[provider];

    if (capabilities.supportsConversation) {
      return this.buildConversationArray(session, currentMessage, capabilities, toolManager);
    } else {
      return this.buildPromptString(session, currentMessage, toolManager);
    }
  }

  private static async buildConversationArray(
    session: ChatSession,
    currentMessage: string,
    capabilities: ProviderCapabilities,
    toolManager: ToolManager
  ): Promise<ConversationMessage[]> {
    const messages: ConversationMessage[] = [];

    // Add system message if supported
    if (capabilities.supportsSystemMessages) {
      const systemContent = await this.buildSystemPrompt(session, toolManager);
      messages.push({
        id: generateId(),
        role: 'system',
        content: systemContent,
        timestamp: Date.now()
      });
    }

    // Add conversation history (excluding internal tool responses for UI)
    const conversationMessages = session.messages
      .filter(msg => !msg.metadata?.internal)
      .slice(-10); // Keep last 10 for context

    messages.push(...conversationMessages);

    // Add current user message
    messages.push({
      id: generateId(),
      role: 'user',
      content: currentMessage,
      timestamp: Date.now()
    });

    return messages;
  }

  private static async buildPromptString(
    session: ChatSession,
    currentMessage: string,
    toolManager: ToolManager
  ): Promise<string> {
    const systemPrompt = await this.buildSystemPrompt(session, toolManager);
    const conversationHistory = session.messages
      .filter(msg => !msg.metadata?.internal)
      .slice(-10)
      .map(msg => `${msg.role}: ${msg.content}`)
      .join('\n\n');

    return [
      `System: ${systemPrompt}`,
      conversationHistory ? `\n\nConversation History:\n${conversationHistory}` : '',
      `\n\nHuman: ${currentMessage}\n\nAssistant:`
    ].filter(Boolean).join('');
  }

  private static async buildSystemPrompt(
    session: ChatSession,
    toolManager: ToolManager
  ): Promise<string> {
    const fileTreeContext = await toolManager.generateFileTreeContext();
    const toolInstructions = toolManager.generateToolInstructions();

    return SystemPromptComposer.compose({
      base: session.systemPrompt || 'You are a helpful AI assistant.',
      fileTree: fileTreeContext,
      tools: toolInstructions,
      context: ''
    });
  }
}
```

### 3. Provider-Specific Implementations

```typescript
// Example: Enhanced OpenAI Provider
export class OpenAIProvider extends BaseLLM {
  getCapabilities(): ProviderCapabilities {
    return PROVIDER_CAPABILITIES.OpenAI;
  }

  supportsConversation(): boolean {
    return true;
  }

  async executeConversation(
    messages: ConversationMessage[],
    options: PromptOptions = {}
  ): Promise<PromptResponse> {
    if (!(await this.isConfigured())) {
      throw new Error("OpenAI provider is not configured");
    }

    const openAIMessages = messages.map(msg => ({
      role: msg.role,
      content: msg.content
    }));

    const response = await fetch("https://api.openai.com/v1/chat/completions", {
      method: "POST",
      headers: {
        "Content-Type": "application/json",
        Authorization: `Bearer ${this.config.apiKey}`,
      },
      body: JSON.stringify({
        model: this.config.model,
        messages: openAIMessages,
        temperature: options.temperature ?? 0.7,
        max_tokens: options.maxTokens,
      }),
    });

    if (!response.ok) {
      const error = await response.json();
      throw new Error(`OpenAI API error: ${error.error?.message || "Unknown error"}`);
    }

    const data = await response.json();
    return {
      content: data.choices[0].message.content,
      raw: data,
    };
  }
}
```

### 4. Tool Response Handling

```typescript
export class ToolResponseHandler {
  static async integrateToolResponses(
    session: ChatSession,
    toolResponses: ToolResponse[],
    provider: string
  ): Promise<void> {
    const capabilities = PROVIDER_CAPABILITIES[provider];
    const xmlResponses = toolResponses
      .map(tr => session.toolManager.generateToolResponse(tr))
      .join('\n\n');

    if (capabilities.supportsConversation) {
      // Add as internal user message for conversation providers
      const toolResponseMessage: ConversationMessage = {
        id: generateId(),
        role: 'user',
        content: xmlResponses,
        timestamp: Date.now(),
        metadata: {
          isToolResponse: true,
          internal: true,
          toolCallId: toolResponses[0]?.id
        }
      };
      session.messages.push(toolResponseMessage);
    } else {
      // For simple prompt providers, tool responses will be integrated
      // into the prompt string during buildPromptString()
      const toolContextMessage: ConversationMessage = {
        id: generateId(),
        role: 'user',
        content: `Tool Results:\n${xmlResponses}`,
        timestamp: Date.now(),
        metadata: {
          isToolResponse: true,
          internal: false, // Visible in simple prompt providers
          toolCallId: toolResponses[0]?.id
        }
      };
      session.messages.push(toolContextMessage);
    }
  }
}
```

## Testing Strategy

### 1. Provider Capability Tests
- Verify each provider reports correct capabilities
- Test conversation vs prompt format selection
- Validate system prompt composition

### 2. Message Format Tests
- Test conversation array building for each provider type
- Verify prompt string fallback for simple providers
- Test Google Gemini custom format handling

### 3. Tool Integration Tests
- Verify tool responses are properly integrated
- Test internal message filtering for UI
- Validate XML tool response formatting

### 4. Migration Tests
- Test backward compatibility with existing sessions
- Verify gradual provider migration
- Test feature flag toggling