contaigents/SHARED_MEMORY_SPEC.md

Modular AI Content Ecosystem with Audio Generation

# Shared Memory Specification for Chat Service

## Overview

The shared memory system enables tools and agents to communicate efficiently without consuming context window space. Tools can store intermediate results, large data sets, or complex state information in memory, and reference it using memory IDs instead of including full content in responses.

## Core Concepts

### Memory Storage
- **Location**: `.memory/` directory (reserved namespace)
- **Format**: Any file type - text, binary, audio, images, etc.
- **Scope**: Session-scoped (isolated per chat session)
- **Persistence**: Memory persists for the duration of the chat session

## Architecture

### Memory Structure
```
.memory/
└── {session_id}/
    ├── mem_1234567890.txt
    ├── mem_1234567890.json
    ├── mem_1234567890.wav
    ├── mem_1234567890.png
    └── mem_1234567890_metadata.json
```

### Memory Metadata (Optional)
```json
{
  "id": "mem_1234567890",
  "title": "Audio Recording",
  "file_type": "audio/wav",
  "created_at": "2025-01-15T10:30:00Z"
}
```

## XML Tag Extensions

### Memory Storage in Tool Calls
Tools can specify memory storage using the `memory_store` attribute:

```xml
<tool_call name="web_fetch" id="1" memory_store="true">
  <url>https://api.github.com/repos/user/repo/issues</url>
  <memory_title>GitHub Issues API Response</memory_title>
</tool_call>
```

### Memory Reference in Tool Responses
Tools can respond with memory references instead of full content:

```xml
<tool_response id="1">
  <tool_call name="command_execution">
    <command>curl -s "https://api.github.com/repos/user/repo/issues" > issues.json</command>
  </tool_call>
  <result>
    <memory_ref id="mem_1234567890">
      <title>GitHub Issues API Response</title>
      <summary>Retrieved 150 GitHub issues with full metadata</summary>
      <file_type>application/json</file_type>
      <size>85KB</size>
    </memory_ref>
  </result>
</tool_response>
```

## Memory Management Tool

### New Tool: `memory_manager`
A dedicated tool for memory operations:

```xml
<!-- Store text content -->
<tool_call name="memory_manager" id="1">
  <operation>store</operation>
  <title>Analysis Results</title>
  <content><![CDATA[Large analysis content here...]]></content>
</tool_call>

<!-- Store binary file -->
<tool_call name="memory_manager" id="2">
  <operation>store</operation>
  <title>Generated Audio</title>
  <file_path>output.wav</file_path>
</tool_call>

<!-- Read from memory -->
<tool_call name="memory_manager" id="3">
  <operation>read</operation>
  <memory_id>mem_1234567890</memory_id>
</tool_call>

<!-- List memory -->
<tool_call name="memory_manager" id="4">
  <operation>list</operation>
</tool_call>
```

## System Prompt Changes

### Enhanced System Prompt
Add memory awareness to the system prompt:

```markdown
## Memory Management

You have access to a shared memory system that allows you to:
- Store large content without consuming context window space
- Reference previously stored information efficiently
- Maintain state across tool calls
- Avoid repeating expensive operations

### When to Use Memory:
1. **API Responses**: Store results from web_fetch, API calls, external services
2. **Command Outputs**: Store results from expensive or long-running commands
3. **Processed Data**: Store analysis results, transformations, and summaries
4. **Large Content**: Store content that would consume significant context space

### When NOT to Use Memory:
1. **File Contents**: Don't store simple file reads - just read the file again when needed
2. **Simple Data**: Don't store small content (<1KB) that fits easily in context
3. **One-time Use**: Don't store data that won't be referenced again
4. **Real-time Data**: Don't store information that changes frequently

### Complex Processing Pattern:
For large data processing, you can combine tools:
1. Create processing script/commands with write_file
2. Execute with command_execution + memory_store
3. Store results in memory for later reference
4. Clean up temporary files

**Available Processing Options:**
- **Node.js scripts**: Always available (this is a Node.js package)
- **Shell commands**: awk, sed, grep, jq, curl, etc.
- **Other languages**: Only if installed on the system (check first)

### Memory Operations:
- Use `memory_store="true"` attribute in tool calls to auto-store results
- Reference stored content using `<memory_ref>` tags
- Use `memory_manager` tool for explicit memory operations

### Response Guidelines:
- **Interim messages** (with tool calls): Keep to 1-2 sentences maximum
- **Final messages** (no tool calls): Be descriptive and comprehensive
- Focus on what you're doing, not how you're doing it
- Save detailed explanations for the final response

**Example of Good Interim Messages:**
✅ "I'll fetch the API data and analyze the trends."
✅ "Processing the log files to extract error patterns."
✅ "Let me analyze the project structure first."

**Example of Bad Interim Messages:**
❌ "I'm going to create a Node.js script that will read multiple JSON files, process each record, categorize the data, calculate statistics, and then output the results in a structured format that I'll store in memory for later reference."
```

## Implementation Flow

### 1. Tool Execution with Memory Storage
```typescript
// When tool call includes memory_store attribute
if (toolCall.memory_store) {
  const memoryId = await memoryManager.store({
    type: toolCall.memory_store,
    content: toolResult.data,
    metadata: {
      source_tool: toolCall.tool_name,
      title: toolCall.parameters.memory_title || 'Untitled'
    }
  });
  
  // Replace large content with memory reference
  toolResult.memory_ref = memoryId;
  toolResult.data = undefined; // Remove large content
}
```

### 2. Memory Reference Resolution
```typescript
// When agent needs to access memory
const memoryContent = await memoryManager.read(memoryId);
// Include in context only when specifically requested
```

### 3. Session Cleanup
```typescript
// Clean up memory when session ends
await memoryManager.cleanupSession(sessionId);
```

## When to Use Memory

### ✅ Good Use Cases:
- **API Responses**: Web fetches, database queries, external service calls
- **Command Output**: Long-running command results, system information
- **Processed Data**: Analysis results, transformed datasets, computed summaries
- **Expensive Operations**: Operations that are slow, rate-limited, or costly to repeat
- **Tool State**: Complex state that needs to persist across multiple tool calls
- **Large Datasets**: Content >1KB that would consume significant context space

### ❌ Avoid Memory For:
- **File Reading**: Files can be re-read easily (unless processing is expensive)
- **Simple Calculations**: Basic operations that are fast to recompute
- **Small Content**: Content <1KB that fits comfortably in context
- **One-time Use**: Data that won't be referenced again
- **Real-time Data**: Information that changes frequently and needs fresh reads

## Complex Processing Pattern

For complex data processing, agents can combine file creation + command execution + memory storage:

### Pattern: Create → Execute → Store
1. **Create processing script**: Use `write_file` to create temporary processing code
2. **Execute processing**: Use `command_execution` to run the script
3. **Store results**: Use `memory_store` to save processed output
4. **Clean up**: Optionally remove temporary files

### Example A: Node.js Processing (Always Available)
```
User: "Analyze all the JSON data files and give me a summary"

Agent: "I'll process the JSON files and generate a comprehensive analysis."

<tool_call name="write_file" id="1">
  <operation_type>create</operation_type>
  <file_path>process_data.js</file_path>
  <content><![CDATA[
const fs = require('fs');
const files = ['data1.json', 'data2.json', 'data3.json'];
const results = { totalRecords: 0, categories: {} };

files.forEach(file => {
  const data = JSON.parse(fs.readFileSync(file, 'utf8'));
  results.totalRecords += data.length;
  data.forEach(record => {
    results.categories[record.category] = (results.categories[record.category] || 0) + 1;
  });
});

console.log(JSON.stringify(results, null, 2));
]]></content>
</tool_call>

<tool_call name="command_execution" id="2" memory_store="data">
  <command>node process_data.js && rm process_data.js</command>
  <memory_title>Multi-file Analysis Results</memory_title>
</tool_call>

[Final response]
Agent: "Analysis complete! I processed 3 JSON files containing 1,247 total records. Here's the breakdown by category:
- Products: 456 records (37%)
- Services: 321 records (26%)
- Users: 289 records (23%)
- Orders: 181 records (14%)

The data appears well-distributed across categories with products being the dominant record type."
```

### Example B: Shell Command Processing
```xml
<!-- Process log files with shell commands and store results -->
<tool_call name="command_execution" id="1" memory_store="data">
  <command>cat access.log | awk '{print $1}' | sort | uniq -c | sort -nr | head -20 | jq -R -s 'split("\n")[:-1] | map(split(" ") | {count: .[0], ip: .[1]})'</command>
  <memory_title>Top 20 IP Addresses from Access Log</memory_title>
</tool_call>
```

### Example C: Complex Shell Pipeline
```xml
<!-- Multi-step shell processing -->
<tool_call name="command_execution" id="1" memory_store="data">
  <command>find . -name "*.js" -exec wc -l {} + | sort -nr | head -10 | awk '{total+=$1; files[NR]=$2" ("$1" lines)"} END {print "Total lines:", total; print "Top files:"; for(i=1;i<=NR;i++) print files[i]}' | jq -R -s 'split("\n") | {summary: .[0], files: .[2:]}'</command>
  <memory_title>JavaScript Files Analysis</memory_title>
</tool_call>
```

## Benefits

1. **Context Window Efficiency**: Large content doesn't consume context space
2. **Tool Communication**: Tools can share data without context pollution
3. **State Persistence**: Maintain complex state across multiple tool calls
4. **Performance**: Faster responses with smaller context payloads
5. **Cost Efficiency**: Avoid repeating expensive API calls or operations
6. **Scalability**: Handle large documents and datasets efficiently

## Usage Examples

### Example 1: Web API Data Processing
```
User: "Analyze the current VSCode GitHub issues and tell me about bug trends"

Agent: "I'll fetch the latest VSCode issues and analyze the bug patterns."

<tool_call name="web_fetch" id="1" memory_store="data">
  <url>https://api.github.com/repos/microsoft/vscode/issues?state=open&per_page=100</url>
  <memory_title>VSCode Open Issues</memory_title>
</tool_call>

[Tool responds with memory reference]

Agent: "Now let me analyze the bug-related issues."

<tool_call name="memory_manager" id="2">
  <operation>read</operation>
  <memory_id>mem_data_abc123</memory_id>
  <partial>true</partial>
  <filter>labels.name=bug</filter>
</tool_call>

[Final comprehensive response]
Agent: "Based on the analysis of 100 current VSCode issues, here are the key bug trends:

- 23% of open issues are labeled as bugs (23 out of 100)
- Most common bug categories: editor performance (8 issues), extension compatibility (6 issues)
- 15 bugs are marked as high priority
- Average bug age is 12 days, indicating good triage response time
- Top affected areas: TypeScript support, debugging, and terminal functionality

The VSCode team appears to be actively managing bugs with consistent labeling and quick initial responses."
```

### Example 2: API Data Fetching
```xml
<!-- Fetch API data and store in memory -->
<tool_call name="command_execution" id="1" memory_store="data">
  <command>curl -s "https://api.github.com/repos/microsoft/vscode/issues?state=open&per_page=100" | jq '{total: length, labels: [.[] | .labels[].name] | group_by(.) | map({label: .[0], count: length})}'</command>
  <memory_title>VSCode Issues Analysis</memory_title>
</tool_call>
```

### Example 3: File System Analysis
```xml
<!-- Analyze project structure and store results -->
<tool_call name="command_execution" id="1" memory_store="data">
  <command>find . -type f -name "*.js" -o -name "*.ts" -o -name "*.json" | head -100 | xargs wc -l | tail -1 | awk '{print "Total lines: " $1}' && echo "File breakdown:" && find . -name "*.js" | wc -l | awk '{print "JavaScript files: " $1}' && find . -name "*.ts" | wc -l | awk '{print "TypeScript files: " $1}'</command>
  <memory_title>Project Code Analysis</memory_title>
</tool_call>
```

### Example 4: Using Other Languages (If Available)
```xml
<!-- Check if Python is available first -->
<tool_call name="command_execution" id="1">
  <command>which python3 && echo "Python available" || echo "Python not available"</command>
</tool_call>

<!-- If Python is available, create and run script -->
<tool_call name="write_file" id="2">
  <operation_type>create</operation_type>
  <file_path>temp_analysis.py</file_path>
  <content><![CDATA[
import json
import sys

# Simple data processing
data = {"processed": True, "timestamp": "2025-01-15"}
print(json.dumps(data))
]]></content>
</tool_call>

<tool_call name="command_execution" id="3" memory_store="data">
  <command>python3 temp_analysis.py && rm temp_analysis.py</command>
  <memory_title>Python Processing Results</memory_title>
</tool_call>
```

## Migration Path

1. **Phase 1**: Implement memory manager tool and basic storage
2. **Phase 2**: Add memory_store attribute support to existing tools
3. **Phase 3**: Update system prompts with memory guidelines
4. **Phase 4**: Optimize tool responses to use memory references
5. **Phase 5**: Add advanced features (partial reads, memory search)

This specification provides a foundation for efficient tool communication while maintaining the simplicity of the current XML-based tool architecture.