pipe-protocol/README.md

A protocol for large scale Interplanetary Intertool Agent Context

# Pipe Protocol

A protocol for wrapping LLM tools with IPFS storage and encryption capabilities.

## Problem Statement

LLM applications often face challenges with:
- Context window limitations when handling large amounts of data
- Inefficient data sharing between tools and agents
- Redundant storage of identical data
- Lack of persistent storage for tool inputs/outputs

Pipe Protocol solves these challenges by providing content-addressable storage through IPFS, allowing efficient data sharing via CIDs instead of raw data.

## Features

- 🔄 Store tool inputs and outputs on IPFS with content-addressable storage
- 📊 Automatic schema generation for stored data
- 🔒 Configurable storage scopes (private/public/machine/user)
- 🎯 Token counting and limiting for LLM context management
- 📌 Configurable pinning options for data persistence
- 🔐 Built-in encryption support with flexible access policies
- 🪝 Pre and post-store hooks for custom data processing
- 🛠️ Tool wrapping with enhanced capabilities
- 📦 Record and Bundle support for schema and data pairing

## Installation

```bash
npm install pipe-protocol
```

## Quick Start

### Basic Usage

Here's a simple example showing how to wrap an OpenAI tool with Pipe Protocol:

```typescript
import { Pipe } from 'pipe-protocol';
import OpenAI from 'openai';
import type { ChatCompletionTool } from 'openai/resources/chat/completions';

// Initialize OpenAI and Pipe
const openai = new OpenAI();

// Option 1: Use default in-memory IPFS node (recommended for getting started)
const pipe = new Pipe();

// Option 2: Configure custom IPFS node
// const pipe = new Pipe({
//   ipfs: {
//     endpoint: 'http://localhost:5001',
//     timeout: 5000,
//     scope: 'private',
//     pin: true
//   }
// });

// Define a tool that generates Fibonacci numbers
const fibonacciTool = {
  name: 'generate_fibonacci',
  description: 'Generate a Fibonacci sequence of given length',
  parameters: {
    type: 'object',
    properties: {
      n: {
        type: 'number',
        description: 'Number of Fibonacci numbers to generate (must be 2 or greater)'
      }
    },
    required: ['n']
  },
  call: async (args: { n: number }) => {
    const sequence = [0, 1];
    while (sequence.length < args.n) {
      sequence.push(sequence[sequence.length - 1] + sequence[sequence.length - 2]);
    }
    return sequence;
  }
};

// Wrap the tool with Pipe
const wrappedTools = pipe.wrap([fibonacciTool]);

// Format for OpenAI
const openAITools: ChatCompletionTool[] = wrappedTools.map(tool => ({
  type: 'function',
  function: {
    name: tool.name,
    description: tool.description,
    parameters: tool.parameters
  }
}));

// Use with OpenAI
const completion = await openai.chat.completions.create({
  model: 'gpt-4',
  messages: [{ role: 'user', content: 'Generate the first 10 Fibonacci numbers' }],
  tools: openAITools
});

// Execute tool call if made
const toolCalls = completion.choices[0].message.tool_calls;
if (toolCalls) {
  const toolCall = toolCalls[0];
  const args = JSON.parse(toolCall.function.arguments);
  const result = await wrappedTools[0].call(args);
  
  // Access stored data and schema
  console.log('Result CID:', result.cid);
  console.log('Schema CID:', result.schemaCid);
  
  // Retrieve stored data
  const storedData = await pipe.retrieve(result.cid);
  console.log('Retrieved data:', storedData);
}
```

### Tool Interface

Each tool should implement this interface:

```typescript
interface Tool {
  name: string;
  description: string;
  parameters: {
    type: 'object';
    properties: Record<string, any>;
    required?: string[];
  };
  returns?: {
    type: string;
    description?: string;
  };
  call: (...args: any[]) => any;
}
```

### Adding Hooks

You can add pre and post-processing hooks for custom data handling:

```typescript
const pipe = new Pipe({
  hooks: [
    {
      name: 'validator',
      type: 'beforeStore',
      handler: async (data) => {
        // Validate data before storage
        if (!data) throw new Error('Invalid data');
        return data;
      }
    },
    {
      name: 'logger',
      type: 'afterStore',
      handler: async (data) => {
        // Log after storage, includes CID and metadata
        console.log('Stored with CID:', data.cid);
        console.log('Metadata:', data.metadata);
        console.log('Schema CID:', data.schemaCid);
        return data;
      }
    }
  ]
});
```

### Configuration Options

```typescript
const pipe = new Pipe({
  ipfs: {
    endpoint: 'http://localhost:5001', // IPFS node endpoint
    timeout: 5000, // Request timeout
    scope: 'private', // Default scope for data
    pin: true // Auto-pin data to IPFS
  },
  defaults: {
    maxTokens: 1000, // Maximum number of tokens for tool outputs
    storeResult: true, // Automatically store tool results
    generateSchema: true, // Auto-generate schemas
    scope: 'private',
    pin: true
  },
  hooks: [], // Array of pre/post store hooks
  storage: 'memory', // Storage type ('memory' or 'persistent')
  storageConfig: {
    directory: '/tmp/ipfs' // Directory for persistent storage
  },
  enableNetworking: false, // Enable/disable networking
  listenAddresses: [], // Libp2p listen addresses
  bootstrapList: [] // Libp2p bootstrap list
});
```

### Storage Scopes

Pipe Protocol supports four storage scopes:

- **private**: Data accessible only within the local context
- **public**: Data that can be shared and replicated across nodes
- **machine**: Data specific to the current machine/environment
- **user**: Data specific to the current user

## API Documentation

For detailed API documentation, visit our [documentation site](https://pipe-protocol.github.io/docs).

## Contributing

Please read [CONTRIBUTING.md](CONTRIBUTING.md) for details on our code of conduct and the process for submitting pull requests.

## License

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.