@coworker-agency/rag/README.md

Retrieval Augmented Generation (RAG) library for document indexing, vector storage, and AI-powered question answering

# @coworker-agency/rag

A Retrieval Augmented Generation (RAG) library for document indexing, vector storage, and AI-powered question answering. This package provides a complete solution for building RAG systems with Supabase and OpenAI.

## Features

- **Document Indexing**: Process and index documents from Supabase storage into vector databases
- **Context-Aware Vectors**: Generate context-aware vector embeddings for improved retrieval quality
- **Memory-Efficient Processing**: Handles large documents without running into memory limitations
- **Query Classification**: Classify user queries into predefined categories
- **Question Answering**: Generate accurate answers from your document collection
- **FAQ Generation**: Automatically generate FAQs from your documents

## Installation

```bash
npm install @coworker-agency/rag
```

## Quick Start

```javascript
import { init } from '@coworker-agency/rag';

// Initialize the library with your configuration
const rag = init({
  supabaseUrl: 'https://your-project.supabase.co',
  supabaseKey: 'your-supabase-key',
  supabaseSecretKey: 'your-supabase-service-role-key',
  supabaseBucket: 'documents',
  tableName: 'vector_documents',
  openaiApiKey: 'your-openai-api-key',
  openaiLlmModel: 'gpt-4o',
  openaiEmbeddingModel: 'text-embedding-3-small'
});

// Process documents from Supabase storage
const result = await rag.processDocuments();
console.log(`Processed ${result.processed} documents`);

// Search for relevant information
const faqs = await rag.searchFaq('What is context-aware retrieval?', 3);
console.log(faqs.questions);

// Get specific answer to a question
const answer = await rag.getAnswer('How do vector embeddings work?');
console.log(answer);
```

## API Reference

### init(options)

Initializes the RAG library with configuration options.

```javascript
const rag = init({
  supabaseUrl,         // Supabase project URL (required)
  supabaseKey,         // Supabase public key (optional if supabaseSecretKey is provided)
  supabaseSecretKey,   // Supabase service role key (required)
  supabaseBucket,      // Supabase storage bucket name (required)
  tableName,           // Vector store table name (default: 'vector_documents')
  openaiApiKey,        // OpenAI API key (required)
  openaiLlmModel,      // OpenAI LLM model (default: 'gpt-4o')
  openaiEmbeddingModel // OpenAI Embedding model (default: 'text-embedding-3-small')
});
```

### processDocuments(options)

Processes documents from Supabase storage and indexes them into the vector store.

```javascript
const result = await rag.processDocuments({
  batchSize: 10,        // Number of files to process in parallel
  maxFileSize: 20971520, // 20MB max file size
  skipExisting: true     // Skip already indexed files
});
```

### searchFaq(query, limit)

Search vector store and generate FAQ responses based on the query.

```javascript
const faqs = await rag.searchFaq('How do I implement RAG?', 5);
// Returns { questions: [{ question, answer, links }] }
```

### getAnswer(question)

Get a specific answer to a question using the vector store.

```javascript
const result = await rag.getAnswer('What are embeddings?');
// Returns { question, answer: { text, links } }
```

### queryClassifier(query, classifications, options)

Classify a query into one of the provided categories.

```javascript
const category = await rag.queryClassifier(
  'How do I reset my password?',
  [
    { category: 'account', description: 'Account management questions', temperature: 0.8 },
    { category: 'technical', description: 'Technical or implementation questions', temperature: 0.7 },
    { category: 'billing', description: 'Billing or payment questions', temperature: 0.6 }
  ],
  { memory: ['Previous user message'] }
);
```

### getContextAwareVectors(documentContent, options)

Generate context-aware vectors from document content.

```javascript
const vectors = await rag.getContextAwareVectors(documentText, {
  chunkSize: 1000,
  chunkOverlap: 200
});
// Returns array of { context, content, vector }
```

## Memory-Efficient Processing

This package includes special handling for large documents to avoid memory issues:

- Automatic detection of large files (>100K chars)
- Split processing into manageable chunks
- Memory-efficient batch processing
- Optional LLM refinement skipping for very large files
- Document summarization and context extraction

## Supabase Setup

This package requires a properly configured Supabase instance with:

1. A vector-enabled table (e.g., `vector_documents`)
2. Storage bucket for document files
3. Optional RPC function for vector search

Example SQL for creating the vector table:

```sql
-- Enable pgvector extension
CREATE EXTENSION IF NOT EXISTS vector;

-- Create table for document vectors
CREATE TABLE vector_documents (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  content TEXT NOT NULL,
  metadata JSONB,
  embedding VECTOR(1536),  -- For text-embedding-3-small (1536 dimensions)
  created_at TIMESTAMPTZ DEFAULT NOW()
);

-- Create function for similarity search
CREATE OR REPLACE FUNCTION match_documents(
  query_embedding VECTOR(1536),
  match_threshold FLOAT,
  match_count INT
) RETURNS TABLE (
  id UUID,
  content TEXT,
  metadata JSONB,
  similarity FLOAT
) LANGUAGE SQL STABLE AS $$
  SELECT
    id,
    content,
    metadata,
    1 - (embedding <=> query_embedding) AS similarity
  FROM vector_documents
  WHERE 1 - (embedding <=> query_embedding) > match_threshold
  ORDER BY similarity DESC
  LIMIT match_count;
$$;
```

## License

MIT