vector-embedding-mcp-server/README.md

MCP Server for AI Embedding and RAG functionality

# Vector Embedding MCP Server

A Model Context Protocol (MCP) server that provides AI embedding and RAG (Retrieval-Augmented Generation) functionality. This server converts your existing AI embedding application into an MCP-compatible service.

## Features

- **Document Ingestion**: Ingest documents and create vector embeddings
- **RAG Chat**: Chat with AI using business rules context from ingested documents
- **Vector Search**: Search for similar documents using vector similarity
- **Embedding Generation**: Generate embeddings for text using HuggingFace models
- **Project Management**: Organize documents by project ID
- **Multiple LLM Support**: Works with Ollama (local) and HuggingFace models

## Prerequisites

- Node.js 18+ 
- PostgreSQL with pgvector extension
- Ollama (for local LLM) or HuggingFace API key
- Python 3.8+ (for some dependencies)

## Installation

1. **Clone and install dependencies:**
   ```bash
   npm install
   ```

2. **Set up PostgreSQL with pgvector:**
   ```bash
   # Install pgvector extension
   sudo -u postgres psql -c "CREATE EXTENSION vector;"
   
   # Run the setup script
   psql -U your_username -d your_database -f setup.sql
   ```

3. **Configure environment:**
   ```bash
   cp env.example .env
   # Edit .env with your configuration
   ```

4. **Install Ollama (optional, for local LLM):**
   ```bash
   # Install Ollama
   curl -fsSL https://ollama.ai/install.sh | sh
   
   # Pull a model
   ollama pull mistral
   ollama pull qwen2.5-coder:7b
   ollama pull nomic-embed-text
   ```

## Configuration

Edit your `.env` file with the following variables:

```env
# Database Configuration
DATABASE_URL=postgresql://username:password@localhost:5432/embedding_db

# HuggingFace API Configuration
HUGGINGFACE_API_KEY=your_huggingface_api_key_here

# OpenAI API Configuration (optional)
OPENAI_API_KEY=your_openai_api_key_here

# Ollama Configuration (for local LLM)
OLLAMA_BASE_URL=http://localhost:11434

# Server Configuration
PORT=3000
NODE_ENV=development
```

## Usage

### Starting the MCP Server

```bash
npm start
```

The server will run on stdio and can be connected to MCP clients.

### Available Tools

#### 1. `ingest_document`
Ingest a document into the vector database.

**Parameters:**
- `filePath` (string): Path to the document file
- `projectId` (string): Project ID to associate with

**Example:**
```json
{
  "name": "ingest_document",
  "arguments": {
    "filePath": "./docs/business-rules.mdc",
    "projectId": "project1"
  }
}
```

#### 2. `chat_with_rules`
Chat with AI using business rules context.

**Parameters:**
- `message` (string): User's question or message
- `projectId` (string): Project ID to search for context

**Example:**
```json
{
  "name": "chat_with_rules",
  "arguments": {
    "message": "What are the business rules for the resource module?",
    "projectId": "project1"
  }
}
```

#### 3. `generate_embedding`
Generate embeddings for text.

**Parameters:**
- `text` (string): Text to generate embeddings for

**Example:**
```json
{
  "name": "generate_embedding",
  "arguments": {
    "text": "This is sample text for embedding"
  }
}
```

#### 4. `vector_search`
Search for similar documents using vector similarity.

**Parameters:**
- `query` (string): Search query text
- `projectId` (string): Project ID to search within
- `topK` (number, optional): Number of results (default: 5)

**Example:**
```json
{
  "name": "vector_search",
  "arguments": {
    "query": "business rules",
    "projectId": "project1",
    "topK": 3
  }
}
```

#### 5. `list_projects`
List all projects with their document counts.

**Example:**
```json
{
  "name": "list_projects",
  "arguments": {}
}
```

#### 6. `get_project_documents`
Get all documents for a specific project.

**Parameters:**
- `projectId` (string): Project ID to get documents for

**Example:**
```json
{
  "name": "get_project_documents",
  "arguments": {
    "projectId": "project1"
  }
}
```

## Integration with MCP Clients

### Claude Desktop

Add to your Claude Desktop configuration:

```json
{
  "mcpServers": {
    "ai-embedding": {
      "command": "node",
      "args": ["/path/to/your/ai-embedding/server.js"],
      "env": {
        "DATABASE_URL": "postgresql://username:password@localhost:5432/embedding_db",
        "HUGGINGFACE_API_KEY": "your_api_key"
      }
    }
  }
}
```

### Other MCP Clients

The server follows the MCP protocol and can be integrated with any MCP-compatible client by running:

```bash
node server.js
```

## Architecture

The application follows a clean MVC (Model-View-Controller) architecture:

- **`server.js`**: Express server entry point
- **`mcp-server.js`**: MCP server entry point
- **`src/`**: Main source code directory
  - **`controllers/`**: Request handlers (ChatController, DocumentController, EmbeddingController)
  - **`services/`**: Business logic (RAGService, IngestionService, EmbeddingService, DocumentService)
  - **`models/`**: Data access layer (Document, DocumentMetadata, ProjectConfig)
  - **`routes/`**: API route definitions
  - **`middleware/`**: Express middleware (error handling, validation)
  - **`utils/`**: Utility functions (file reading, text splitting, vector formatting)
  - **`mcp/`**: MCP server implementation
  - **`config/`**: Configuration (database, environment, logging)
- **`scripts/`**: Utility scripts (migrate, ingest-all)

For detailed architecture documentation, see [ARCHITECTURE.md](./ARCHITECTURE.md).

## Supported File Types

- Text files (`.txt`, `.md`, `.mdc`)
- PDF files (`.pdf`)
- Word documents (`.docx`)

## Database Schema

The server uses a PostgreSQL database with the following schema:

```sql
CREATE TABLE documents (
    id SERIAL PRIMARY KEY,
    project_id VARCHAR(255) NOT NULL,
    file_name VARCHAR(500) NOT NULL,
    content TEXT NOT NULL,
    embedding vector(384),
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);
```

## Troubleshooting

### Common Issues

1. **Database Connection Error**: Ensure PostgreSQL is running and the connection string is correct
2. **pgvector Extension Missing**: Install the pgvector extension in your PostgreSQL database
3. **Ollama Connection Error**: Ensure Ollama is running and the models are pulled
4. **HuggingFace API Error**: Check your API key and rate limits

### Debug Mode

Run with debug logging:

```bash
DEBUG=* npm start
```

## Development

### Adding New Tools

To add new tools to the MCP server:

1. Add the tool definition to the `ListToolsRequestSchema` handler
2. Add the tool implementation to the `CallToolRequestSchema` handler
3. Update this README with the new tool documentation

### Testing

Test individual components:

```bash
# Run all tests
npm test

# Test MCP functionality
npm run test:mcp

# Test Cursor integration
npm run test:cursor

# Test using services directly (Node.js REPL)
node
> const IngestionService = require('./src/services/IngestionService');
> const service = new IngestionService();
> await service.ingestDocument('./docs/business-rules.mdc', 'test');
```

## License

ISC License - see LICENSE file for details.

## Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Test thoroughly
5. Submit a pull request

## Support

For issues and questions:
1. Check the troubleshooting section
2. Review the logs for error messages
3. Ensure all dependencies are properly installed
4. Verify your environment configuration