dbx-mcp-server/CLAUDE.md

A Model Context Protocol server for Dropbox integration with AI-powered PDF analysis, multi-directory indexing, and parallel processing

# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## Common Development Commands

### Building and Running
- `npm run build` - Compile TypeScript to build directory and set executable permissions
- `npm run build:skipcheck` - Alternative build using build.sh script
- `npm run watch` - Watch TypeScript files for changes and recompile
- `npm start` - Run the built server from build/src/index.js
- `npm run setup` - Initialize Dropbox authentication setup

### Testing
- `npm test` - Run all tests using Jest
- `npm run test:watch` - Run tests in watch mode
- `npm run test:coverage` - Run tests with coverage reporting
- `npm run test:integration` - Run integration tests specifically
- Run specific test file: `npm test -- tests/dropbox/search-delete.test.ts`
- Run tests matching pattern: `npm test -- -t "should search for files"`

### Development Tools
- `npm run inspector` - Launch MCP inspector for debugging

## Architecture Overview

This is a Model Context Protocol (MCP) server that provides Dropbox integration. The architecture follows a modular design:

### Core Components

**Server Entry Point (`src/index.ts`)**
- Main executable that starts the DbxServer instance
- Uses StdioServerTransport for MCP communication

**DbxServer (`src/dbx-server.ts`)**
- Main MCP server class extending @modelcontextprotocol/sdk Server
- Handles all MCP protocol interactions (tools, resources, prompts)
- Integrates authentication, tool execution, and resource management

**Authentication System (`src/auth.ts`)**
- OAuth 2.0 with PKCE flow for Dropbox authentication
- Token encryption/decryption using security-utils
- Automatic token refresh with retry logic
- Stores encrypted tokens in .tokens.json

**Configuration (`src/config.js`)**
- Environment variable validation and Winston logging setup
- Creates logs directory and configures file-based logging
- Handles encrypted configuration values

### Tool Architecture

**Tool Definitions (`src/tool-definitions.ts`)**
- Defines all available MCP tools and their schemas
- Covers file operations, metadata, search, and account tools

**Dropbox API Layer (`src/dbx-api.ts`)**
- Wrapper around Dropbox SDK with enhanced error handling
- Implements all Dropbox operations used by tools
- Handles API rate limiting and retry logic

### Resource System

**Resource Handler (`src/resource-handler.ts`)**
- Implements MCP resource protocol for accessing Dropbox items
- Supports URI templates: `dbx://{path}` and `dbx:///shared/{share_id}`
- Provides file content and metadata as MCP resources

**Prompt System (`src/prompt-handler.ts`)**
- Implements MCP prompts for common Dropbox operations
- Includes file review and analysis prompts

### Security Architecture

**Security Utils (`src/security-utils.ts`)**
- AES-256-GCM encryption for sensitive token data
- Secure token storage and retrieval
- Rate limiting and retry mechanisms for token operations

### Testing Structure

Tests are organized by functionality:
- `tests/dropbox/` - Dropbox API operation tests
- `tests/resource/` - Resource system tests  
- `tests/mocks/` - Mock implementations for testing
- Test helpers provide utilities for test data management and cleanup

### Key Design Patterns

- **Modular separation**: Auth, API, tools, resources, and prompts are separate modules
- **Error handling**: Consistent McpError usage throughout with proper error codes
- **Security-first**: All sensitive data is encrypted at rest
- **Resource templates**: Dynamic URI-based resource access
- **Async/await**: Modern async patterns throughout the codebase

## Environment Setup

### Required Environment Variables

**Core Dropbox Integration:**
- `DROPBOX_APP_KEY` - Your Dropbox app's App Key (from Dropbox App Console)
- `DROPBOX_APP_SECRET` - Your Dropbox app's App Secret (from Dropbox App Console)
- `DROPBOX_REDIRECT_URI` - OAuth redirect URI (typically `http://localhost:3000/callback`)
- `TOKEN_ENCRYPTION_KEY` - 32+ character encryption key for secure token storage

**PDF Analysis Features:**
- `OPENROUTER_API_KEY` - API key for AI analysis (get from openrouter.ai)

### Setup Process

1. **Create Dropbox App** (if not already done):
   - Go to [Dropbox App Console](https://www.dropbox.com/developers/apps)
   - Click "Create app" 
   - Choose "Scoped access" API
   - Choose app folder access or full Dropbox access based on your needs
   - Name your app and create it

2. **Configure App Permissions**:
   - In your app settings, go to "Permissions" tab
   - Enable these required scopes:
     - `files.metadata.read` - Read file and folder metadata
     - `files.content.read` - Download files 
     - `files.content.write` - Upload files
     - `sharing.write` - Create sharing links
     - `account_info.read` - Access account information

3. **Set OAuth Redirect URI**:
   - In "Settings" tab, add redirect URI: `http://localhost:3000/callback`

4. **Get Credentials**:
   - Copy the "App key" and "App secret" from the Settings tab
   - These are needed for `DROPBOX_APP_KEY` and `DROPBOX_APP_SECRET`

5. **Configure Environment**:
   ```bash
   # Add to .env file:
   DROPBOX_APP_KEY=your_app_key_here
   DROPBOX_APP_SECRET=your_app_secret_here  
   DROPBOX_REDIRECT_URI=http://localhost:3000/callback
   TOKEN_ENCRYPTION_KEY=your_32_plus_character_encryption_key
   OPENROUTER_API_KEY=your_openrouter_api_key_here
   ```

6. **Run Setup**:
   ```bash
   npm run setup
   ```
   This will guide you through OAuth authentication and create the necessary token files.

### Testing the Setup

After configuration, test the connection:
```bash
npm run inspector
```
Then use the web interface to test tools like `get_account_info` and `list_files`.

## PDF Analysis Features

This server includes comprehensive PDF analysis capabilities powered by OpenRouter's AI models.

### PDF Analysis Tools

**Direct Analysis (`dropbox_analyze_pdf`)**
- Analyzes PDF files with custom prompts using Gemini 2.5 Flash
- Supports thinking mode for complex analysis
- 20MB file size limit
- Real-time analysis without persistent storage

**Document Indexing System**
- `dropbox_index_file` - Index single PDF files with structured AI analysis
- `dropbox_index_folder` - Batch index entire folders of PDFs
- `dropbox_search_indexed` - Search indexed documents by content and metadata
- `dropbox_document_stats` - Business intelligence and document analytics

### Document Types Supported

The AI system automatically classifies and extracts structured data from:
1. **Contracts** - Legal agreements with party extraction, dates, values
2. **Invoices** - Bills with line items, payment terms, due dates
3. **Proposals** - Business proposals with deliverables, timelines
4. **Reports** - Research and analysis documents with findings

### Database Architecture

**SQLite Database (`documents.db`)**
- Stores document metadata, AI analysis results, and type-specific data
- Automatic schema migration and indexing
- Business intelligence queries (expiring contracts, overdue invoices)
- Full-text search capabilities

**Security & Performance**
- Encrypted token storage for API keys
- Batch processing with rate limiting
- Retry logic and error handling
- Confidence scoring for AI analysis results

### Environment Variables for PDF Analysis

Additional required variables:
- `OPENROUTER_API_KEY` - API key for AI analysis (get from openrouter.ai)

### Usage Examples

Analyze a PDF with custom prompt:
```bash
# Use dropbox_analyze_pdf tool
{
  "path": "/contracts/service_agreement.pdf",
  "prompt": "Extract key terms, parties, and expiration date",
  "useThinking": true
}
```

Index documents for structured search:
```bash
# Index single file
dropbox_index_file: {"path": "/invoices/invoice_2024_001.pdf"}

# Index entire folder
dropbox_index_folder: {"path": "/documents", "recursive": true}

# Search indexed documents
dropbox_search_indexed: {"query": "overdue invoices", "docType": "invoice"}
```

## Enhanced Parallel Processing & Multi-Directory Indexing

Version 1.1.0 introduces significant performance improvements and multi-directory support:

### High-Performance Parallel Processing

**Optimized AI Processing:**
- **Parallel AI calls**: Process up to 25-30 documents simultaneously using Gemini 2.5 Flash
- **OpenRouter structured outputs**: Reliable JSON schema-based responses eliminate parsing errors
- **Intelligent retry logic**: Handles service limits (503 errors) and rate limiting automatically
- **Concurrent database operations**: Batch SQLite operations with optimized transactions

**Performance Improvements:**
- **3-5x faster indexing**: From sequential to high-concurrency parallel processing
- **Configurable batch sizes**: 10-30 documents (default: 25) for optimal resource utilization
- **Reduced rate limiting**: Only 100ms delays between batches
- **Smart error handling**: Continue processing even if some documents fail

### Multi-Directory Indexing

**New `dropbox_index_folder` Parameters:**

```javascript
{
  // NEW: Multiple directory support
  "paths": ["/contracts", "/invoices", "/reports"],  // Index multiple directories
  "path": "/single-directory",                       // OR single directory (backward compatible)
  
  // NEW: Directory scoping
  "scopeToDirectories": true,                        // Only process files within specified paths
  
  // Enhanced performance options
  "batchSize": 25,                                   // 10-30 documents processed in parallel
  "recursive": true,                                 // Search subdirectories
  
  // Flexible exclusions
  "excludePaths": ["/temp", "/archive"],            // Exclude specific paths
  "excludePatterns": ["*/backup/*", "*/draft/*"],   // Exclude pattern-based paths
  "respectGlobalExclusions": true                    // Apply global exclusion rules
}
```

**Usage Examples:**

```bash
# Index multiple specific directories
dropbox_index_folder: {
  "paths": ["/Surge/Internal", "/Contracts/Active", "/Invoices/2024"],
  "scopeToDirectories": true,
  "batchSize": 20
}

# High-performance single directory indexing
dropbox_index_folder: {
  "path": "/Documents",
  "batchSize": 30,
  "recursive": true,
  "scopeToDirectories": true
}

# Targeted indexing with exclusions
dropbox_index_folder: {
  "paths": ["/Legal", "/Finance"],
  "excludePatterns": ["*/archive/*", "*/backup/*"],
  "respectGlobalExclusions": false,
  "batchSize": 25
}
```

### Smart Directory Scoping

**Benefits of `scopeToDirectories: true`:**
- **Targeted processing**: Only index files within specified directories
- **Scoped exclusions**: Global exclusions only apply within target directories
- **No manual exclusions**: No need to manually exclude everything outside target paths
- **Cleaner results**: Focus on specific document collections

**Example Workflow:**
```bash
# Before: Required manual exclusions
dropbox_index_folder: {
  "path": "/Surge/Internal",
  "excludePaths": ["/Archive", "/Temp", "/Other", "/External"]  # Manual work
}

# After: Simple scoped indexing
dropbox_index_folder: {
  "path": "/Surge/Internal",
  "scopeToDirectories": true  # Automatically focuses on this directory only
}
```

### Performance Monitoring

The enhanced implementation provides detailed performance metrics:

```bash
# Example output for multi-directory indexing
Multi-directory indexing complete for 3 directories (/contracts, /invoices, /reports):
- Indexed: 45 files
- Skipped (up-to-date): 12 files  
- Errors: 2 files

Directory breakdown:
- /contracts: 15 PDFs found
- /invoices: 28 PDFs found
- /reports: 16 PDFs found
```

### OpenRouter Integration Improvements

**Structured Output Schema:**
- **Reliable parsing**: JSON schema ensures consistent AI responses
- **Type safety**: Structured fields for contracts, invoices, proposals, reports
- **Confidence scoring**: AI-generated confidence levels for analysis quality
- **Error resilience**: Fallback handling for edge cases

**Enhanced AI Prompts:**
- **Optimized for structured output**: Cleaner, more focused prompts
- **Better field extraction**: Improved accuracy for document-specific data
- **Faster processing**: Reduced token usage and response times