server-log-monitor/README.md

AI-powered server log monitoring with BlackBox AI analysis, ElevenLabs conversational agents, file source detection, and intelligent voice alerts

# Server Log Monitor

AI-powered server log monitoring with BlackBox AI analysis and Twilio alerts.

## Features

- 🔍 **Polling-Based Log Monitoring**: Monitors multiple log files and directories with configurable polling intervals
- 🤖 **AI-Powered Analysis**: Uses BlackBox AI to analyze log content intelligently
- 📱 **Smart Alerts**: Twilio SMS and phone call notifications for critical issues
- 🗣️ **Conversational Alerts**: ElevenLabs AI voice agents for interactive critical incident support
- ⚙️ **Configurable**: Flexible configuration via JSON files or environment variables
- 🚀 **Easy Deployment**: NPM package with CLI interface
- 📊 **Controlled Analysis**: Analyzes last N lines from logs on each poll

## Quick Start

### 1. Installation

**Global installation (recommended for CLI usage):**

```bash
npm install -g server-log-monitor
```

**On Linux/Ubuntu, you may need sudo:**

```bash
sudo npm install -g server-log-monitor
```

**Alternative - Use without installation:**

```bash
npx server-log-monitor config create
npx server-log-monitor start
```

**Or for local project:**

```bash
npm install server-log-monitor
```

### 2. Create Configuration

```bash
server-log-monitor config create
```

This creates a `config.sample.json` file with example values.

**⚠️ Important:** You must copy the sample to `config.json` and add your real API keys:

```bash
# Copy the sample configuration
cp config.sample.json config.json

# Edit config.json with your actual API keys
nano config.json  # or use your preferred editor
```

Your `config.json` should look like this with **real values**:

```json
{
  "blackboxAI": {
    "apiKey": "your_blackbox_api_key_here"
  },
  "twilio": {
    "accountSid": "your_twilio_account_sid",
    "authToken": "your_twilio_auth_token",
    "phoneNumber": "+12315999850",
    "toPhoneNumbers": ["+0987654321"]
  },
  "elevenlabs": {
    "apiKey": "your_elevenlabs_api_key_here",
    "agentId": "agent_id_will_be_set_after_setup",
    "phoneNumberId": "phone_number_id_will_be_set_after_setup",
    "voiceId": "your_preferred_voice_id_optional"
  },
  "logMonitor": {
    "paths": ["/var/log", "./logs"],
    "maxLines": 100,
    "pollInterval": 5000
  }
}
```

### 3. Start Monitoring

```bash
server-log-monitor start
```

## Usage

### CLI Commands

```bash
# Start monitoring (default command)
server-log-monitor start

# Monitor a specific file only
server-log-monitor start --file /var/log/nginx/access.log

# Monitor with verbose output (show log content before AI analysis)
server-log-monitor start --verbose

# Monitor specific file with verbose output
server-log-monitor start --file ../log-simulator/logs/app.log --verbose

# Analyze a specific log file
server-log-monitor analyze /var/log/nginx/error.log

# Analyze with specific number of lines
server-log-monitor analyze /var/log/nginx/error.log --lines 50

# Aggregate all logs into one file and analyze (one-time)
server-log-monitor aggregate

# Aggregate with custom settings (one-time)
server-log-monitor aggregate --lines 50 --output my-logs.txt --verbose

# Continuously aggregate and analyze logs (uses config default interval)
server-log-monitor aggregate-watch

# Continuous aggregation with 30-second interval
server-log-monitor aggregate-watch --interval 30

# Continuous aggregation with custom output and verbose mode
server-log-monitor aggregate-watch --output my-logs.txt --verbose

# Check status
server-log-monitor status

# Validate configuration
server-log-monitor config validate

# Health check
server-log-monitor health

# Test SMS alerts
server-log-monitor test-sms

# Test voice calls
server-log-monitor test-call

# ElevenLabs Conversational Agent Setup
server-log-monitor elevenlabs:setup

# Test ElevenLabs integration
server-log-monitor elevenlabs-test

# Update ElevenLabs agent configuration
server-log-monitor elevenlabs-update

# ElevenLabs Tools Integration
server-log-monitor tool-server

# Start tool server with custom settings
server-log-monitor tool-server --port 3002 --codebase /path/to/your/project

# Update to latest version
server-log-monitor update

# Show version
server-log-monitor --version

# Help
server-log-monitor help
```

### Command Options

- `--config <path>` - Path to configuration file
- `--file <path>` - Monitor only a specific file (for start command)
- `--verbose, --show-logs` - Show detailed output (for start/aggregate commands)
- `--lines <number>` - Number of lines to collect/analyze (for analyze/aggregate commands)
- `--output <path>` - Output file for aggregated logs (for aggregate command, default: aggregated-logs.txt)
- `--interval <number>` - Analysis interval in seconds (for aggregate-watch command, uses config default if not specified)
- `--version, -v` - Show version number

### Environment Variables

You can also configure using environment variables:

```bash
export BLACKBOX_API_KEY="your_api_key"
export TWILIO_ACCOUNT_SID="your_sid"
export TWILIO_AUTH_TOKEN="your_token"
export ELEVENLABS_API_KEY="your_elevenlabs_api_key"
export ELEVENLABS_VOICE_ID="your_preferred_voice_id"
export LOG_PATHS="/var/log,./logs"
```

## Configuration

### BlackBox AI Settings

- `apiKey`: Your BlackBox AI API key (required)
- `model`: AI model to use (default: "blackboxai/anthropic/claude-sonnet-4")
- `timeout`: Request timeout in milliseconds (default: 30000)

### ElevenLabs Settings

- `apiKey`: Your ElevenLabs API key (required for conversational alerts)
- `agentId`: Agent ID (auto-set after running setup command)
- `phoneNumberId`: Phone number ID (auto-set after setup)
- `voiceId`: Voice ID from ElevenLabs voice library (optional, improves voice quality)

### Log Monitoring Settings (Polling Mode)

- `paths`: Array of log file paths or directories to monitor
- `patterns`: File patterns to match (default: ["*.log", "*.txt"])
- `maxLines`: **Maximum lines to read from end of each log file per poll** (default: 100)
- `pollInterval`: **Polling frequency in milliseconds** - how often to check log files (default: 200000)
- `encoding`: File encoding (default: 'utf8')
- `recursive`: Recursively scan subdirectories (default: false)

**How Polling Works:**
- Every `pollInterval` ms, reads the last `maxLines` from each monitored log file
- Only analyzes files that have been modified since last poll
- Sends the entire tail content (up to `maxLines`) to AI for analysis

### Alert Settings

- `cooldownPeriod`: Minimum time between alerts (default: 5 minutes)
- `enableSMS`: Enable SMS notifications
- `enableCalls`: Enable phone call notifications
- `onlyCallForCritical`: Only make calls for critical issues

## Log Aggregation

The `aggregate` command collects logs from all monitored files into a single file and analyzes them together. This is useful for getting a comprehensive view across all your services and detecting patterns that span multiple log files.

The `aggregate-watch` command provides continuous monitoring with periodic aggregation and analysis, perfect for ongoing system monitoring.

### How It Works

1. **Discovery**: Finds all log files based on your configuration (paths and patterns)
2. **Collection**: Reads the last N lines from each file (default: 100 lines per file)
3. **Aggregation**: Combines all content into a single file with clear separators
4. **Analysis**: Sends the entire aggregated content to AI for comprehensive analysis

### Usage Examples

```bash
# Basic aggregation (uses default config paths like /var/log)
server-log-monitor aggregate

# Aggregate with custom line count and output file
server-log-monitor aggregate --lines 50 --output today-logs.txt

# Aggregate with verbose output to see what's being collected
server-log-monitor aggregate --verbose

# Use custom config for specific log directories
server-log-monitor aggregate --config /path/to/custom-config.json

# Continuous aggregation and monitoring (uses config default interval)
server-log-monitor aggregate-watch

# Continuous monitoring with custom 30-second interval
server-log-monitor aggregate-watch --interval 30

# Continuous monitoring with custom output file and verbose mode
server-log-monitor aggregate-watch --output my-logs.txt --verbose
```

### Benefits

- **Cross-service analysis**: Detect issues that affect multiple services
- **Pattern recognition**: Find correlations between different log sources
- **Comprehensive reports**: Get a single analysis covering your entire system
- **Efficient troubleshooting**: All relevant logs in one place

The aggregated file includes headers showing which file each section came from, making it easy to trace issues back to specific services.

## Conversational AI Alerts (ElevenLabs)

For critical incidents, the system can make intelligent voice calls using ElevenLabs conversational AI agents. These agents can:

- **Answer your questions** about the specific alert during the call
- **Explain technical issues** in plain language
- **Provide troubleshooting guidance** based on the log context
- **Assess severity** and recommend next steps
- **Access real-time log data** during the conversation

### Setup ElevenLabs Integration

1. **Get ElevenLabs API Key**: Sign up at [ElevenLabs](https://elevenlabs.io) and get your API key
2. **Add to Configuration**: Update your `config.json` with the ElevenLabs section
3. **Run Setup Command**:
   ```bash
   server-log-monitor elevenlabs:setup
   ```
4. **Test the Integration**:
   ```bash
   server-log-monitor elevenlabs:test
   server-log-monitor elevenlabs:test-call  # Makes actual test call
   ```

### How Conversational Alerts Work

When a **critical** alert is detected:
- The system prepares context from recent logs and error details
- Makes an outbound call using your ElevenLabs agent
- The AI agent has access to:
  - Current alert severity and summary
  - Specific error messages and counts
  - Recent log entries for context
  - System status information

**Alert Routing Logic:**
- **Critical Alerts**: Conversational voice calls only (no SMS)
- **Warning/Normal Alerts**: SMS only (no voice calls)
- **Fallback**: If voice calls fail, SMS is sent as backup

### Example Conversation

When you answer a critical alert call:

**Agent**: "Hello! I'm your server monitoring assistant. I've detected a critical alert that needs your attention. I found a database connection failure affecting your application. Would you like me to explain what's happening?"

**You**: "Yes, what exactly is wrong?"

**Agent**: "I'm seeing 3 critical errors in your database logs. The main issue is connection timeouts to your database server at 192.168.1.100. This started about 5 minutes ago and is preventing users from accessing your application. I recommend checking if the database service is running and verifying network connectivity first."

## ElevenLabs Tools Integration

The server log monitor includes a powerful tool server that extends ElevenLabs conversational agents with the ability to search codebases, execute diagnostic commands, and investigate errors directly in your project files.

### Features

- 🔍 **Codebase Search**: Find code patterns, functions, or error messages in your project
- ⚡ **Terminal Execute**: Run safe diagnostic commands to investigate issues
- 📄 **File Content Viewer**: Examine specific files and line ranges
- 🎯 **Error Location Finder**: Locate where specific errors occur using messages or stack traces
- 📁 **Project Structure**: Get an overview of your project organization

### Quick Start

1. **Start the Tool Server**:
   ```bash
   # Start with default settings (port 3001, blackboxHackathon codebase)
   server-log-monitor tool-server

   # Or customize the configuration
   server-log-monitor tool-server --port 3002 --codebase /path/to/your/project
   ```

2. **Configure ElevenLabs Agent**: Add the server tools to your ElevenLabs agent configuration using the URLs provided when the server starts.

3. **Test the Integration**: Your ElevenLabs agent can now search your codebase and execute diagnostic commands during conversations.

### Available Tool Endpoints

When the tool server starts, it provides these endpoints for ElevenLabs integration:

- `POST /tools/codebase-search` - Search for code patterns in the project
- `POST /tools/terminal-execute` - Execute safe diagnostic commands
- `POST /tools/file-content` - View specific file contents
- `POST /tools/find-error-location` - Locate errors using messages/stack traces
- `POST /tools/project-structure` - Get project organization overview
- `GET /health` - Health check endpoint

### Security Features

- Command injection prevention
- File access restricted to codebase root only
- Whitelist of allowed commands (ls, grep, find, cat, git, npm, python, etc.)
- All operations logged for audit purposes

### Example Agent Workflow

When your ElevenLabs agent receives a critical alert:

1. **Agent**: "I see a database connection error. Let me search your codebase for database-related files."
2. **Uses**: Codebase Search tool to find database connection code
3. **Agent**: "I found the issue in `src/database.js` line 25. Let me examine that file."
4. **Uses**: File Content tool to view the problematic code
5. **Agent**: "The connection timeout is set too low. Let me check if the database service is running."
6. **Uses**: Terminal Execute tool to run `ps aux | grep postgres`
7. **Agent**: "The database service is running. I recommend increasing the timeout from 5 seconds to 30 seconds in your database configuration."

For complete setup and configuration details, see the [ElevenLabs Tools Integration Guide](docs/ELEVENLABS_TOOLS_INTEGRATION.md).

## AI Analysis

The system uses BlackBox AI to analyze log content and returns:

- **Status**: `critical`, `warning`, or `normal`
- **Analysis**: Brief description of findings
- **Errors**: Top 3 most problematic errors (if critical)
- **Recommendations**: Actionable suggestions

### Status Classification

- **Critical**: System failures, crashes, security breaches, database failures
- **Warning**: High error rates, performance issues, timeouts
- **Normal**: Standard operations, successful requests

## Development

### Project Structure

```
server-log-monitor/
├── src/
│   ├── components/          # Core components
│   │   ├── AIAnalyzer.js    # BlackBox AI integration
│   │   └── LogMonitor.js    # Log file monitoring
│   ├── config/              # Configuration management
│   │   └── ConfigManager.js
│   └── index.js             # Main entry point
├── bin/
│   └── cli.js               # CLI executable
├── config/
│   └── config.sample.json   # Sample configuration
├── prompts/
│   └── log-analysis.js      # AI prompts
└── package.json
```

### Running Tests

```bash
npm test
```

### Linting

```bash
npm run lint
```

## Troubleshooting

### Common Configuration Issues

#### "BlackBox AI API key is required" Error

```
Failed to load configuration: Invalid configuration: BlackBox AI API key is required
```

**Solution:**
1. Make sure you created `config.json` (not just `config.sample.json`):
   ```bash
   cp config.sample.json config.json
   ```

2. Edit `config.json` and add your **real** BlackBox AI API key:
   ```json
   {
     "blackboxAI": {
       "apiKey": "your_actual_api_key_here"
     }
   }
   ```

3. Get your API key from [BlackBox AI](https://www.blackbox.ai/)

#### "Config file not found" Error

The monitor looks for `config.json` in the current directory. Either:
- Run from the directory containing `config.json`
- Use `--config` to specify the path: `server-log-monitor start --config /path/to/config.json`

#### Permission Issues (Linux/Ubuntu)

If you get permission errors:
```bash
# Install with sudo
sudo npm install -g server-log-monitor

# Or use npx (no installation)
npx server-log-monitor start
```

#### Twilio Configuration Issues

For SMS/voice alerts, make sure you have:
- Valid Twilio Account SID
- Valid Auth Token
- A Twilio phone number
- Destination phone numbers in E.164 format (+1234567890)

## License

MIT License - see LICENSE file for details.