logs-interceptor
Version:
High-performance, production-ready log interceptor for Node.js applications with Loki integration
427 lines (340 loc) • 12.6 kB
Markdown
# logs-interceptor
[](https://badge.fury.io/js/logs-interceptor)
[](https://opensource.org/licenses/MIT)
[](https://github.com/leozw/logs-interceptor/actions)
High-performance, production-ready log interceptor for Node.js applications with Grafana Loki integration. Built for enterprise-grade observability with minimal performance overhead.
## 🚀 Features
- **High Performance**: Optimized buffering, batching, and HTTP connection pooling
- **Production Ready**: Comprehensive error handling, retries, and graceful degradation
- **Multiple Initialization Methods**: Programmatic API, environment variables, or NODE_OPTIONS
- **Console Interception**: Seamlessly capture all console.log calls without code changes
- **OpenTelemetry Integration**: Automatic trace/span ID correlation
- **Advanced Filtering**: Log levels, sampling, message patterns, and length limits
- **Real-time Metrics**: Performance monitoring and health checks
- **Zero Dependencies**: Minimal footprint with only essential dependencies
- **TypeScript Support**: Full type definitions and IntelliSense support
## 📦 Installation
```bash
npm install logs-interceptor
# or
yarn add logs-interceptor
```
## 🎯 Quick Start
### Programmatic Usage
```typescript
import { init, logger } from 'logs-interceptor';
// Initialize with configuration
init({
transport: {
url: 'http://localhost:3100/loki/api/v1/push',
tenantId: 'my-tenant',
authToken: 'optional-token'
},
appName: 'my-application',
environment: 'production',
interceptConsole: true
});
// Use the logger
logger.info('Application started', { version: '1.0.0' });
logger.error('Something went wrong', { error: 'ECONNREFUSED' });
logger.trackEvent('user_login', { userId: 12345, method: 'oauth' });
```
### Environment Variables
```bash
# Required
export LOGS_INTERCEPTOR_URL="http://localhost:3100/loki/api/v1/push"
export LOGS_INTERCEPTOR_TENANT_ID="my-tenant"
export LOGS_INTERCEPTOR_APP_NAME="my-application"
# Optional
export LOGS_INTERCEPTOR_AUTH_TOKEN="your-token"
export LOGS_INTERCEPTOR_ENVIRONMENT="production"
export LOGS_INTERCEPTOR_LABELS="service=api,datacenter=us-east-1"
export LOGS_INTERCEPTOR_BUFFER_SIZE="100"
export LOGS_INTERCEPTOR_FLUSH_INTERVAL="5000"
# Run your application
node app.js
```
### Auto-initialization with NODE_OPTIONS
```bash
NODE_OPTIONS="--require logs-interceptor/preload" \
LOGS_INTERCEPTOR_URL="http://localhost:3100/loki/api/v1/push" \
LOGS_INTERCEPTOR_TENANT_ID="my-tenant" \
LOGS_INTERCEPTOR_APP_NAME="my-app" \
node app.js
```
## ⚙️ Configuration
### Complete Configuration Example
```typescript
import { init } from 'logs-interceptor';
init({
// Transport Configuration
transport: {
url: 'http://localhost:3100/loki/api/v1/push',
tenantId: 'my-tenant',
authToken: 'optional-bearer-token',
timeout: 5000, // HTTP timeout in ms
maxRetries: 3, // Number of retries on failure
retryDelay: 1000, // Delay between retries in ms
compression: true // Enable gzip compression
},
// Application Metadata
appName: 'my-application',
version: '1.0.0',
environment: 'production',
// Static Labels (added to all logs)
labels: {
service: 'api',
datacenter: 'us-east-1',
team: 'backend'
},
// Dynamic Labels (computed at runtime)
dynamicLabels: {
hostname: () => require('os').hostname(),
pid: () => process.pid.toString(),
memory: () => Math.round(process.memoryUsage().heapUsed / 1024 / 1024) + 'MB'
},
// Buffer Configuration
buffer: {
maxSize: 100, // Max logs before forced flush
flushInterval: 5000, // Auto-flush interval in ms
maxAge: 30000, // Max age before flush in ms
autoFlush: true // Enable automatic flushing
},
// Filtering and Sampling
filter: {
levels: ['info', 'warn', 'error', 'fatal'], // Allowed log levels
patterns: [/error/i, /warning/i], // Include patterns
samplingRate: 1.0, // 1.0 = 100%, 0.1 = 10%
maxMessageLength: 8192 // Max message length
},
// Console Interception
interceptConsole: true, // Capture console.log calls
preserveOriginalConsole: true, // Keep original console output
// Monitoring
enableMetrics: true, // Track performance metrics
enableHealthCheck: true, // Enable health monitoring
// Debug Options
debug: false, // Enable debug logging
silentErrors: false // Suppress error logs
});
```
### Environment Variables Reference
| Variable | Description | Default |
|----------|-------------|---------|
| `LOGS_INTERCEPTOR_URL` | Loki endpoint URL | Required |
| `LOGS_INTERCEPTOR_TENANT_ID` | Loki tenant ID | Required |
| `LOGS_INTERCEPTOR_APP_NAME` | Application name | Required |
| `LOGS_INTERCEPTOR_AUTH_TOKEN` | Bearer token for auth | Optional |
| `LOGS_INTERCEPTOR_ENVIRONMENT` | Environment name | `"production"` |
| `LOGS_INTERCEPTOR_VERSION` | Application version | `"1.0.0"` |
| `LOGS_INTERCEPTOR_LABELS` | Static labels (key=value,key2=value2) | `{}` |
| `LOGS_INTERCEPTOR_BUFFER_SIZE` | Buffer size before flush | `100` |
| `LOGS_INTERCEPTOR_FLUSH_INTERVAL` | Auto-flush interval (ms) | `5000` |
| `LOGS_INTERCEPTOR_LOG_LEVEL` | Comma-separated levels | `"debug,info,warn,error,fatal"` |
| `LOGS_INTERCEPTOR_SAMPLING_RATE` | Sampling rate (0.0-1.0) | `1.0` |
| `LOGS_INTERCEPTOR_DEBUG` | Enable debug mode | `false` |
| `LOGS_INTERCEPTOR_ENABLED` | Enable/disable logger | `true` |
## 🔧 API Reference
### Logger Methods
```typescript
import { logger } from 'logs-interceptor';
// Standard logging methods
logger.debug('Debug message', { context: 'optional' });
logger.info('Info message', { userId: 123 });
logger.warn('Warning message', { code: 'WARN_001' });
logger.error('Error message', { error: error.message });
logger.fatal('Fatal error', { stack: error.stack });
// Event tracking
logger.trackEvent('user_action', {
action: 'click',
element: 'signup-button',
userId: 12345
});
// Manual flush
await logger.flush();
// Get metrics
const metrics = logger.getMetrics();
console.log('Logs processed:', metrics.logsProcessed);
// Health check
const health = logger.getHealth();
console.log('Healthy:', health.healthy);
```
### Advanced Usage
```typescript
import { LogsInterceptor } from 'logs-interceptor';
// Create multiple instances
const apiLogger = new LogsInterceptor({
transport: { url: 'http://loki:3100/loki/api/v1/push', tenantId: 'api' },
appName: 'api-service',
labels: { service: 'api' }
});
const workerLogger = new LogsInterceptor({
transport: { url: 'http://loki:3100/loki/api/v1/push', tenantId: 'worker' },
appName: 'worker-service',
labels: { service: 'worker' }
});
// Event handling
apiLogger.on('error', (error) => {
console.error('Logger error:', error);
});
apiLogger.on('flush', ({ count, duration }) => {
console.log(`Flushed ${count} logs in ${duration}ms`);
});
// Graceful shutdown
process.on('SIGTERM', async () => {
await apiLogger.destroy();
await workerLogger.destroy();
process.exit(0);
});
```
## 🛠️ CLI Tools
The package includes a CLI for testing and configuration management:
```bash
# Test connectivity and configuration
npx logs-interceptor test --url http://localhost:3100/loki/api/v1/push --tenant my-tenant --app my-app
# Validate configuration
npx logs-interceptor validate --config ./config.json
# Generate sample configuration
npx logs-interceptor generate-config --output ./logs-config.json
# Show environment variables example
npx logs-interceptor env-example
```
## 📊 Monitoring & Metrics
### Built-in Metrics
```typescript
const metrics = logger.getMetrics();
console.log({
logsProcessed: metrics.logsProcessed, // Total logs processed
logsDropped: metrics.logsDropped, // Logs dropped (sampling)
flushCount: metrics.flushCount, // Number of flushes
errorCount: metrics.errorCount, // HTTP errors
bufferSize: metrics.bufferSize, // Current buffer size
avgFlushTime: metrics.avgFlushTime, // Average flush time (ms)
lastFlushTime: metrics.lastFlushTime // Last successful flush timestamp
});
```
### Health Monitoring
```typescript
const health = logger.getHealth();
console.log({
healthy: health.healthy, // Overall health status
lastSuccessfulFlush: health.lastSuccessfulFlush,
consecutiveErrors: health.consecutiveErrors,
bufferUtilization: health.bufferUtilization, // 0.0 - 1.0
uptime: health.uptime // Uptime in milliseconds
});
```
## 🏗️ Architecture
### Performance Optimizations
- **Smart Buffering**: Automatic batching with configurable thresholds
- **Connection Pooling**: Reuse HTTP connections for better performance
- **Compression**: Optional gzip compression for reduced bandwidth
- **Async Processing**: Non-blocking log processing
- **Memory Management**: Circular reference handling and memory leak prevention
### Error Handling
- **Retry Logic**: Exponential backoff with configurable retries
- **Circuit Breaker**: Temporary failure handling
- **Graceful Degradation**: Continue operation during transport failures
- **Process Handlers**: Automatic flush on process exit
### Security
- **Secure Defaults**: No sensitive data exposure in logs
- **Token Management**: Secure authentication token handling
- **Data Sanitization**: Automatic circular reference and sensitive data handling
## 🔌 Integrations
### Docker Compose Example
```yaml
version: '3.8'
services:
app:
build: .
environment:
- NODE_OPTIONS=--require logs-interceptor/preload
- LOGS_INTERCEPTOR_URL=http://loki:3100/loki/api/v1/push
- LOGS_INTERCEPTOR_TENANT_ID=app
- LOGS_INTERCEPTOR_APP_NAME=my-app
- LOGS_INTERCEPTOR_ENVIRONMENT=production
depends_on:
- loki
loki:
image: grafana/loki:latest
ports:
- "3100:3100"
volumes:
- ./loki-config.yaml:/etc/loki/local-config.yaml
command: -config.file=/etc/loki/local-config.yaml
grafana:
image: grafana/grafana:latest
ports:
- "3000:3000"
environment:
- GF_SECURITY_ADMIN_PASSWORD=admin
```
### Kubernetes Deployment
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-app
spec:
replicas: 3
selector:
matchLabels:
app: my-app
template:
metadata:
labels:
app: my-app
spec:
containers:
- name: app
image: my-app:latest
env:
- name: NODE_OPTIONS
value: "--require logs-interceptor/preload"
- name: LOGS_INTERCEPTOR_URL
value: "http://loki:3100/loki/api/v1/push"
- name: LOGS_INTERCEPTOR_TENANT_ID
value: "my-app"
- name: LOGS_INTERCEPTOR_APP_NAME
value: "my-app"
- name: LOGS_INTERCEPTOR_ENVIRONMENT
value: "production"
- name: LOGS_INTERCEPTOR_LABELS
value: "namespace=default,pod=$(POD_NAME)"
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
```
## 🧪 Testing
```bash
# Run tests
npm test
# Run with coverage
npm run test:coverage
# Test specific configuration
npm run test -- --config ./test-config.json
```
## 🤝 Contributing
1. Fork the repository
2. Create a feature branch (`git checkout -b feature/amazing-feature`)
3. Commit your changes (`git commit -m 'Add amazing feature'`)
4. Push to the branch (`git push origin feature/amazing-feature`)
5. Open a Pull Request
## 📋 Roadmap
- [ ] Support for multiple transport backends (Elasticsearch, AWS CloudWatch)
- [ ] Built-in log rotation and archival
- [ ] Advanced query and search capabilities
- [ ] Real-time log streaming
- [ ] Integration with popular frameworks (Express, Fastify, NestJS)
- [ ] Grafana dashboard templates
- [ ] Performance benchmarking tools
## 📄 License
This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
## 🆘 Support
- 📖 [Documentation](https://github.com/leozw/logs-interceptor/wiki)
- 🐛 [Issue Tracker](https://github.com/leozw/logs-interceptor/issues)
- 💬 [Discussions](https://github.com/leozw/logs-interceptor/discussions)
- 📧 [Email Support](mailto:leonardo@zwirtes.com)
---
Made with ❤️ for the Node.js observability community