logs-interceptor-node14
Version:
High-performance, production-ready log interceptor for Node.js 14 applications with Loki integration
489 lines (383 loc) • 12.3 kB
Markdown
# 📊 Logs Interceptor
[](https://badge.fury.io/js/logs-interceptor)
[](https://opensource.org/licenses/MIT)
[](https://nodejs.org)
Enterprise-grade, production-ready log interceptor for Node.js applications with Grafana Loki integration. Zero-impact performance, automatic log collection from all sources, and built-in security features.
## ✨ Key Features
- 🚀 **Zero Performance Impact** - Circuit breaker, memory management, and adaptive sampling
- 🔒 **Security First** - Automatic sanitization of sensitive data (passwords, tokens, credit cards)
- 📝 **Universal Logging** - Intercepts console, Winston, Pino, Morgan, and Bunyan
- 🌐 **Distributed Tracing** - OpenTelemetry integration with trace/span correlation
- 💾 **Smart Buffering** - Automatic batching with memory-aware flushing
- 🔄 **Resilient** - Retry logic, circuit breaker, and emergency file fallback
- 📊 **Observability** - Built-in metrics, health checks, and performance monitoring
- 🎯 **Context Propagation** - AsyncLocalStorage for request-scoped logging
- 🗜️ **Compression** - Automatic gzip compression for network efficiency
## 📦 Installation
```bash
npm install logs-interceptor
# or
yarn add logs-interceptor
# or
pnpm add logs-interceptor
```
## 🚀 Quick Start
### Basic Usage
```typescript
import { init } from 'logs-interceptor';
const logger = init({
transport: {
url: 'https://loki.example.com/loki/api/v1/push',
tenantId: 'my-tenant',
authToken: process.env.LOKI_AUTH_TOKEN,
},
appName: 'my-app',
environment: 'production',
interceptConsole: true, // Automatically capture console.log
});
// Now all console.log, console.error, etc. are captured
console.log('This goes to Loki!');
logger.info('So does this!');
```
### Environment Variables Setup
Create a `.env` file:
```env
LOGS_INTERCEPTOR_URL=https://loki.example.com/loki/api/v1/push
LOGS_INTERCEPTOR_TENANT_ID=my-tenant
LOGS_INTERCEPTOR_AUTH_TOKEN=your-auth-token
LOGS_INTERCEPTOR_APP_NAME=my-app
LOGS_INTERCEPTOR_ENVIRONMENT=production
LOGS_INTERCEPTOR_ENABLED=true
```
Then simply:
```typescript
import { init } from 'logs-interceptor';
const logger = init(); // Auto-configures from environment
```
## 🔧 Advanced Configuration
```typescript
const logger = init({
transport: {
url: 'https://loki.example.com/loki/api/v1/push',
tenantId: 'my-tenant',
authToken: process.env.LOKI_AUTH_TOKEN,
timeout: 10000,
maxRetries: 5,
compression: true,
},
appName: 'production-api',
version: '2.0.0',
environment: 'production',
labels: {
region: 'us-east-1',
cluster: 'prod-cluster',
service: 'api-gateway',
},
dynamicLabels: {
hostname: () => require('os').hostname(),
pid: () => process.pid,
},
buffer: {
maxSize: 500, // Max logs before auto-flush
flushInterval: 10000, // Auto-flush every 10s
maxMemoryMB: 100, // Max memory usage
},
filter: {
levels: ['info', 'warn', 'error', 'fatal'],
samplingRate: 0.8, // Sample 80% of logs
sanitize: true, // Auto-sanitize sensitive data
maxMessageLength: 8192,
sensitivePatterns: [
/password/i,
/token/i,
/api[_-]?key/i,
/credit[_-]?card/i,
/ssn/i,
/cpf/i,
],
},
circuitBreaker: {
enabled: true,
failureThreshold: 10,
resetTimeout: 120000,
},
integrations: {
winston: true,
pino: true,
morgan: true,
},
performance: {
useWorkers: true,
compressionLevel: 6,
},
});
```
## 🔌 Framework Integrations
### Express.js
```typescript
import express from 'express';
import { init } from 'logs-interceptor';
import { expressMiddleware } from 'logs-interceptor/middleware';
const app = express();
const logger = init({ /* config */ });
// Add request/response logging
app.use(expressMiddleware(logger));
// Morgan integration
import morgan from 'morgan';
app.use(morgan('combined', {
stream: logger.getMorganStream(),
}));
```
### Koa.js
```typescript
import Koa from 'koa';
import { init } from 'logs-interceptor';
import { koaMiddleware } from 'logs-interceptor/middleware';
const app = new Koa();
const logger = init({ /* config */ });
app.use(koaMiddleware(logger));
```
### Fastify
```typescript
import fastify from 'fastify';
import { init } from 'logs-interceptor';
import { fastifyPlugin } from 'logs-interceptor/middleware';
const app = fastify();
const logger = init({ /* config */ });
app.register(fastifyPlugin(logger));
```
## 🔗 Logger Integrations
### Winston
```typescript
import winston from 'winston';
const logger = init({
integrations: { winston: true },
});
const winstonLogger = winston.createLogger({
transports: [
logger.getWinstonTransport(),
new winston.transports.Console(),
],
});
winstonLogger.info('This goes to both console and Loki!');
```
### Pino
```typescript
import pino from 'pino';
const logger = init({
integrations: { pino: true },
});
const pinoLogger = pino({
destination: logger.getPinoStream(),
});
pinoLogger.info('High-performance logging to Loki!');
```
## 🎯 Context Propagation
```typescript
// Add context to all logs within a request
app.use((req, res, next) => {
logger.runWithContext({
userId: req.user?.id,
sessionId: req.session?.id,
requestId: req.headers['x-request-id'],
}, next);
});
// All logs within this context will include the metadata
app.get('/api/users/:id', async (req, res) => {
logger.info('Fetching user'); // Includes userId, sessionId, requestId
try {
const user = await getUserById(req.params.id);
logger.info('User fetched successfully');
res.json(user);
} catch (error) {
logger.error('Failed to fetch user', { error: error.message });
res.status(500).json({ error: 'Internal server error' });
}
});
```
## 🔒 Security Features
### Automatic Sensitive Data Sanitization
```typescript
// These will be automatically sanitized
logger.info('User login', {
username: 'john@example.com',
password: 'secret123', // → [REDACTED]
creditCard: '4111111111111111', // → [REDACTED]
apiKey: 'sk_live_abc123', // → [REDACTED]
});
```
### Custom Sanitization Rules
```typescript
const logger = init({
filter: {
sanitize: true,
sensitivePatterns: [
/password/i,
/secret/i,
/\b\d{3}-\d{2}-\d{4}\b/, // SSN
/custom-pattern/,
],
},
});
```
## 📊 Monitoring & Health Checks
```typescript
// Health endpoint
app.get('/health', (req, res) => {
const health = logger.getHealth();
const metrics = logger.getMetrics();
res.json({
status: health.healthy ? 'healthy' : 'unhealthy',
health: {
...health,
circuitBreaker: health.circuitBreakerState,
memoryUsageMB: health.memoryUsageMB,
bufferUtilization: health.bufferUtilization,
},
metrics: {
logsProcessed: metrics.logsProcessed,
logsDropped: metrics.logsDropped,
logsSanitized: metrics.logsSanitized,
avgFlushTime: metrics.avgFlushTime,
errorCount: metrics.errorCount,
},
});
});
```
## 🛡️ Error Handling & Graceful Shutdown
```typescript
// Capture unhandled errors
process.on('unhandledRejection', (reason, promise) => {
logger.fatal('Unhandled Promise Rejection', { reason });
});
process.on('uncaughtException', (error) => {
logger.fatal('Uncaught Exception', {
error: error.message,
stack: error.stack,
});
logger.flush().then(() => process.exit(1));
});
// Graceful shutdown
async function gracefulShutdown(signal) {
console.log(`Received ${signal}, shutting down gracefully...`);
server.close(); // Stop accepting new connections
await logger.flush(); // Flush remaining logs
await logger.destroy(); // Cleanup resources
process.exit(0);
}
process.on('SIGTERM', () => gracefulShutdown('SIGTERM'));
process.on('SIGINT', () => gracefulShutdown('SIGINT'));
```
## 🚀 Performance Optimization
### Adaptive Sampling
```typescript
const logger = init({
filter: {
samplingRate: 0.8, // Sample 80% of logs in production
levels: ['info', 'warn', 'error', 'fatal'], // Skip debug in production
},
});
```
### Circuit Breaker
Automatically stops sending logs when Loki is down to prevent application impact:
```typescript
const logger = init({
circuitBreaker: {
enabled: true,
failureThreshold: 5, // Open after 5 failures
resetTimeout: 60000, // Try again after 1 minute
halfOpenRequests: 3, // Test with 3 requests
},
});
```
### Memory Management
Automatic memory pressure handling:
```typescript
const logger = init({
buffer: {
maxMemoryMB: 50, // Auto-flush when memory usage exceeds 50MB
maxSize: 1000, // Max 1000 logs in buffer
},
});
```
## 📝 API Reference
### Logger Methods
```typescript
logger.debug(message: string, context?: object): void
logger.info(message: string, context?: object): void
logger.warn(message: string, context?: object): void
logger.error(message: string, context?: object): void
logger.fatal(message: string, context?: object): void
logger.trackEvent(eventName: string, properties?: object): void
logger.flush(): Promise<void>
logger.getMetrics(): LoggerMetrics
logger.getHealth(): HealthStatus
logger.runWithContext<T>(context: object, fn: () => T): T
logger.destroy(): Promise<void>
```
### Environment Variables
| Variable | Description | Default |
|----------|-------------|---------|
| `LOGS_INTERCEPTOR_URL` | Loki push endpoint | Required |
| `LOGS_INTERCEPTOR_TENANT_ID` | Loki tenant ID | Required |
| `LOGS_INTERCEPTOR_AUTH_TOKEN` | Authentication token | Optional |
| `LOGS_INTERCEPTOR_APP_NAME` | Application name | Required |
| `LOGS_INTERCEPTOR_ENVIRONMENT` | Environment name | `production` |
| `LOGS_INTERCEPTOR_VERSION` | App version | `1.0.0` |
| `LOGS_INTERCEPTOR_BUFFER_SIZE` | Buffer size | `100` |
| `LOGS_INTERCEPTOR_FLUSH_INTERVAL` | Flush interval (ms) | `5000` |
| `LOGS_INTERCEPTOR_LOG_LEVEL` | Log levels (comma-separated) | `debug,info,warn,error,fatal` |
| `LOGS_INTERCEPTOR_SAMPLING_RATE` | Sampling rate (0-1) | `1.0` |
| `LOGS_INTERCEPTOR_CIRCUIT_BREAKER` | Enable circuit breaker | `true` |
| `LOGS_INTERCEPTOR_SANITIZE` | Sanitize sensitive data | `true` |
| `LOGS_INTERCEPTOR_MAX_MEMORY_MB` | Max memory usage | `50` |
| `LOGS_INTERCEPTOR_DEBUG` | Debug mode | `false` |
| `LOGS_INTERCEPTOR_ENABLED` | Enable/disable logging | `true` |
## 🧪 Testing
```typescript
// Test mode - logs to memory instead of Loki
const logger = init({
transport: {
url: 'memory://test',
tenantId: 'test',
},
debug: true,
});
// Access logs in tests
logger.on('log', (entry) => {
console.log('Log captured:', entry);
});
```
## 📊 Grafana Dashboard
Import our pre-configured Grafana dashboard for monitoring:
1. Import dashboard JSON from `dashboards/logs-interceptor.json`
2. Configure Loki data source
3. Set variables for `app_name` and `environment`
## 🐛 Troubleshooting
### Logs not appearing in Loki
1. Check circuit breaker status: `logger.getHealth()`
2. Verify network connectivity to Loki
3. Check authentication token
4. Enable debug mode: `LOGS_INTERCEPTOR_DEBUG=true`
### High memory usage
1. Reduce buffer size
2. Increase flush frequency
3. Enable sampling
4. Check for memory leaks in context data
### Performance impact
1. Enable circuit breaker
2. Reduce sampling rate
3. Use worker threads
4. Increase compression level
## 🤝 Contributing
Contributions are welcome! Please read our [Contributing Guide](CONTRIBUTING.md) for details.
## 📄 License
MIT © Leonardo Zwirtes
## 🔗 Links
- [GitHub Repository](https://github.com/leozw/logs-interceptor-js)
- [NPM Package](https://www.npmjs.com/package/logs-interceptor)
- [Documentation](https://logs-interceptor.dev)
- [Examples](https://github.com/leozw/logs-interceptor-js/tree/main/examples)
## 💬 Support
- [GitHub Issues](https://github.com/leozw/logs-interceptor-js-js/issues)
- [Discussions](https://github.com/leozw/logs-interceptor-js-js/discussions)