@vtex/diagnostics-nodejs/README.md

Diagnostics library for Node.js applications

# OpenTelemetry Diagnostics Library for Node.js

A comprehensive Node.js library that simplifies telemetry collection using OpenTelemetry, providing unified interfaces for metrics, logs, and traces with advanced sampling, hot-reload configuration, and easy-to-use instrumentation setups for popular frameworks.

## Features

✅ **Unified Telemetry**: Metrics, logs, and traces in one library  
✅ **Auto-Instrumentation**: Express, Fastify, Koa, NestJS support  
✅ **Advanced Sampling**: Rule-based, parent-based, debug override, hot-reload  
✅ **Remote Configuration**: File-based, S3, with live reload and polling  
✅ **Configuration Override**: Runtime config updates and hot-swapping  
✅ **Resource Discovery**: Automatic cloud and Kubernetes detection  
✅ **Context Propagation**: Baggage and trace context across services  
✅ **Custom Processors**: VTEX-specific attribute enrichment  
✅ **Distributed Resampling**: Override parent sampling decisions  
✅ **gRPC Export**: High-performance OTLP export  

## Installation

```bash
npm install @vtex/diagnostics-nodejs
```

## Quick Start

```javascript
import { NewTelemetryClient, Instrumentation } from '@vtex/diagnostics-nodejs';

async function main() {
  // 1. Define your instrumentations
  const instrumentations = Instrumentation.CommonInstrumentations.express();

  // 2. Initialize telemetry client BEFORE importing frameworks
  const telemetryClient = await NewTelemetryClient(
    'my-app-id',           // Application ID
    'my-client-instance',  // Client instance name  
    'my-service-name'      // Service name
  );

  // 3. Create telemetry clients
  const tracesClient = await telemetryClient.newTracesClient({
    setGlobalProvider: true,
  });
  const metricsClient = await telemetryClient.newMetricsClient();
  const logsClient = await telemetryClient.newLogsClient();

  // 4. Register the instrumentations. This activates them.
  telemetryClient.registerInstrumentations(instrumentations);
  
  // 5. NOW import your frameworks
  const express = require('express');
  const app = express();
  
  // Your app is automatically instrumented!
  app.get('/api/users', (req, res) => {
    logsClient.info('Users endpoint called', { userId: req.query.id });
    res.json({ users: [] });
  });

  app.listen(3000);
}

main().catch(console.error);
```

## Remote Configuration Management

### File-Based Configuration with Hot Reload

```javascript
const telemetryClient = await NewTelemetryClient(
  'my-app-id',
  'my-client', 
  'my-service',
  {
    config: {
      configPath: './config/telemetry.json',
      enableHotReload: true,           // Enable live reload
      pollIntervalMs: 15000           // Check for changes every 15s
    }
  }
);

// Config changes are automatically applied without restart!
// Watch logs for: "🔄 Hot reloading sampling configuration..."
```

### S3-Based Remote Configuration

```javascript
const telemetryClient = await NewTelemetryClient(
  'my-app', 'my-client', 'my-service',
  {
    config: {
      s3Config: {
        bucket: 'my-telemetry-configs',
        key: 'environments/prod/my-service.json',
        region: 'us-east-1'
      },
      pollIntervalMs: 30000,          // Poll S3 every 30s
      enableHotReload: true
    }
  }
);

// Centralized config management:
// 1. Update config in S3
// 2. All instances pick up changes automatically
// 3. No service restarts required
```

### Configuration Override at Runtime

```javascript
// The ConfigManager is an internal component and not exposed via a public `getConfigManager()` method.
// Configuration updates are handled automatically via file/S3 polling when `enableHotReload` is true.
// The `onConfigChange` functionality would require implementing a custom configuration provider.

console.log('Configuration from file/S3 will be hot-reloaded automatically if enabled.');
```

### Environment-Specific Configuration

```javascript
// Base configuration
const telemetryClient = await NewTelemetryClient(
  'my-app', 'my-client', 'my-service',
  {
    config: {
      configPath: `./config/${process.env.NODE_ENV || 'development'}.json`,
      enableHotReload: process.env.NODE_ENV !== 'production'
    },
    
    // Override config based on environment
    environment: process.env.DEPLOYMENT_ENV,
    region: process.env.AWS_REGION,
    
    // Additional runtime attributes
    additionalAttrs: {
      'deployment.version': process.env.GIT_SHA,
      'deployment.timestamp': new Date().toISOString()
    }
  }
);
```

### Using Configuration Files

```javascript
const telemetryClient = await NewTelemetryClient(
  'my-app',
  'my-client', 
  'my-service',
  {
    config: {
      configPath: './config/telemetry.json',
      enableHotReload: true,
      pollIntervalMs: 15000
    }
  }
);
```

### Configuration Override Examples

```json
// Development: High sampling for debugging
// config/development.json
{
  "traces": {
    "sampling": {
      "defaultRate": 1.0,
      "rules": [
        {
          "name": "debug_everything",
          "sampleRate": 1.0,
          "conditions": []
        }
      ]
    }
  }
}
```json
// Production: Selective sampling
// config/production.json  
{
  "traces": {
    "sampling": {
      "defaultRate": 0.01,
      "rules": [
        {
          "name": "critical_endpoints",
          "sampleRate": 1.0,
          "conditions": [
            {
              "attribute": "http.target",
              "operator": "Regex",
              "value": "^/api/(checkout|payment|auth)"
            }
          ]
        },
        {
          "name": "error_traces",
          "sampleRate": 1.0,
          "conditions": [
            {
              "attribute": "http.status_code",
              "operator": "GreaterThanEqual", 
              "value": "400"
            }
          ]
        }
      ]
    }
  }
}
```

### Live Configuration Updates

```javascript
// Monitor configuration changes in real-time
const telemetryClient = await NewTelemetryClient('app', 'client', 'service', {
  config: { enableHotReload: true }
});

// Define instrumentations
const instrumentations = Instrumentation.CommonInstrumentations.autoDetect();

// Initialize clients
const tracesClient = await telemetryClient.newTracesClient();

// Register instrumentations
telemetryClient.registerInstrumentations(instrumentations);

// Example: Change sampling rules via file edit
// 1. Edit config/telemetry.json
// 2. Change defaultRate from 0.1 to 0.5  
// 3. Save file
// 4. Library automatically detects change
// 5. New sampling rules applied within seconds
// 6. See logs: "🔄 Hot reloading sampling configuration..."

console.log('Config will auto-reload from file changes...');
```

### Conditional Configuration Loading

```javascript
async function createTelemetryClient() {
  const isProd = process.env.NODE_ENV === 'production';
  const isK8s = !!process.env.KUBERNETES_SERVICE_HOST;
  
  const config = {
    config: {
      // Production: Use S3 remote config
      ...(isProd && {
        s3Config: {
          bucket: 'prod-telemetry-configs',
          key: `services/${process.env.SERVICE_NAME}/config.json`,
          region: process.env.AWS_REGION
        }
      }),
      
      // Development: Use local file
      ...(!isProd && {
        configPath: './config/dev.json'
      }),
      
      enableHotReload: true,
      pollIntervalMs: isProd ? 60000 : 15000  // Slower polling in prod
    },
    
    // Kubernetes-specific attributes
    ...(isK8s && {
      additionalAttrs: {
        'k8s.namespace': process.env.KUBERNETES_NAMESPACE,
        'k8s.pod': process.env.HOSTNAME,
        'k8s.node': process.env.NODE_NAME
      }
    }),
    
    disableCloudDetect: !isProd,
    debug: !isProd
  };
  
  return await NewTelemetryClient('app', 'client', 'service', config);
}
```

### Global vs Service-Specific Configuration

```json
// Hierarchical configuration override
const baseConfig = {
  "globalResourceAttributes": {
    "service.namespace": "ecommerce",
    "deployment.environment": "production"
  },
  "traces": {
    "exporters": {
      "mode": "otlp",
      "otlp": { "endpoint": "otel-collector:4317" }
    },
    "sampling": {
      "parentBased": true,
      "defaultRate": 0.05
    }
  }
};
```json
// Service-specific overrides
const serviceOverrides = {
  "user-service": {
    "traces": {
      "sampling": {
        "defaultRate": 0.1,
        "rules": [
          {
            "name": "auth_endpoints",
            "sampleRate": 1.0,
            "conditions": [
              { "attribute": "http.target", "operator": "StartsWith", "value": "/auth" }
            ]
          }
        ]
      }
    }
  },
  
  "payment-service": {
    "traces": {
      "sampling": {
        "defaultRate": 1.0,
        "parentBased": false
      }
    }
  },
  
  "inventory-service": {
    "traces": {
      "sampling": {
        "parentBased": false,
        "defaultRate": 0.01,
        "rules": [
          {
            "name": "resample_checkout_flow",
            "sampleRate": 1.0,
            "conditions": [
              { "attribute": "http.target", "operator": "Contains", "value": "checkout" }
            ]
          }
        ]
      }
    }
  }
};
```json
{
  "environment": "production",
  "globalResourceAttributes": {
    "service.namespace": "ecommerce",
    "deployment.environment": "prod"
  },
  "traces": {
    "exporters": {
      "mode": "otlp",
      "otlp": { "endpoint": "otel-collector:4317" }
    },
    "sampling": {
      "parentBased": true,
      "defaultRate": 0.1,
      "rules": [
        {
          "name": "sample_critical_endpoints",
          "sampleRate": 1.0,
          "conditions": [
            {
              "attribute": "http.target",
              "operator": "StartsWith",
              "value": "/api/checkout"
            }
          ]
        },
        {
          "name": "debug_override",
          "sampleRate": 1.0,
          "conditions": [
            {
              "attribute": "debug",
              "operator": "Equals", 
              "value": "true"
            }
          ]
        }
      ]
    }
  }
}
```

## Advanced Sampling

### Rule-Based Sampling

Control sampling with flexible rules that support hot-reload:

```json
// Configuration supports multiple operators and conditions
{
  "sampling": {
    "parentBased": true,
    "defaultRate": 0.05,
    "rules": [
      {
        "name": "high_priority_routes",
        "sampleRate": 1.0,
        "conditions": [
          {
            "attribute": "http.method",
            "operator": "Equals",
            "value": "POST"
          },
          {
            "attribute": "http.target", 
            "operator": "Regex",
            "value": "^/api/(orders|payments)"
          }
        ]
      }
    ]
  }
}
```

### Resampling in Distributed Traces

Override parent sampling decisions in specific services:

```json
// Service A: parentBased: true (follows upstream decisions)
// Service B: parentBased: false (makes independent decisions)
{
  "sampling": {
    "parentBased": false,
    "rules": [
      {
        "name": "force_sample_inventory_operations",
        "sampleRate": 1.0,
        "conditions": [...]
      }
    ]
  }
}
```

### Debug Sampling

Force sampling with debug flags:

```bash
# Always samples when debug is present
curl "http://api/users?debug=true"
curl -H "debug: true" http://api/users
```

### Supported Sampling Operators

- `Equals`, `NotEquals`
- `Contains`, `StartsWith`, `EndsWith`  
- `In`, `NotIn` (arrays)
- `Regex`
- `GreaterThan`, `LessThan`, `GreaterThanEqual`, `LessThanEqual`

## Instrumentation

### Framework Support

```javascript
// Choose your framework bundle
const instrumentations = {
  express: Instrumentation.CommonInstrumentations.express(),
  fastify: Instrumentation.CommonInstrumentations.fastify(), 
  koa: Instrumentation.CommonInstrumentations.koa(),
  nestjs: Instrumentation.CommonInstrumentations.nestjs(),
  minimal: Instrumentation.CommonInstrumentations.minimal(), // HTTP only
  autoDetect: Instrumentation.CommonInstrumentations.autoDetect()
};
```

### Adding Database & External Service Tracing

```javascript
import { MongoDBInstrumentation } from '@opentelemetry/instrumentation-mongodb';
import { RedisInstrumentation } from '@opentelemetry/instrumentation-redis';
import { NewTelemetryClient, Instrumentation } from '@vtex/diagnostics-nodejs';

async function setup() {
  // 1. Create a list of all desired instrumentations
  const myInstrumentations = [
    ...Instrumentation.CommonInstrumentations.express(),
    new MongoDBInstrumentation(),
    new RedisInstrumentation(),
    // Add any other OpenTelemetry community instrumentation
  ];

  // 2. Initialize the client
  const telemetryClient = await NewTelemetryClient('app-id', 'client-name', 'service-name');
  await telemetryClient.newTracesClient(); // and other clients

  // 3. Register all instrumentations
  telemetryClient.registerInstrumentations(myInstrumentations);
}
```

## Context Propagation & Baggage

### Adding Baggage

```javascript
import { Traces } from '@vtex/diagnostics-nodejs';

app.get('/api/users', (req, res) => {
  // Add baggage items that propagate to downstream services
  Traces.addBaggage({
    'user.id': req.user.id,
    'request.priority': 'high',
    'trace.source': 'web-app'
  });
  
  // Single baggage item
  Traces.setBaggage('operation.type', 'user-lookup');
  
  // Retrieve baggage
  const baggage = Traces.getActiveBaggage();
  console.log('Current baggage:', baggage);
});
```

### Middleware for Context Propagation

```javascript
import { Instrumentation } from '@vtex/diagnostics-nodejs';

// Express
app.use(Instrumentation.Middlewares.ContextMiddlewares.Express.ContextPropagationMiddleware());

// Fastify
fastify.register(Instrumentation.Middlewares.ContextMiddlewares.Fastify.createContextPlugin());

// Koa  
app.use(Instrumentation.Middlewares.ContextMiddlewares.Koa.ContextPropagationMiddleware());
```

## Manual Tracing

### Creating Custom Spans

```javascript
const span = tracesClient.startSpan('database-migration', {
  attributes: { 'migration.version': '1.2.0' },
  kind: SpanKind.INTERNAL
});

try {
  span.setAttributes({
    'migration.tables': ['users', 'orders'],
    'migration.duration_estimate': '5min'
  });
  
  span.addEvent('migration-started', { timestamp: Date.now() });
  
  // Your operation
  await runMigration();
  
  span.addEvent('migration-completed');
  
} catch (error) {
  span.recordException(error);
  span.setStatus({ code: SpanStatusCode.ERROR, message: error.message });
  throw error;
} finally {
  span.end();
}
```

### Trace Context Injection/Extraction

```javascript
// Inject trace context into HTTP headers
const headers = {};
tracesClient.inject(headers);
console.log(headers); // { traceparent: "00-...", tracestate: "..." }

// Extract trace context from incoming headers
const context = tracesClient.extract(req.headers);
context.execute(() => {
  // Code executed within the extracted trace context
  logsClient.info('Processing with extracted context');
});
```

## Metrics

### Counter Metrics

```javascript
import { Metrics } from '@vtex/diagnostics-nodejs';

const httpRequestsCounter = metricsClient.createCounter(
  'http_requests_total',
  { 
    description: 'Total HTTP requests processed',
    unit: '1'
  }
);

// Increment counter
httpRequestsCounter.increment({ method: 'GET', status: '200' });
httpRequestsCounter.add(5, { method: 'POST', status: '201' });
```

### Histogram Metrics

```javascript
const responseTimeHistogram = metricsClient.createHistogram(
  'http_response_time_seconds',
  {
    description: 'HTTP response time distribution',
    unit: 's',
    boundaries: [0.01, 0.05, 0.1, 0.5, 1, 5, 10]
  }
);

responseTimeHistogram.record(0.145, { route: '/api/users', method: 'GET' });
```

### Gauge Metrics

```javascript
const activeConnectionsGauge = metricsClient.createGauge(
  'active_connections',
  { description: 'Current active database connections' }
);

activeConnectionsGauge.set(42);
activeConnectionsGauge.add(5);    // Increment by 5
activeConnectionsGauge.subtract(2); // Decrement by 2
```

## Structured Logging

### Log Levels & Attributes

```javascript
// Structured logging with attributes
logsClient.info('User login successful', {
  'user.id': '12345',
  'login.method': 'oauth',
  'client.ip': req.ip,
  'vtex.search_index': 'user_events' // Custom attributes
});

logsClient.warn('Rate limit approaching', {
  'rate_limit.current': 95,
  'rate_limit.max': 100,
  'time_window': '1min'
});

logsClient.error('Database query failed', {
  'query.table': 'users',
  'query.duration_ms': 5000,
  'error.type': 'timeout'
});
```

### Log Correlation with Traces

Logs automatically include trace context when created within an active span:

```javascript
const span = tracesClient.startSpan('process-order');
// Logs will automatically include trace and span IDs
logsClient.info('Processing order', { orderId: '12345' });
span.end();
```

## Resource Discovery

### Automatic Attribute Detection

```javascript
const telemetryClient = await NewTelemetryClient(
  'my-app', 'my-client', 'my-service',
  {
    // Environment variable mappings
    envMappings: [
      { envVar: 'REGION', attributeName: 'cloud.region' },
      { envVar: 'VERSION', attributeName: 'service.version' }
    ],
    
    // Additional static attributes
    additionalAttrs: {
      'service.team': 'platform',
      'service.tier': 'critical'
    },
    
    // Cloud detection
    disableCloudDetect: false,   // Auto-detect AWS/GCP/Azure
    disableK8sDetect: false,     // Auto-detect Kubernetes
    
    // Environment and region
    environment: 'production',
    region: 'us-east-1'
  }
);
```

### Custom Attributes from Files

```json
// Read attributes from JSON file
{
  "customAttributesFile": "./custom-attributes.json"
}
```json
// custom-attributes.json
{
  "environmentVariables": {
    "POD_NAME": "k8s.pod.name",
    "NODE_NAME": "k8s.node.name"
  },
  "attributes": {
    "service.owner": "platform-team",
    "cost.center": "engineering"
  }
}
```

## Export Configuration

### OTLP gRPC Export

```javascript
// Local development (insecure)
const tracesClient = await telemetryClient.newTracesClient({
  exporter: CreateExporter(
    CreateTracesExporterConfig({ 
      endpoint: 'localhost:4317',
      compression: 'gzip',
      timeoutSeconds: 30
    }), 
    'otlp'
  )
});
```javascript
// Production (secure)
const tracesClient = await telemetryClient.newTracesClient({
  exporter: CreateExporter(
    CreateTracesExporterConfig({ 
      endpoint: 'otel-collector.company.com:443',
      compression: 'gzip',
      insecure: false
    }), 
    'otlp'
  )
});
```

### Console Export (Development)

```javascript
const tracesClient = await telemetryClient.newTracesClient({
  exporter: CreateExporter(
    CreateTracesExporterConfig({ endpoint: 'stdout' }), 
    'stdout'
  )
});
```

## Advanced Configuration

### Configuration Providers

```javascript
// File Provider (local development)
import { ConfigManager, FileConfigProvider } from '@vtex/diagnostics-nodejs';

const configManager = new ConfigManager('my-app', 'my-service', {
  configPath: './config/telemetry.json',
  enableHotReload: true
});
```javascript
// S3 Provider (production)
import { S3ConfigProvider } from '@vtex/diagnostics-nodejs';

const s3Provider = new S3ConfigProvider(
  'telemetry-configs-bucket',
  'services/my-service/config.json', 
  'us-east-1',
  30000  // Poll interval
);
```javascript
// Custom provider implementation
class CustomConfigProvider {
  async load() {
    // Fetch from your custom source (API, database, etc.)
    return {
      traces: {
        sampling: {
          defaultRate: await this.getSamplingRateFromAPI()
        }
      }
    };
  }
  
  onChange(callback) {
    // Subscribe to config changes
    this.webhookSubscriber.on('config-changed', callback);
  }
}
```

### Environment Variable Mappings

```javascript
const telemetryClient = await NewTelemetryClient(
  'my-app', 'my-client', 'my-service',
  {
    // Map environment variables to resource attributes
    envMappings: [
      { envVar: 'REGION', attributeName: 'cloud.region' },
      { envVar: 'VERSION', attributeName: 'service.version' },
      { envVar: 'TEAM', attributeName: 'service.team' },
      { envVar: 'POD_NAME', attributeName: 'k8s.pod.name' },
      { envVar: 'NODE_NAME', attributeName: 'k8s.node.name' }
    ],
    
    // Prefix-based mapping (maps all TELEMETRY_ATTR_* vars)
    envPrefix: 'TELEMETRY_ATTR_',
    disableEnvPrefixDetection: false,
    
    // Custom attributes file
    customAttributesFile: './custom-attributes.json'
  }
);

// Environment variables automatically mapped:
// REGION=us-east-1 → cloud.region=us-east-1
// VERSION=1.2.3 → service.version=1.2.3
// TELEMETRY_ATTR_COST_CENTER=eng → cost.center=eng
```

```javascript
const telemetryClient = await NewTelemetryClient(
  'my-app', 'my-client', 'my-service',
  {
    config: {
      s3Config: {
        bucket: 'my-telemetry-configs',
        key: 'services/my-service/config.json',
        region: 'us-east-1'
      },
      pollIntervalMs: 30000
    }
  }
);
```

## Advanced Features

### Configuration Validation & Error Handling

```javascript
const telemetryClient = await NewTelemetryClient(
  'my-app', 'my-client', 'my-service',
  {
    config: {
      configPath: './config/telemetry.json',
      enableHotReload: true,
      
      // Validation options
      validateConfig: true,
      fallbackToDefaults: true,  // Use defaults if config is invalid
      
      // Error handling
      onConfigError: (error) => {
        console.error('Config error:', error);
        // Send alert, fallback to safe defaults, etc.
      },
      
      onConfigChange: (newConfig, oldConfig) => {
        console.log('Config changed:', {
          oldSamplingRate: oldConfig.traces?.sampling?.defaultRate,
          newSamplingRate: newConfig.traces?.sampling?.defaultRate
        });
      }
    }
  }
);
```

### Feature Flags Integration

```json
// Dynamic feature flags with configuration
{
  "features": {
    "enableAdvancedSampling": true,
    "enableCustomProcessors": true,
    "enableBaggagePropagation": true
  },
  "traces": {
    "sampling": {
      "defaultRate": "{{features.enableAdvancedSampling ? 0.1 : 0.01}}",
      "rules": [
        {
          "name": "feature_flag_sampling",
          "sampleRate": 1.0,
          "enabled": "{{features.enableAdvancedSampling}}",
          "conditions": [
            { "attribute": "feature.flag", "operator": "Equals", "value": "new_checkout" }
          ]
        }
      ]
    }
  }
}
```

### Multi-Environment Configuration

```json
// config/base.json (shared configuration)
{
  "globalResourceAttributes": {
    "service.namespace": "ecommerce"
  },
  "traces": {
    "exporters": {
      "mode": "otlp"
    }
  }
}
```json
// config/production.json (production overrides)
{
  "extends": "./base.json",
  "globalResourceAttributes": {
    "deployment.environment": "production"
  },
  "traces": {
    "exporters": {
      "otlp": { "endpoint": "prod-collector:4317" }
    },
    "sampling": {
      "defaultRate": 0.01
    }
  }
}
```json
// config/development.json (development overrides)
{
  "extends": "./base.json", 
  "globalResourceAttributes": {
    "deployment.environment": "development"
  },
  "traces": {
    "exporters": {
      "otlp": { "endpoint": "localhost:4317" }
    },
    "sampling": {
      "defaultRate": 1.0
    }
  }
}
```

### Circuit Breaker Configuration

```json
// Automatic fallback when OTLP collector is down
{
  "traces": {
    "exporters": {
      "mode": "otlp",
      "otlp": {
        "endpoint": "otel-collector:4317",
        "timeoutMs": 5000,
        "maxRetries": 3
      },
      "fallback": {
        "mode": "console",
        "enabled": true
      }
    },
    "circuitBreaker": {
      "enabled": true,
      "failureThreshold": 5,
      "recoveryTimeMs": 30000
    }
  }
}
```

```javascript
import { Processors } from '@vtex/diagnostics-nodejs';

const telemetryClient = await NewTelemetryClient(...);
const tracesClient = await telemetryClient.newTracesClient({
  // VTEX-specific attribute processor (enabled by default)
  enableVtexProcessor: true,
  vtexProcessorConfig: {
    customAttributes: {
      'vtex.workspace': process.env.VTEX_WORKSPACE
    }
  },
  
  // Baggage processor (enabled by default)
  enableBaggageProcessor: true
});
```

### Performance & Resource Management

```json
// Fine-tune performance settings
{
  "traces": {
    "batchTimeoutMs": 5000,
    "maxBatchSize": 512,
    "maxQueueSize": 2048,
    "useSimpleProcessor": false
  },
  "metrics": {
    "exportIntervalMs": 30000,
    "maxMetricsPoints": 1000,
    "cardinalityLimit": 1000
  },
  "logs": {
    "batchTimeoutMs": 1000,
    "maxBatchSize": 100,
    "maxQueueSize": 500
  }
}
```

### Configuration Security

```json
// Secure configuration with encrypted values
{
  "traces": {
    "exporters": {
      "otlp": {
        "endpoint": "{{decrypt:aws-kms:endpoint}}",
        "headers": {
          "authorization": "{{env:OTEL_API_KEY}}"
        }
      }
    }
  },
  "encryption": {
    "provider": "aws-kms",
    "keyId": "arn:aws:kms:us-east-1:123456789:key/12345678"
  }
}
```javascript
// Runtime decryption
const telemetryClient = await NewTelemetryClient(
  'my-app', 'my-client', 'my-service',
  {
    config: {
      configPath: './config/encrypted.json',
      decryptionProvider: new AWSKMSDecryption(),
      enableConfigValidation: true
    }
  }
);
```

### Custom Processors

```javascript
import { Processors } from '@vtex/diagnostics-nodejs';

const telemetryClient = await NewTelemetryClient(...);
const tracesClient = await telemetryClient.newTracesClient({
  // Built-in VTEX processor (enabled by default)
  enableVtexProcessor: true,
  vtexProcessorConfig: {
    customAttributes: {
      'vtex.workspace': process.env.VTEX_WORKSPACE,
      'vtex.account': process.env.VTEX_ACCOUNT
    }
  },
  
  // Baggage processor (enabled by default)
  enableBaggageProcessor: true,
  
  // Custom span processor
  customProcessors: [
    new MyCustomSpanProcessor({
      addTimestamp: true,
      enrichWithBusinessContext: true
    })
  ]
});
```

```javascript
// Before: Manual OpenTelemetry setup
const tracer = trace.getTracer('my-service');
const meter = metrics.getMeter('my-service');

// After: Diagnostics library
const telemetryClient = await NewTelemetryClient('app', 'client', 'service');

const instrumentations = Instrumentation.CommonInstrumentations.autoDetect();
const tracesClient = await telemetryClient.newTracesClient();
const metricsClient = await telemetryClient.newMetricsClient();

telemetryClient.registerInstrumentations(instrumentations);
```

## Troubleshooting

### Common Issues

1. **No traces generated**: Ensure telemetry client is initialized and instrumentations are registered *before* importing frameworks.
2. **Sampling not working**: Check `http.target` attribute in rules (not `http.route`).  
3. **Config not reloading**: Verify file permissions and polling interval.
4. **gRPC connection failed**: Use correct port (4317 for insecure, 443 for TLS).
5. **S3 config not loading**: Check AWS credentials and bucket permissions.
6. **High memory usage**: Reduce batch sizes and queue limits in the configuration.
7. **Missing baggage**: Ensure context propagation middleware is installed and used correctly.

### Debug & Monitoring

```javascript
// Enable comprehensive debugging
const telemetryClient = await NewTelemetryClient(
  'app', 'client', 'service',
  { 
    debug: true,  // OpenTelemetry debug logs
    config: {
      enableHotReload: true,
      onConfigError: (error) => console.error('Config error:', error),
      onConfigChange: (newConfig) => console.log('Config updated:', newConfig)
    }
  }
);

// The ConfigManager is internal and not exposed via a public `getConfigManager()` method.
// To monitor config, use the `onConfigChange` callback in the config options.

// Check sampling decisions by enabling debug logs or creating a custom span processor.
```

### Performance Monitoring

```javascript
// Monitor telemetry overhead
const telemetryMetrics = metricsClient.createHistogram('telemetry_overhead_ms');
const configReloadCounter = metricsClient.createCounter('config_reloads_total');

// The onConfigChange callback can be used to increment the counter.
// Example:
// onConfigChange: (newConfig) => {
//   configReloadCounter.increment();
// }

// Monitoring export success/failure would require custom exporters or processors.
```

### Debug Logging

```javascript
const telemetryClient = await NewTelemetryClient(
  'app', 'client', 'service',
  { debug: true }  // Enables OpenTelemetry debug logs
);
```

## Examples & Use Cases

### Microservices with Remote Configuration
```bash
# Centralized configuration management
my-microservices/
├── services/
│   ├── api-gateway/     # parentBased: true, high sampling
│   ├── user-service/    # parentBased: true, follows gateway  
│   ├── inventory/       # parentBased: false, resamples
│   └── payment/         # parentBased: true, 100% sampling
├── configs/
│   ├── s3-bucket: telemetry-configs
│   ├── api-gateway.json
│   ├── user-service.json  
│   ├── inventory.json
│   └── payment.json
└── shared/
    └── base-config.json
```

### A/B Testing with Configuration

```json
// Dynamic sampling based on feature flags
{
  "traces": {
    "sampling": {
      "rules": [
        {
          "name": "new_checkout_flow",
          "sampleRate": 1.0,
          "conditions": [
            { "attribute": "feature.checkout_v2", "operator": "Equals", "value": "true" }
          ]
        },
        {
          "name": "legacy_checkout_flow", 
          "sampleRate": 0.1,
          "conditions": [
            { "attribute": "feature.checkout_v2", "operator": "Equals", "value": "false" }
          ]
        }
      ]
    }
  }
}
```javascript
// Usage in code
app.post('/checkout', (req, res) => {
  const useNewCheckout = featureFlag.isEnabled('checkout_v2', req.user.id);
  
  Traces.addBaggage({
    'feature.checkout_v2': useNewCheckout.toString(),
    'user.segment': req.user.segment
  });
  
  // Automatically sampled based on feature flag
});
```

### Disaster Recovery Configuration

```json
// Multi-region failover
{
  "traces": {
    "exporters": {
      "primary": {
        "mode": "otlp",
        "otlp": { "endpoint": "us-east-1-collector:4317" }
      },
      "secondary": {
        "mode": "otlp", 
        "otlp": { "endpoint": "us-west-2-collector:4317" }
      },
      "fallback": {
        "mode": "console"
      }
    },
    "failover": {
      "enabled": true,
      "healthCheckIntervalMs": 10000,
      "failoverTimeoutMs": 5000
    }
  }
}
```

### Cost Optimization

```json
// Smart sampling for cost control
{
  "traces": {
    "sampling": {
      "defaultRate": 0.001,
      "rules": [
        {
          "name": "errors_always",
          "sampleRate": 1.0,
          "conditions": [
            { "attribute": "http.status_code", "operator": "GreaterThanEqual", "value": "400" }
          ]
        },
        {
          "name": "slow_requests",
          "sampleRate": 1.0,
          "conditions": [
            { "attribute": "http.response_time_ms", "operator": "GreaterThan", "value": "1000" }
          ]
        },
        {
          "name": "business_critical",
          "sampleRate": 0.1,
          "conditions": [
            { "attribute": "http.target", "operator": "Regex", "value": "^/api/(checkout|payment|auth)" }
          ]
        },
        {
          "name": "high_value_users",
          "sampleRate": 0.5,
          "conditions": [
            { "attribute": "user.tier", "operator": "Equals", "value": "premium" }
          ]
        }
      ]
    }
  }
}
```

## Migration from Basic OpenTelemetry

- [Express with Auto-Instrumentation](../../../examples/javascript/nodejs/nodejs-server/)
- [Microservices with Distributed Tracing](../../../examples/javascript/nodejs/nodejs-server/)
- [Custom Sampling Rules](../../../examples/javascript/nodejs/nodejs-server-with-remote-config/)
- [Configuration Hot-Reload](../../../examples/javascript/nodejs/nodejs-server-with-remote-config/)
---

**VTEX Proprietary Software** - This library is proprietary to VTEX and licensed for internal use only.