@delta-base/observability/README.md

Observability framework for delta-base applications

# Observability Framework

A comprehensive, configurable observability framework built on OpenTelemetry with semantic wide events, automatic middleware, intelligent error categorization, and utility functions for common patterns.

## Installation

```bash
npm install @delta-base/observability
```

## Quick Start

```typescript
import { observabilityMiddleware } from '@delta-base/observability';
import { BUILD_INFO } from './build-info'; // Generated by collect-build-info script

const app = new OpenAPIHono();

// Add observability middleware - automatically handles tracing, HTTP tracking, 
// intelligent error detection, and build info initialization
app.use('*', observabilityMiddleware({
  buildInfo: BUILD_INFO // Automatically initializes build info
}));

// Your routes now have automatic observability with smart error categorization
app.post('/users', async (c) => {
  const tracker = c.get('wideEvent');
  
  // Set operation and track user data being created
  tracker
    .setOperation('user.create', { domain: 'user-management', layer: 'api' })
    .trackUser({
      email: 'john@example.com',
      actorType: 'public'
    });
  
  // Your business logic here
  const result = await createUser(userData);
  
  // Track the created entity
  tracker.trackEntity({
    type: 'user',
    id: result.id
  });
  
  return c.json(result, 201);
});
```

> 💡 **Tip**: Add `"prebuild": "collect-build-info --package-json-dir . --output-dir src"` to your `package.json` scripts to automatically generate build info before each build.

## Configuration

The framework now supports comprehensive configuration to customize behavior for different domains and use cases.

### Simple Configuration

```typescript
import { configureObservability, observabilityMiddleware } from '@delta-base/observability';
import { BUILD_INFO } from './build-info';

// Option 1: Configure globally
configureObservability({
  serviceName: 'my-service',
  autoTrackHttp: true,
  autoTrackErrorResponses: true,
  debug: false
});

// Option 2: Configure via middleware (recommended for build info)
app.use('*', observabilityMiddleware({
  serviceName: 'my-service',
  autoTrackHttp: true,
  autoTrackErrorResponses: true,
  debug: false,
  buildInfo: BUILD_INFO  // Automatically initializes build info
}));
```

### Advanced Configuration

```typescript
import { configureObservabilityAdvanced } from '@delta-base/observability';

// Configure with comprehensive options
configureObservabilityAdvanced({
  tracing: {
    enabled: true,
    serviceName: 'my-service',
    serviceVersion: '1.0.0',
    environment: 'production'
  },
  http: {
    trackRequests: true,
    trackResponses: true,
    trackBodies: false,
    excludeHeaders: ['authorization', 'cookie']
  },
  errors: {
    autoTrackErrorResponses: true,
    autoCategorizeErrors: true,
    defaultUnexpected: true
  },
  performance: {
    trackTiming: true,
    categorizePerformance: true,
    performanceThresholds: {
      fast: 100,
      slow: 1000
    }
  },
  contextSchema: {
    user: ['id', 'email', 'role', 'organization_id'],
    operation: ['type', 'domain', 'layer', 'tenant_id'],
    // ... more categories
  }
});
```

### Build Information Integration

The framework includes comprehensive build information tracking with multiple initialization methods:

#### Method 1: Automatic via Middleware (Recommended)

The simplest approach is to pass build info directly to the middleware:

```typescript
import { observabilityMiddleware } from '@delta-base/observability';
import { BUILD_INFO } from './build-info'; // Generated by collect-build-info script

const app = new OpenAPIHono();

// Automatically initializes build info when middleware is set up
app.use('*', observabilityMiddleware({ 
  buildInfo: BUILD_INFO 
}));

// Your routes now have build info automatically available
app.post('/users', async (c) => {
  const tracker = c.get('wideEvent');
  // Build info is already initialized - no manual setup needed
});
```

#### Method 2: Manual Initialization

If you need more control or want to initialize build info elsewhere:

```typescript
import { initializeBuildInfo } from '@delta-base/observability';

// Initialize build info manually in your application
initializeBuildInfo({
  'service.name': 'my-service',
  'service.version': '1.0.0',
  'service.environment': 'production',
  'git.commit_hash': 'abc123',
  'git.branch': 'main',
  'build.timestamp': new Date().toISOString()
});

// Then use middleware without buildInfo parameter
app.use('*', observabilityMiddleware());
```

#### Method 3: Build Info Collection Script

Generate build info automatically using the CLI tool:

```bash
# Add to your package.json scripts
"prebuild": "collect-build-info --package-json-dir . --output-dir src"
```

This creates a `src/build-info.ts` file with comprehensive build information:

```typescript
// Auto-generated build info - do not edit manually
export const BUILD_INFO = {
  "service.name": "@my-org/my-service",
  "service.version": "1.0.0",
  "service.environment": "production",
  "git.commit_hash": "abc123def456",
  "git.branch": "main",
  "git.commit_message": "feat: add user management",
  "git.commit_author": "developer",
  "build.timestamp": "2024-01-01T12:00:00.000Z",
  "deployment.user": "ci-cd",
  "deployment.trigger": "push"
} as const;
```

Then use it in your middleware:

```typescript
import { BUILD_INFO } from './build-info';

app.use('*', observabilityMiddleware({ buildInfo: BUILD_INFO }));
```

#### Build Info Benefits

With build info properly initialized, your observability data includes:
- **Service identification**: `service.name`, `service.version`, `service.environment`
- **Git context**: `git.commit_hash`, `git.branch`, `git.commit_author`
- **Build context**: `build.timestamp`, `build.id`, `deployment.user`
- **Deployment context**: `deployment.trigger`, `deployment.ci_build_url`

This appears in all traces and logs:

```json
{
  "service.name": "@my-org/my-service",
  "service.version": "1.0.0",
  "git.commit_hash_short": "abc123d",
  "git.branch": "main",
  "operation": "user.create",
  "request_id": "uuid-here",
  // ... other tracking data
}
```

### Context Schema Customization

The framework uses a configurable context schema to determine which fields are tracked and extracted from spans. You can customize this for your domain:

```typescript
import { addContextFields } from '@delta-base/observability';

// Add custom user fields
addContextFields('user', ['role', 'organization_id', 'permissions']);

// Add custom operation fields  
addContextFields('operation', ['tenant_id', 'workspace_id']);

// Add custom infrastructure fields
addContextFields('infrastructure', ['queue.name', 'cache.hit_rate']);
```

### Custom Error Patterns

Extend the automatic error detection with domain-specific patterns:

```typescript
import { addErrorPattern } from '@delta-base/observability';

// Add custom error pattern for your API
addErrorPattern({
  name: 'CustomBusinessError',
  matcher: (body) => 
    body?.error?.type === 'BUSINESS_RULE_VIOLATION' && 
    body?.error?.code,
  tracker: (wideEvent, body) => {
    wideEvent.trackError({
      type: 'BusinessRuleViolation',
      message: body.error.message,
      businessRule: {
        rule: body.error.code,
        violation: body.error.details
      },
      unexpected: false // Expected - business rule violation
    });
  }
});
```

## Core Concepts

### Wide Events

Wide events are context-rich, structured logs that capture comprehensive information about operations. Instead of multiple log lines, you emit a single event per service hop with all relevant context.

### Intelligent Error Categorization

The framework automatically categorizes errors into **expected** and **unexpected** types:

#### Expected Errors (`unexpected: false`)
- **Validation errors** - User submitted invalid data
- **Business rule violations** - Duplicate email, insufficient permissions
- **Authentication/authorization failures** - Wrong password, missing permissions
- **Rate limiting** - User exceeded API limits
- **Resource not found** - User requested non-existent data

#### Unexpected Errors (`unexpected: true`)
- **System failures** - Database down, network issues
- **Programming errors** - Null pointer exceptions, type errors
- **Infrastructure issues** - Service unavailable, timeouts
- **Unhandled edge cases** - Code paths that shouldn't be reached

This categorization enables:
- **Smart alerting** - Only alert on unexpected errors
- **SLA tracking** - Only count unexpected errors against service level agreements
- **User experience monitoring** - Track expected errors separately for UX insights
- **Debugging prioritization** - Unexpected errors need immediate attention

### Semantic Tracking Methods

The framework provides semantic methods for tracking different types of data:

- `setOperation()` - Set operation type and context (domain, layer, etc.)
- `trackUser()` - User and actor information
- `trackEntity()` - Primary resource being operated on
- `trackInfrastructure()` - Technical infrastructure details
- `trackError()` - Error information with automatic categorization
- `recordEvent()` - Business events during the operation

## Configuration API Reference

### ObservabilityConfig

The comprehensive configuration interface for the observability framework.

```typescript
interface ObservabilityConfig {
  contextSchema: ContextSchemaConfig;   // Defines which fields are tracked
  tracing: TracingConfig;               // OpenTelemetry tracing configuration  
  http: HttpTrackingConfig;             // HTTP tracking configuration
  errors: ErrorTrackingConfig;          // Error detection and categorization
  performance: PerformanceConfig;       // Performance tracking and thresholds
  debug: boolean;                       // Debug logging
}
```

### ContextSchemaConfig

Defines which fields are tracked for each category of context data.

```typescript
interface ContextSchemaConfig {
  user: string[];           // User-related fields (id, email, role, etc.)
  entity: string[];         // Entity-related fields (type, id, status, etc.)
  operation: string[];      // Operation-related fields (type, domain, layer, etc.)
  infrastructure: string[]; // Infrastructure fields (database, event_store, etc.)
  error: string[];          // Error-related fields (type, message, unexpected, etc.)
  http: string[];           // HTTP-related fields (method, path, status_code, etc.)
  performance: string[];    // Performance fields (duration_ms, category, etc.)
  service: string[];        // Service-related fields (name, version, environment, etc.)
}
```

### ErrorResponsePattern

Defines a pattern for detecting and categorizing error responses.

```typescript
interface ErrorResponsePattern {
  name: string;                              // Human-readable pattern name
  matcher: (body: any) => boolean;           // Test if response matches pattern
  tracker: (wideEvent: WideEventsWrapper, body: any) => void; // Track the error
}
```

### Configuration Functions

#### observabilityMiddleware(options?: ObservabilityOptions)

Create observability middleware with configuration options.

```typescript
interface ObservabilityOptions {
  /** Custom service name (defaults to environment variable SERVICE_NAME) */
  serviceName?: string;
  
  /** Whether to automatically track HTTP request/response (defaults to true) */
  autoTrackHttp?: boolean;
  
  /** Whether to automatically detect and track error responses (defaults to true) */
  autoTrackErrorResponses?: boolean;
  
  /** Whether to enable debug logging (defaults to false) */
  debug?: boolean;
  
  /** Build info to initialize (optional) */
  buildInfo?: Record<string, string | number | boolean>;
}

// Simple usage
app.use('*', observabilityMiddleware());

// With configuration
app.use('*', observabilityMiddleware({
  serviceName: 'my-service',
  autoTrackHttp: true,
  autoTrackErrorResponses: true,
  debug: false,
  buildInfo: BUILD_INFO
}));
```

#### configureObservability(options: ObservabilityOptions)

Simple configuration for basic use cases.

```typescript
configureObservability({
  serviceName: 'my-service',
  autoTrackHttp: true,
  autoTrackErrorResponses: true,
  debug: false
});
```

#### configureObservabilityAdvanced(config: Partial<ObservabilityConfig>)

Advanced configuration with full control over all options.

```typescript
configureObservabilityAdvanced({
  tracing: {
    serviceName: 'my-service',
    environment: 'production'
  },
  contextSchema: {
    user: ['id', 'email', 'role'],
    operation: ['type', 'domain', 'tenant_id']
  }
});
```

#### addContextFields(category: keyof ContextSchemaConfig, fields: string[])

Add custom fields to a context category.

```typescript
addContextFields('user', ['organization_id', 'permissions']);
addContextFields('operation', ['workspace_id', 'feature_flag']);
```

#### addErrorPattern(pattern: ErrorResponsePattern)

Add a custom error detection pattern.

```typescript
addErrorPattern({
  name: 'RateLimitError',
  matcher: (body) => body?.error?.code === 'RATE_LIMIT_EXCEEDED',
  tracker: (wideEvent, body) => {
    wideEvent.trackError({
      type: 'RateLimitError',
      message: body.error.message,
      unexpected: false // Expected - rate limiting
    });
  }
});
```

## API Reference

### WideEventsWrapper

The main class for tracking operation context.

```typescript
import { WideEventsWrapper } from '@delta-base/observability';

const tracker = new WideEventsWrapper('Create User', 'user.create');

await tracker.execute(async (t) => {
  // Set operation and track different types of context
  t.setOperation('user.create', { domain: 'user-management', layer: 'api' })
   .trackUser({ id: '123', email: 'john@example.com' })
   .trackEntity({ type: 'user', id: '123' });
   
  // Your business logic here
});
```

#### trackUser(userData: UserData)

Track user and actor information.

```typescript
tracker.trackUser({
  id: '123',
  email: 'john@example.com',
  firstName: 'John',
  lastName: 'Doe',
  actorType: 'authenticated_user'
});
```

#### trackEntity(entityData: EntityData)

Track the primary entity/resource being operated on.

```typescript
tracker.trackEntity({
  type: 'user',
  id: 'user-123',
  status: 'active'
});
```

#### setOperation(operationType: string, context?: OperationData)

Set the operation type and context for this operation.

```typescript
// Simple operation
tracker.setOperation('user.create');

// With context
tracker.setOperation('user.create', {
  domain: 'user-management',
  layer: 'api'
});

// With custom properties
tracker.setOperation('user.change-password', {
  domain: 'user-management',
  layer: 'business-logic',
  trigger: 'user-initiated'
});
```

#### trackInfrastructure(infrastructureData: InfrastructureData)

Track infrastructure and technical details.

```typescript
tracker.trackInfrastructure({
  database: {
    operation: 'INSERT',
    table: 'users',
    duration: 45
  },
  eventStore: {
    streamId: 'user-123',
    expectedVersion: 0,
    nextVersion: 1
  },
  externalService: {
    name: 'notification-service',
    operation: 'send-welcome-email',
    duration: 120
  }
});
```

#### trackError(errorData: ErrorData)

Track error information with automatic categorization.

```typescript
// Expected error - validation failure
tracker.trackError({
  type: 'ValidationError',
  message: 'Email is required',
  validation: {
    issueCount: 2,
    fields: ['email', 'firstName']
  },
  unexpected: false // Expected - user submitted invalid data
});

// Expected error - business rule violation
tracker.trackError({
  type: 'BusinessRuleViolation',
  message: 'User already exists',
  businessRule: {
    rule: 'unique_email',
    violation: 'already_exists'
  },
  unexpected: false // Expected - business constraint violation
});

// Unexpected error - system failure
tracker.trackError({
  type: 'DatabaseConnectionError',
  message: 'Connection pool exhausted',
  unexpected: true // Unexpected - infrastructure issue
});
```

#### recordEvent(eventName: string, eventData?: object)

Record business events during the operation.

```typescript
tracker.recordEvent('user.created', {
  userId: '123',
  source: 'api'
});

tracker.recordEvent('validation.completed', {
  validation_type: 'input'
});
```

### Middleware

#### observabilityMiddleware(options?)

Automatic observability middleware that:
- Initializes OpenTelemetry tracing
- Creates a WideEventsWrapper for the request
- Automatically tracks HTTP request/response details
- Intelligently detects and categorizes error responses using pattern matching
- Makes the tracker available via `c.get('wideEvent')`
- Provides simple fallback operation type derivation

```typescript
import { observabilityMiddleware } from '@delta-base/observability';

// Basic usage
app.use('*', observabilityMiddleware());

// With custom options
app.use('*', observabilityMiddleware({
  serviceName: 'my-custom-service',
  autoTrackHttp: true,
  autoTrackErrorResponses: true
}));
```

The middleware includes intelligent error response detection using pattern matching:

```typescript
// Automatic detection of OpenAPIHono validation errors
{
  name: 'OpenAPIHono_ZodValidation',
  matcher: (body) => 
    body?.success === false &&
    body?.error?.name === 'ZodError' &&
    Array.isArray(body?.error?.issues),
  tracker: (wideEvent, body) => {
    wideEvent.trackError({
      type: 'ValidationError',
      message: 'Request validation failed',
      validation: {
        issueCount: body.error.issues.length,
        fields: body.error.issues.map(issue => issue.path.join('.'))
      },
      unexpected: false // Expected - user submitted invalid data
    });
  }
}
```

#### createObservabilityErrorHandler(options?)

Error handler that integrates with observability tracking for exceptions that bypass normal response flow.

```typescript
import { observabilityMiddleware, createObservabilityErrorHandler } from '@delta-base/observability';

const app = new OpenAPIHono();
app.use('*', observabilityMiddleware());
app.onError(createObservabilityErrorHandler());
```

#### extractUserContext(request: Request)

Extract user context from HTTP request.

```typescript
const userContext = extractUserContext(c.req.raw);
if (userContext) {
  tracker.trackUser(userContext);
}
```

### Utility Functions

#### withDomainTracking

Track domain operations with automatic span management.

```typescript
import { withDomainTracking } from '@delta-base/observability';

const result = await withDomainTracking(tracker, 'user', 'create', async () => {
  // Domain logic here
  return await createUser(data);
});
```

#### withEventStoreTracking

Track event store operations with automatic span management.

```typescript
import { withEventStoreTracking } from '@delta-base/observability';

const result = await withEventStoreTracking(tracker, 'user-123', 'append', async () => {
  return await eventStore.append(streamId, events);
});
```

#### withValidationTracking

Track validation operations with automatic error handling and categorization.

```typescript
import { withValidationTracking } from '@delta-base/observability';

const validatedData = await withValidationTracking(tracker, 'input', async () => {
  return await validateUserInput(data);
});
```

#### withExternalServiceTracking

Track external service calls with automatic timing.

```typescript
import { withExternalServiceTracking } from '@delta-base/observability';

const result = await withExternalServiceTracking(tracker, 'user-service', 'create', async () => {
  return await userService.createUser(data);
});
```

#### withDatabaseTracking

Track database operations with automatic timing.

```typescript
import { withDatabaseTracking } from '@delta-base/observability';

const result = await withDatabaseTracking(tracker, 'INSERT', 'users', async () => {
  return await db.insert(userData);
});
```

#### trackUserFromSources

Automatically extract and track user context from various sources.

```typescript
import { trackUserFromSources } from '@delta-base/observability';

trackUserFromSources(tracker, {
  request: c.req.raw, // HTTP request
  body: requestBody,  // Request body
  headers: customHeaders, // Custom headers
});
```

## Build Information System

The framework includes a comprehensive build information system that automatically collects and tracks build metadata:

### Automatic Build Info Collection

```bash
# Use the CLI tool to collect build info
npx collect-build-info --package-json-dir . --output-dir src --service-name my-service

# Or programmatically
import { collectBuildInfo } from '@delta-base/observability';

const buildInfo = collectBuildInfo({
  packageJsonDir: process.cwd(),
  outputDir: 'src',
  defaultServiceName: 'my-service'
});
```

### Build Info in Package.json

Add the build info collection to your package.json:

```json
{
  "scripts": {
    "prebuild": "collect-build-info --package-json-dir . --output-dir src --service-name my-service",
    "build": "npm run prebuild && your-build-command"
  }
}
```

### Build Info Integration

The framework automatically includes build information in all wide events:

```typescript
import { initializeBuildInfo } from '@delta-base/observability';

// Initialize with your build info
initializeBuildInfo({
  'service.name': 'my-service',
  'service.version': '1.0.0',
  'git.commit_hash': 'abc123',
  'git.branch': 'main',
  'build.timestamp': new Date().toISOString()
});
```

## Advanced Usage

### Error Categorization in Practice

The framework automatically categorizes errors, but you can also manually track errors with explicit categorization:

```typescript
const tracker = c.get('wideEvent');

try {
  // Your operation
  await performOperation();
} catch (error) {
  if (error instanceof ValidationError) {
    // Expected error - user input issue
    tracker.trackError({
      type: 'ValidationError',
      message: error.message,
      validation: {
        issueCount: 1,
        fields: ['email']
      },
      unexpected: false // Expected - user submitted invalid data
    });
  } else if (error instanceof DatabaseConnectionError) {
    // Unexpected error - system issue
    tracker.trackError({
      type: 'DatabaseConnectionError',
      message: error.message,
      unexpected: true // Unexpected - infrastructure issue
    });
  }
  throw error;
}
```

### Alerting Based on Error Categories

```typescript
// Example alerting logic
if (error.unexpected === true) {
  await sendAlert({
    severity: 'high',
    message: `Unexpected error in ${operation}`,
    error: error
  });
} else {
  // Expected errors might be tracked for user experience metrics
  await trackUserExperienceMetric({
    type: 'validation_error',
    operation: operation,
    error: error
  });
}
```

### SLA and Error Budget Tracking

```typescript
// Example SLA tracking
const errorBudget = {
  recordRequest: (success: boolean, unexpected: boolean) => {
    if (!success && unexpected) {
      // Only count unexpected errors against SLA
      errorBudget.recordFailure();
    } else {
      errorBudget.recordSuccess();
    }
  }
};

// In your monitoring system
if (event.success === false && event.error?.unexpected === true) {
  slaTracker.recordViolation();
}
```

### Custom Error Response Patterns

You can extend the error response pattern matching by adding custom patterns:

```typescript
// Example of adding custom error patterns
const customErrorPatterns = [
  {
    name: 'CustomBusinessError',
    matcher: (body) => 
      body?.error?.type === 'BUSINESS_RULE_VIOLATION' &&
      body?.error?.code,
    tracker: (wideEvent, body) => {
      wideEvent.trackError({
        type: 'BusinessRuleViolation',
        message: body.error.message,
        businessRule: {
          rule: body.error.code,
          violation: body.error.details
        },
        unexpected: false // Expected - business rule violation
      });
    }
  }
];
```

### Creating Child Spans

```typescript
const validationSpan = tracker.createOperationSpan('validation', {
  'operation.layer': 'validation'
});

try {
  // Validation logic
  validationSpan.setStatus({ code: SpanStatusCode.OK });
} catch (error) {
  validationSpan.recordException(error);
} finally {
  validationSpan.end();
}
```

### Comprehensive Route Example

```typescript
app.post('/users', async (c) => {
  const tracker = c.get('wideEvent'); // From middleware
  const body = c.req.valid('json');
  
  try {
    // Set operation and track user data being created
    tracker
      .setOperation('user.create', {
        domain: 'user-management',
        layer: 'api'
      })
      .trackUser({
        email: body.email,
        firstName: body.firstName,
        lastName: body.lastName,
        actorType: 'public'
      });

    // Use utility for domain tracking
    const result = await withDomainTracking(tracker, 'user', 'create', async () => {
      return await createUserHandler(eventStore, actor, body, tracker);
    });

    // Track created entity and infrastructure
    tracker
      .trackEntity({
        type: 'user',
        id: result.id,
        status: 'created'
      })
      .trackInfrastructure({
        eventStore: {
          streamId: result.id,
          expectedVersion: 0,
          nextVersion: result.nextExpectedStreamVersion,
          createdNewStream: true
        }
      });

    return c.json({
      success: true,
      message: 'User created successfully',
      user: result.user,
      nextExpectedStreamVersion: result.nextExpectedStreamVersion
    }, 201);

  } catch (error) {
    // Error tracking is automatic via middleware pattern matching
    // But you can add specific context if needed
    if (error instanceof IllegalStateError && error.message.includes('already exists')) {
      tracker.trackError({
        type: 'BusinessRuleViolation',
        message: error.message,
        businessRule: {
          rule: 'unique_email',
          violation: 'already_exists'
        },
        unexpected: false // Expected - business constraint violation
      });
    }
    throw error;
  }
});
```

## Output Format

The framework generates structured wide events with the following format:

```json
{
  "request_id": "8bfdf7ecdd485694",
  "timestamp": "2024-09-08T06:14:05.680Z",
  "operation": "user.create",
  "success": true,
  
  "http.method": "POST",
  "http.path": "/users",
  "http.status_code": 201,
  
  "user.email": "john@example.com",
  "user.first_name": "John",
  "user.last_name": "Doe",
  "actor.type": "public",
  
  "entity.type": "user",
  "entity.id": "user-123",
  
  "operation.domain": "user-management",
  "operation.layer": "api",
  
  "event_store.stream_id": "user-123",
  "event_store.expected_version": 0,
  "event_store.next_version": 1,
  "event_store.created_new_stream": true,
  
  "timing.domain_ms": 45,
  "timing.event_store_ms": 67,
  "timing.total_ms": 124,
  
  "performance_category": "fast",
  "error_category": null,
  
  "spans": [
    {
      "name": "validation",
      "duration_ms": 12,
      "status": "ok"
    },
    {
      "name": "domain",
      "duration_ms": 45,
      "status": "ok"
    },
    {
      "name": "event_store",
      "duration_ms": 67,
      "status": "ok"
    }
  ],
  
  "service.name": "my-service",
  "service.version": "1.0.0",
  "git.commit_hash": "abc123",
  "git.branch": "main",
  "build.timestamp": "2024-09-08T06:14:05.680Z"
}
```

### Error Output Format

When errors occur, the framework adds comprehensive error context:

```json
{
  "request_id": "8bfdf7ecdd485694",
  "timestamp": "2024-09-08T06:14:05.680Z",
  "operation": "user.create",
  "success": false,
  
  "http.method": "POST",
  "http.path": "/users",
  "http.status_code": 400,
  
  "error.type": "ValidationError",
  "error.message": "Request validation failed",
  "error.unexpected": false,
  "error.validation.issue_count": 2,
  "error.validation.fields": ["email", "firstName"],
  
  "user.email": "invalid-email",
  "user.first_name": "John",
  "actor.type": "public",
  
  "operation.domain": "user-management",
  "operation.layer": "api",
  
  "timing.total_ms": 12,
  "performance_category": "fast",
  "error_category": "expected",
  
  "service.name": "my-service",
  "service.version": "1.0.0",
  "git.commit_hash": "abc123",
  "git.branch": "main",
  "build.timestamp": "2024-09-08T06:14:05.680Z"
}
```

## Environment Variables

- `SERVICE_NAME` - Name of your service (used in spans and events)
- `SERVICE_VERSION` - Version of your service
- `ENVIRONMENT_NAME` - Environment (development, staging, production)

## Error Response Pattern Matching

The framework uses an extensible pattern matching system to intelligently detect and categorize different types of error responses:

```typescript
interface ErrorResponsePattern {
  name: string;                          // Human-readable name
  matcher: (body: any) => boolean;       // Pattern matching function
  tracker: (wideEvent: WideEventsWrapper, body: any) => void; // Error tracking function
}
```

Built-in patterns include:
- **OpenAPIHono ZodValidation**: Detects Zod validation errors from OpenAPIHono
- **Business Validation**: Detects business rule violations
- **Generic HTTP Errors**: Fallback for unmatched error responses

## Best Practices

### Framework Usage
1. **Use Middleware**: Always use `observabilityMiddleware()` for automatic setup and intelligent error detection
2. **Configure Early**: Configure observability at application startup before registering middleware
3. **Leverage Automatic Categorization**: Trust the framework's error categorization - it properly distinguishes expected vs unexpected errors
4. **Semantic Methods**: Use the semantic tracking methods (`trackUser`, `trackEntity`, etc.) instead of generic setters
5. **Utility Functions**: Use utility functions like `withDomainTracking` for common patterns

### Configuration
6. **Context Schema**: Customize the context schema for your domain - add fields that are important for your use case
7. **Error Patterns**: Add custom error patterns for domain-specific error types
8. **Environment Specific**: Use different configurations for different environments (development, staging, production)
9. **Performance Thresholds**: Adjust performance thresholds based on your service's characteristics

### Error Handling
10. **Error Context**: The middleware automatically adds error context, but you can add specific business context when needed
11. **Business Events**: Record significant business events with `recordEvent()`
12. **Infrastructure Details**: Track infrastructure operations for performance analysis
13. **Alerting Strategy**: Only alert on unexpected errors (`unexpected: true`)
14. **SLA Tracking**: Only count unexpected errors against service level agreements
15. **User Experience**: Track expected errors separately for UX insights

### Development Workflow
16. **Debug Mode**: Enable debug mode in development to see detailed logging
17. **Test Error Patterns**: Test your custom error patterns with different error scenarios
18. **Validate Context**: Regularly review the context fields being tracked to ensure they're useful
19. **Monitor Performance**: Use the performance categorization to identify slow operations

## Extending the Framework

### Creating Custom Tracking Utilities

You can create domain-specific utilities that wrap the framework's functionality:

```typescript
// Custom utility for tracking database operations
export async function withDatabaseQuery<T>(
  tracker: WideEventsWrapper | undefined,
  query: string,
  table: string,
  fn: () => Promise<T>
): Promise<T> {
  if (!tracker) return await fn();
  
  return await withDatabaseTracking(tracker, 'SELECT', table, async () => {
    tracker.recordEvent('database.query.started', { query, table });
    const result = await fn();
    tracker.recordEvent('database.query.completed', { query, table });
    return result;
  });
}

// Custom utility for tracking external API calls
export async function withExternalAPI<T>(
  tracker: WideEventsWrapper | undefined,
  apiName: string,
  endpoint: string,
  fn: () => Promise<T>
): Promise<T> {
  if (!tracker) return await fn();
  
  return await withExternalServiceTracking(tracker, apiName, endpoint, async () => {
    tracker.recordEvent('external_api.call.started', { api: apiName, endpoint });
    const result = await fn();
    tracker.recordEvent('external_api.call.completed', { api: apiName, endpoint });
    return result;
  });
}
```

### Domain-Specific Configuration Presets

Create configuration presets for different types of services:

```typescript
// User management service configuration
export const userServiceConfig = {
  contextSchema: {
    user: ['id', 'email', 'role', 'organization_id'],
    operation: ['type', 'domain', 'layer', 'user_id'],
    infrastructure: ['database.operation', 'cache.hit_rate'],
    error: ['type', 'message', 'unexpected', 'validation_error']
  },
  performance: {
    performanceThresholds: {
      fast: 100,
      slow: 1000
    }
  }
};

// API gateway service configuration  
export const apiGatewayServiceConfig = {
  contextSchema: {
    user: ['id', 'email', 'role', 'organization_id'],
    operation: ['type', 'domain', 'layer', 'route', 'method'],
    infrastructure: ['upstream.service', 'upstream.duration_ms'],
    error: ['type', 'message', 'unexpected', 'auth_failure', 'rate_limit']
  },
  performance: {
    performanceThresholds: {
      fast: 100,
      slow: 2000  // API calls can be slower
    }
  }
};

// Apply preset configuration
configureObservabilityAdvanced(userServiceConfig);
```

## Troubleshooting

### Common Issues

#### Context Fields Not Appearing
```typescript
// Problem: Custom context fields not showing in wide events
// Solution: Ensure fields are added to the context schema
addContextFields('user', ['custom_field']);

// Or check if the field name matches what you're tracking
tracker.trackUser({ customField: 'value' }); // This becomes 'user.custom_field'
```

#### Error Patterns Not Matching
```typescript
// Problem: Custom error patterns not being detected
// Solution: Add debug logging to test your matcher
addErrorPattern({
  name: 'CustomError',
  matcher: (body) => {
    console.log('Testing error pattern:', body); // Debug logging
    return body?.error?.type === 'CUSTOM_ERROR';
  },
  tracker: (wideEvent, body) => {
    // ... tracker logic
  }
});
```

#### Performance Issues
```typescript
// Problem: Observability causing performance degradation
// Solution: Disable expensive features in production
configureObservabilityAdvanced({
  http: {
    trackBodies: false,        // Disable body tracking
    maxBodySize: 1024         // Limit body size
  },
  performance: {
    autoCreateSpans: false     // Disable automatic spans
  }
});
```

### Debug Mode

Enable debug mode to see detailed information about what the framework is doing:

```typescript
configureObservability({
  debug: true
});

// You'll see console output like:
// [Observability] Detected OpenAPIHono_ZodValidation error response, tracking...
// [Observability] Extracted business context: user.id, user.email, operation.type
// [Observability] Wide event emitted: {...}
```

## Summary of Features

This enhanced observability framework provides:

### ✅ **Configurable Context Schema**
- No more hard-coded context fields
- Customize what fields are tracked for your domain
- Add fields incrementally with `addContextFields()`

### ✅ **Pluggable Error Patterns**
- Extend automatic error detection
- Domain-specific error categorization
- Proper expected vs unexpected error handling

### ✅ **Comprehensive Configuration**
- Simple and advanced configuration options
- Environment-specific settings
- Performance tuning options

### ✅ **Build Information Integration**
- Automatic build info collection from git and package.json
- CLI tool for build pipeline integration
- Comprehensive metadata tracking

### ✅ **Developer Experience**
- Clear TypeScript interfaces
- Extensive documentation and examples
- Debug mode for troubleshooting

### ✅ **Production Ready**
- Comprehensive error handling
- Performance optimizations
- Environment-specific configurations

This framework provides a comprehensive, configurable observability solution that can be adapted to any domain while maintaining high-quality error categorization and debugging capabilities.