qerrors
Version:
Intelligent error handling middleware with AI-powered analysis, environment validation, caching, and production-ready logging. Provides OpenAI-based error suggestions, queue management, retry mechanisms, and comprehensive configuration options for Node.js
1 lines • 333 kB
JSON
{"file_contents":{"AGENTS.md":{"content":"# AGENTS.md\n\n## VISION\n\nThe qerrors module represents a paradigm shift from traditional error logging to intelligent error analysis. The core vision is to bridge the gap between error occurrence and resolution by leveraging AI to provide contextual debugging suggestions at the moment errors happen. This transforms error handling from a reactive debugging process into a proactive assistance system.\n\nThe business rationale centers on reducing developer debugging time and improving error resolution speed in production environments. By caching AI-generated advice and implementing queue-based analysis, the module balances the cost of AI API calls with the value of intelligent debugging assistance. The design assumes that most errors in production are repetitive patterns that can benefit from AI analysis, while avoiding infinite loops through careful axios error detection.\n\nThe module's economic model is built around cost-effective AI usage - caching prevents redundant API calls for identical errors, while concurrency limits prevent rate limiting charges. The queue-based approach ensures that expensive AI analysis never blocks critical application responses, maintaining user experience while providing debugging value.\n\nThe architecture prioritizes graceful degradation - the module must never break an application due to its own failures. This defensive approach influences every design decision, from the Promise-based async analysis to the careful separation of response generation and AI processing.\n\n## FUNCTIONALITY\n\nAI agents working on this codebase must understand that qerrors implements a sophisticated error handling middleware that must never cause additional errors. The module is designed to be \"error-safe\" meaning any failure in qerrors itself should fail and simply console.error rather than propagate.\n\nKey agent boundaries:\n- Never implement recursive error handling where qerrors processes its own errors\n- Maintain the promise-based async pattern for AI analysis to prevent blocking application responses\n- Preserve the LRU cache mechanism which is critical for cost control with AI APIs\n- Respect the concurrency limiting system which prevents API rate limit violations\n\nThe AI analysis prompt engineering is specifically tuned for console output readability and practical debugging advice. Agents should not modify the prompt structure without understanding its optimization for avoiding generic responses and formatting issues in log files.\n\nExpected behaviors for agents:\n- Always test error scenarios without API tokens to ensure graceful degradation\n- Verify that Express middleware contracts are maintained (proper next() handling)\n- Ensure content negotiation works for both HTML and JSON responses\n- Test queue overflow scenarios to verify rejection counting works correctly\n- Validate that cache cleanup intervals don't interfere with application performance\n- Confirm that verbose logging can be toggled without breaking functionality\n\n## SCOPE\n\n**In Scope:**\n- Error handling middleware functionality and AI-powered analysis features\n- Logging configuration and transport management\n- Environment variable validation and configuration helpers\n- Caching mechanisms for AI advice and queue management\n- Express.js middleware integration and response handling\n- Test coverage for all error handling scenarios\n\n**Out of Scope:**\n- Frontend error handling or client-side error reporting\n- Database schema changes or data persistence beyond log files\n- Authentication or authorization mechanisms\n- Real-time error monitoring dashboards or alerting systems\n- Integration with external monitoring services beyond OpenAI\n- Performance profiling or application performance monitoring features\n\n**Change Restrictions:**\n- Do not modify the core error handling flow that prevents infinite recursion\n- Do not alter the async analysis pattern that keeps responses fast\n- Do not change the Express middleware signature without extensive testing\n\n## CONSTRAINTS\n\n**Protected Components:**\n- The axios error detection logic in `analyzeError()` function must not be modified without understanding recursion prevention\n- Environment variable defaults in `lib/config.js` require careful consideration as they affect production behavior\n- The LRU cache TTL and cleanup mechanisms are performance-critical and must be preserved\n- Test stubs in `/stubs` directory are dependency replacements and must maintain API compatibility\n- The `postWithRetry()` function's exponential backoff algorithm is tuned for OpenAI API rate limits\n- Socket connection pooling settings in the axios instance are optimized for AI API usage patterns\n- The queue metrics collection system must not be modified without understanding memory implications\n\n**Special Processes:**\n- Any changes to OpenAI API integration require testing with and without API tokens\n- Logging transport configuration changes need verification across different environments\n- Cache limit modifications must consider memory usage implications in production\n- Concurrency limit changes require load testing to avoid rate limiting issues\n\n**Workflow Exceptions:**\n- The module intentionally avoids standard function start/end logging to prevent noise in error scenarios\n- Dependencies are intentionally minimal and should not be expanded without justification\n- The module must remain compatible with Node.js 18+ as specified in package.json\n\n## POLICY\n\n**Module-Specific Policies:**\n- This is an npm module that should remain framework-agnostic while providing Express middleware capabilities\n- All AI API calls must implement retry logic with exponential backoff to handle transient failures\n- Error messages must be developer-friendly and avoid generic phrases that don't aid debugging\n- The module should gracefully handle OpenAI API changes and response format variations\n- Version compatibility must be maintained for Node.js LTS versions (currently 18+)\n- The module must never require external databases or persistent storage beyond file system logs\n- AI prompt engineering should prioritize actionable debugging advice over generic error descriptions\n\n**Testing Requirements:**\n- Every new feature must include tests for both success and failure scenarios\n- API integration tests must work with stubbed responses to avoid external dependencies\n- Cache behavior must be tested for both hit and miss scenarios\n- Queue limits and concurrency controls must be verified under load\n\n**Security Considerations:**\n- Environment variables containing API keys must never be logged or exposed in error messages\n- User-provided error data must be sanitized when generating HTML responses\n- The module must not introduce XSS vulnerabilities through error message display\n\n**Maintenance Standards:**\n- Keep dependencies minimal and prefer Node.js built-in modules where possible\n- Maintain backward compatibility for major version increments\n- Document any breaking changes clearly in changelog and migration guides\n- Performance benchmarks must be maintained for key operations (cache lookups, queue processing)\n- Memory usage patterns should be monitored, especially for long-running applications\n- OpenAI API cost implications must be documented for any changes to prompt structure or frequency","size_bytes":7266},"README.md":{"content":"# qerrors\n\nIntelligent error handling middleware that combines traditional logging with AI-powered debugging assistance. When errors occur, qerrors automatically generates contextual suggestions using OpenAI's GPT models while maintaining fast response times through asynchronous analysis and intelligent caching.\n\n## Environment Variables\n\n\nqerrors reads several environment variables to tune its behavior. A small configuration file in the library sets sensible defaults when these variables are not defined. Only `OPENAI_API_KEY` must be provided manually to enable AI analysis. Obtain your key from [OpenAI](https://openai.com) and set the variable in your environment.\n\nIf `OPENAI_API_KEY` is omitted qerrors still logs errors, but AI-generated advice will be skipped.\n\n**Security Note**: Keep your OpenAI API key secure. Never commit it to version control or expose it in client-side code. Use environment variables or secure configuration management.\n\n**Dependencies**: This package includes production-grade security improvements with the `escape-html` library for safe HTML output.\n\n* `OPENAI_API_KEY` – your OpenAI API key.\n\n* `QERRORS_OPENAI_URL` – OpenAI API endpoint (default `https://api.openai.com/v1/chat/completions`).\n* `QERRORS_CONCURRENCY` – maximum concurrent analyses (default `5`, raise for high traffic, values over `1000` are clamped).\n\n\n* `QERRORS_CACHE_LIMIT` – size of the advice cache (default `50`, set to `0` to disable caching, values over `1000` are clamped).\n* `QERRORS_CACHE_TTL` – seconds before cached advice expires (default `86400`).\n* `QERRORS_QUEUE_LIMIT` – maximum queued analyses before rejecting new ones (default `100`, raise when under heavy load, values over `QERRORS_SAFE_THRESHOLD` are clamped).\n* `QERRORS_SAFE_THRESHOLD` – limit at which `QERRORS_CONCURRENCY` and `QERRORS_QUEUE_LIMIT` are clamped (default `1000`, increase to raise their allowed upper bound).\n\n\n* `QERRORS_RETRY_ATTEMPTS` – attempts when calling OpenAI (default `2`).\n* `QERRORS_RETRY_BASE_MS` – base delay in ms for retries (default `100`).\n* `QERRORS_RETRY_MAX_MS` – cap on retry backoff in ms (default `2000`).\n* `QERRORS_TIMEOUT` – axios request timeout in ms (default `10000`).\n* `QERRORS_MAX_SOCKETS` – maximum sockets per agent (default `50`, increase for high traffic).\n* `QERRORS_MAX_FREE_SOCKETS` – maximum idle sockets per agent (default `256`).\n\n* `QERRORS_MAX_TOKENS` – max tokens for each OpenAI request (default `2048`). Uses GPT-4o model for error analysis.\n\n* `QERRORS_METRIC_INTERVAL_MS` – interval for queue metric logging in milliseconds (default `30000`, set to `0` to disable).\n\n\n* `QERRORS_LOG_MAXSIZE` – logger rotation size in bytes (default `1048576`).\n* `QERRORS_LOG_MAXFILES` – number of rotated log files (default `5`).\n * `QERRORS_LOG_MAX_DAYS` – days to retain daily logs (default `0`). A value of `0` keeps all logs forever and emits a startup warning; set a finite number in production to manage disk usage.\n* `QERRORS_VERBOSE` – control console logging (`true` by default). Set `QERRORS_VERBOSE=false` for production deployments to suppress console output and rely on file logging only.\n* `QERRORS_LOG_DIR` – directory for logger output (default `logs`).\n* `QERRORS_DISABLE_FILE_LOGS` – disable file transports when set.\n* `QERRORS_LOG_LEVEL` – logger output level (default `info`).\n* `QERRORS_SERVICE_NAME` – service name added to logger metadata (default `qerrors`).\n\nFor high traffic scenarios raise `QERRORS_CONCURRENCY`, `QERRORS_QUEUE_LIMIT`, `QERRORS_MAX_SOCKETS`, and `QERRORS_MAX_FREE_SOCKETS`. Set `QERRORS_VERBOSE=false` in production to reduce console overhead and rely on file logging.\n\n\nSet QERRORS_CONCURRENCY to adjust how many analyses run simultaneously;\nif not set the default limit is 5; raise this for high traffic.\n\nUse QERRORS_QUEUE_LIMIT to cap how many analyses can wait in line before rejection;\nif not set the default limit is 100; increase when expecting heavy load.\nThe pending queue uses a double ended queue from the denque package for efficient O(1) dequeues.\n\nWhenever the queue rejects an analysis the module increments an internal counter.\nCheck it with `qerrors.getQueueRejectCount()`.\n\nCall `qerrors.clearAdviceCache()` to manually empty the advice cache.\nUse `qerrors.startAdviceCleanup()` to begin automatic purging of expired entries.\nCall `qerrors.stopAdviceCleanup()` if you need to halt the cleanup interval.\nCall `qerrors.purgeExpiredAdvice()` to run a purge instantly.\nAfter each purge or clear operation the module checks the cache size and stops cleanup when it reaches zero, restarting the interval when new advice is cached.\nCheck the current cache limit with `qerrors.getAdviceCacheLimit()`.\n\nUse `qerrors.getQueueLength()` to monitor how many analyses are waiting.\n\nThe module logs `queueLength` and `queueRejects` at a regular interval (default `30s`). Use `QERRORS_METRIC_INTERVAL_MS` to change the period or set `0` to disable logging. Logging starts with the first queued analysis and stops automatically when no analyses remain.\n\nCall `qerrors.startQueueMetrics()` to manually begin metric logging and `qerrors.stopQueueMetrics()` to halt it when needed.\n\nQERRORS_MAX_SOCKETS lets you limit how many sockets the http agents open;\nif not set the default is 50; raise this to handle high traffic.\nQERRORS_MAX_FREE_SOCKETS caps idle sockets the agents keep for reuse;\nif not set the default is 256 which matches Node's agent default.\nQERRORS_MAX_TOKENS sets the token limit for OpenAI responses;\nif not set the default is 2048 which balances cost and detail.\n\n\n\nThe retry behaviour can be tuned with QERRORS_RETRY_ATTEMPTS, QERRORS_RETRY_BASE_MS and QERRORS_RETRY_MAX_MS which default to 2, 100 and 2000 respectively.\nWhen the API responds with 429 or 503 qerrors uses the `Retry-After` header to wait before retrying; if the header is missing the computed delay is doubled.\n\nYou can optionally set `QERRORS_CACHE_LIMIT` to adjust how many advice entries are cached; set `0` to disable caching (default is 50, values over `1000` are clamped). Use `QERRORS_CACHE_TTL` to control how long each entry stays valid in seconds (default is 86400).\n\nAdditional options control the logger's file rotation:\n\n* `QERRORS_LOG_MAXSIZE` - max log file size in bytes before rotation (default `1048576`)\n* `QERRORS_LOG_MAXFILES` - number of rotated files to keep (default `5`)\n * `QERRORS_LOG_MAX_DAYS` - number of days to keep daily logs (default `0`). A value of `0` retains logs forever and triggers a startup warning; specify a finite number in production to manage disk usage.\n* `QERRORS_LOG_DIR` - path for log files (default `logs`)\n* `QERRORS_DISABLE_FILE_LOGS` - omit file logs when set\n* `QERRORS_SERVICE_NAME` - service name added to logger metadata (default `qerrors`)\n\n\n\n\n## License\n\nISC\n\n## Installation\n\n**Requirements**: Node.js 18 or higher\n\n```bash\nnpm install qerrors\n```\n\n## Usage\n\n### Basic Setup\n\nFirst, set your OpenAI API key:\n```bash\nexport OPENAI_API_KEY=\"your-openai-api-key-here\"\n```\n\nImport the module:\n```javascript\n// Import just qerrors:\nconst {qerrors} = require('qerrors');\n// OR import both qerrors and logger:\nconst { qerrors, logger } = require('qerrors');\nconst log = await logger; //await logger initialization before use\n\n// Import centralized error handling utilities:\nconst { \n qerrors, \n handleControllerError, \n withErrorHandling, \n createTypedError,\n ErrorTypes,\n ErrorSeverity,\n ErrorFactory,\n errorMiddleware\n} = require('qerrors');\n```\n\n## Centralized Error Handling\n\nThe module now includes centralized error handling utilities that provide standardized error classification, severity-based logging, and automated response formatting:\n\n### Error Classification\n\n```javascript\n// Create typed errors with automatic classification\nconst validationError = createTypedError(\n 'Invalid email format',\n ErrorTypes.VALIDATION,\n 'INVALID_EMAIL'\n);\n\nconst dbError = createTypedError(\n 'Connection timeout',\n ErrorTypes.DATABASE,\n 'DB_TIMEOUT'\n);\n\n// Available error types:\nErrorTypes.VALIDATION // 400 - User input errors\nErrorTypes.AUTHENTICATION // 401 - Auth failures \nErrorTypes.AUTHORIZATION // 403 - Permission errors\nErrorTypes.NOT_FOUND // 404 - Resource not found\nErrorTypes.RATE_LIMIT // 429 - Rate limiting\nErrorTypes.NETWORK // 502 - External service errors\nErrorTypes.DATABASE // 500 - Database errors\nErrorTypes.SYSTEM // 500 - Internal system errors\nErrorTypes.CONFIGURATION // 500 - Config/setup errors\n```\n\n### Convenient Error Factory\n\n```javascript\n// Use ErrorFactory for common error scenarios with consistent formatting\nconst validationError = ErrorFactory.validation('Email is required', 'email');\nconst authError = ErrorFactory.authentication('Invalid credentials');\nconst notFoundError = ErrorFactory.notFound('User');\nconst dbError = ErrorFactory.database('Connection failed', 'INSERT');\n\n// All factory methods accept optional context\nconst networkError = ErrorFactory.network(\n 'API timeout', \n 'payment-service', \n { timeout: 5000, retries: 3 }\n);\n```\n\n### Controller Error Handling\n\n```javascript\n// Standardized error handling in Express controllers\napp.get('/api/users/:id', async (req, res) => {\n try {\n const user = await getUserById(req.params.id);\n if (!user) {\n const error = createTypedError(\n 'User not found',\n ErrorTypes.NOT_FOUND,\n 'USER_NOT_FOUND'\n );\n return handleControllerError(res, error, 'getUserById', { userId: req.params.id });\n }\n res.json(user);\n } catch (error) {\n handleControllerError(res, error, 'getUserById', { userId: req.params.id });\n }\n});\n```\n\n### Async Operation Wrapper\n\n```javascript\n// Wrap async operations with automatic error handling\nconst result = await withErrorHandling(\n async () => {\n return await complexAsyncOperation();\n },\n 'complexAsyncOperation',\n { userId: req.user.id },\n { fallback: 'default_value' } // optional fallback\n);\n```\n\n### Severity-Based Logging\n\n```javascript\n// Log errors with appropriate severity levels\nawait logErrorWithSeverity(\n error,\n 'functionName',\n { context: 'additional info' },\n ErrorSeverity.CRITICAL\n);\n\n// Available severity levels:\nErrorSeverity.LOW // Expected errors, user mistakes\nErrorSeverity.MEDIUM // Operational issues, recoverable \nErrorSeverity.HIGH // Service degradation, requires attention\nErrorSeverity.CRITICAL // Service disruption, immediate response needed\n```\n\n### Global Error Middleware\n\n```javascript\n// Add global error handling to your Express app\nconst express = require('express');\nconst app = express();\n\n// Your routes here...\napp.get('/api/users/:id', async (req, res) => {\n const user = await getUserById(req.params.id);\n if (!user) {\n throw ErrorFactory.notFound('User');\n }\n res.json(user);\n});\n\n// Add error middleware as the last middleware\napp.use(errorMiddleware);\n\n// The middleware will automatically:\n// - Log errors with qerrors AI analysis\n// - Send standardized JSON responses\n// - Map error types to appropriate HTTP status codes\n// - Include request context for debugging\n```\n\n## Basic Usage\n\n```javascript\n// Example of using qerrors as Express middleware:\napp.use((err, req, res, next) => {\n qerrors(err, 'RouteName', req, res, next);\n});\n\n// Using qerrors in any catch block:\nfunction doFunction(req, res, next) {\n try {\n //code\n } catch (error) {\n qerrors(error, \"doFunction\", req, res, next); //req res and next are optional\n }\n}\n\n// Response Format: qerrors automatically detects client type\n// - Browser requests (Accept: text/html) receive HTML error pages\n// - API requests receive JSON error responses with structured data\n\n// Example for javascript that is not express related (node / service code / biz logic)\nfunction doFunction(param) {\n try {\n //code\n } catch (error) {\n qerrors(error, \"doFunction\", param);\n }\n}\n\n// ... or if multiple params:\nfunction doFunction(param1, param2) {\n try {\n //code\n } catch (error) {\n qerrors(error, \"doFunction\", {param1, param2}); \n }\n}\n\n// Using the Winston logger directly:\nlog.info('Application started');\nlog.warn('Something might be wrong');\nlog.error('An error occurred', { errorDetails: error });\n// Optional helpers for consistent function logging\nawait logger.logStart('myFunction', {input});\nawait logger.logReturn('myFunction', {result});\n```\n\n### Environment Validation Helpers\n\nUse the optional utilities in `lib/envUtils.js` to verify configuration before starting your application.\n\n```javascript\nconst { throwIfMissingEnvVars, warnIfMissingEnvVars, getMissingEnvVars } = require('qerrors/lib/envUtils');\n\nthrowIfMissingEnvVars(['OPENAI_API_KEY']); // aborts if mandatory variables are missing\nwarnIfMissingEnvVars(['MY_OPTIONAL_VAR']); // logs a warning but continues\nconst missing = getMissingEnvVars(['OPTIONAL_ONE', 'OPTIONAL_TWO']);\n```\n\n\n### Features\n\n- **AI-Powered Analysis**: Automatically generates debugging suggestions using OpenAI GPT-4o model\n- **Express Middleware**: Seamless integration with Express.js applications\n- **Content Negotiation**: Returns HTML pages for browsers, JSON for API clients\n- **Intelligent Caching**: Prevents duplicate API calls for identical errors\n- **Queue Management**: Handles high-traffic scenarios with configurable concurrency limits\n- **Graceful Degradation**: Functions normally even without OpenAI API access\n- **Comprehensive Logging**: Multi-transport Winston logging with file rotation\n\n### Logging\n\nFile transports output JSON objects with timestamps and stack traces. Console\noutput, enabled when `QERRORS_VERBOSE=true`, uses a compact printf format for\nreadability.\n\n### Error Response Formats\n\n**HTML Response** (for browsers):\n```html\n<!DOCTYPE html>\n<html>\n<head><title>Error: 500</title></head>\n<body>\n <h1 class=\"error\">Error: 500</h1>\n <h2>Internal Server Error</h2>\n <pre>Error stack trace...</pre>\n</body>\n</html>\n```\n\n**JSON Response** (for APIs):\n```json\n{\n \"error\": {\n \"uniqueErrorName\": \"ERROR:TypeError_abc123\",\n \"timestamp\": \"2024-01-01T00:00:00.000Z\",\n \"message\": \"Cannot read property 'foo' of undefined\",\n \"statusCode\": 500,\n \"context\": \"userController\",\n \"stack\": \"TypeError: Cannot read property...\"\n }\n}\n```\n\n## Testing\n\nThe test suite uses Node's built-in test runner with custom stubs for offline testing.\nTests include comprehensive coverage of error handling, AI integration, and middleware functionality.\nCurrent test status: 157/157 tests passing (100% success rate).\n\nRun tests from the project directory:\n```bash\nnpm test\n```\n\nUse the dedicated test runner for enhanced output:\n```bash\nnode test-runner.js\n```\n\nOr run tests directly:\n```bash\nnode -r ./setup.js --test test/\n```\n\n**Test Coverage Includes:**\n- Core error handling and middleware functionality\n- OpenAI API integration with mock responses\n- Environment variable validation and configuration\n- Cache management and TTL behavior\n- Queue concurrency and rejection handling\n- Logger configuration across different environments\n\nGitHub Actions runs this test suite automatically on every push and pull request using Node.js LTS. The workflow caches npm dependencies to speed up subsequent runs.\n\n\n","size_bytes":15478},"index.js":{"content":"\n'use strict'; //enforce strict parsing and error handling across module\n\n/**\n * Main entry point for the qerrors package - an intelligent error handling middleware\n * that combines traditional error logging with AI-powered error analysis.\n * \n * This module exports both the core qerrors function and the underlying logger,\n * providing flexibility for different use cases while maintaining a clean API.\n * \n * Design rationale:\n * - Separates concerns by keeping qerrors logic and logging logic in separate modules\n * - Provides both individual exports and a default export for different import patterns\n * - Maintains backward compatibility through multiple export strategies\n * - Uses strict mode to catch common JavaScript pitfalls early\n */\n\nconst qerrors = require('./lib/qerrors'); //load primary error handler implementation\nconst logger = require('./lib/logger'); //load configured winston logger used by qerrors\nconst errorTypes = require('./lib/errorTypes'); //load error classification and handling utilities\nconst sanitization = require('./lib/sanitization'); //load data sanitization utilities\nconst queueManager = require('./lib/queueManager'); //load queue management utilities\nconst utils = require('./lib/utils'); //load common utility functions\nconst config = require('./lib/config'); //load configuration utilities\nconst envUtils = require('./lib/envUtils'); //load environment validation utilities\nconst aiModelManager = require('./lib/aiModelManager'); //load AI model management utilities\n\n/**\n * Error logger middleware that logs errors and provides AI-powered suggestions.\n * @param {Error} error - The error object\n * @param {string} context - Context where the error occurred\n * @param {Object} [req] - Express request object (optional)\n * @param {Object} [res] - Express response object (optional)\n * @param {Function} [next] - Express next function (optional)\n * @returns {Promise<void>}\n */\n\nmodule.exports = { //(primary export object allows destructuring imports like { qerrors, logger, errorTypes } providing clear explicit imports while keeping related functionality grouped)\n qerrors, //(main error handling function users interact with)\n logger, //(winston logger instance for consistent logging, exposes same configured logger qerrors uses internally)\n errorTypes, //(error classification and handling utilities for standardized error management)\n logErrorWithSeverity: qerrors.logErrorWithSeverity, //(severity-based logging function for enhanced error categorization)\n handleControllerError: qerrors.handleControllerError, //(standardized controller error handler with automatic response formatting)\n withErrorHandling: qerrors.withErrorHandling, //(async operation wrapper with integrated error handling)\n createTypedError: errorTypes.createTypedError, //(typed error factory for consistent error classification)\n createStandardError: errorTypes.createStandardError, //(standardized error object factory)\n ErrorTypes: errorTypes.ErrorTypes, //(error type constants for classification)\n ErrorSeverity: errorTypes.ErrorSeverity, //(severity level constants for monitoring)\n ErrorFactory: errorTypes.ErrorFactory, //(convenient error creation utilities for common scenarios)\n errorMiddleware: errorTypes.errorMiddleware, //(Express global error handling middleware)\n handleSimpleError: errorTypes.handleSimpleError, //(simplified error response handler for basic scenarios)\n\n // Enhanced logging utilities with security and performance monitoring\n logDebug: logger.logDebug, //(enhanced debug logging with sanitization)\n logInfo: logger.logInfo, //(enhanced info logging with sanitization)\n logWarn: logger.logWarn, //(enhanced warn logging with performance monitoring)\n logError: logger.logError, //(enhanced error logging with performance monitoring)\n logFatal: logger.logFatal, //(enhanced fatal logging with performance monitoring)\n logAudit: logger.logAudit, //(enhanced audit logging for compliance)\n createPerformanceTimer: logger.createPerformanceTimer, //(performance timer utility for operation monitoring)\n createEnhancedLogEntry: logger.createEnhancedLogEntry, //(enhanced log entry creator with metadata)\n LOG_LEVELS: logger.LOG_LEVELS, //(log level constants with priorities and colors)\n \n // Simple Winston logger for basic logging needs\n simpleLogger: logger.simpleLogger, //(basic Winston logger instance with console output)\n createSimpleWinstonLogger: logger.createSimpleWinstonLogger, //(factory for creating simple Winston loggers)\n\n // Data sanitization utilities for security\n sanitizeMessage: sanitization.sanitizeMessage, //(message sanitization utility for security)\n sanitizeContext: sanitization.sanitizeContext, //(context sanitization utility for security)\n addCustomSanitizationPattern: sanitization.addCustomSanitizationPattern, //(register custom sanitization rules)\n clearCustomSanitizationPatterns: sanitization.clearCustomSanitizationPatterns, //(clear custom patterns for testing)\n sanitizeWithCustomPatterns: sanitization.sanitizeWithCustomPatterns, //(enhanced sanitization with custom rules)\n\n // Queue management and monitoring utilities\n createLimiter: queueManager.createLimiter, //(concurrency limiting utility)\n getQueueLength: queueManager.getQueueLength, //(queue depth monitoring)\n getQueueRejectCount: queueManager.getQueueRejectCount, //(reject count monitoring)\n startQueueMetrics: queueManager.startQueueMetrics, //(start periodic metrics)\n stopQueueMetrics: queueManager.stopQueueMetrics, //(stop periodic metrics)\n\n // Common utility functions\n generateUniqueId: utils.generateUniqueId, //(unique identifier generation)\n createTimer: utils.createTimer, //(performance timing utilities)\n deepClone: utils.deepClone, //(deep object cloning)\n safeRun: utils.safeRun, //(safe function execution wrapper)\n verboseLog: utils.verboseLog, //(conditional verbose logging)\n\n // Configuration and environment utilities\n getEnv: config.getEnv, //(environment variable getter with defaults)\n getInt: config.getInt, //(integer parsing with validation)\n getMissingEnvVars: envUtils.getMissingEnvVars, //(environment validation)\n throwIfMissingEnvVars: envUtils.throwIfMissingEnvVars, //(required environment validation)\n warnIfMissingEnvVars: envUtils.warnIfMissingEnvVars, //(optional environment validation)\n\n // AI model management utilities (LangChain integration)\n getAIModelManager: aiModelManager.getAIModelManager, //(get AI model manager singleton)\n resetAIModelManager: aiModelManager.resetAIModelManager, //(reset AI model manager for testing)\n MODEL_PROVIDERS: aiModelManager.MODEL_PROVIDERS, //(available AI providers)\n createLangChainModel: aiModelManager.createLangChainModel //(create LangChain model instances)\n};\n\nmodule.exports.default = qerrors; //(default export for backward compatibility allowing both 'const qerrors = require(\"qerrors\")' and destructuring patterns, dual strategy accommodates different developer preferences)\n","size_bytes":6983},"replit.md":{"content":"# qerrors - Intelligent Error Handling Middleware\n\n## Overview\n\nqerrors is a Node.js middleware library that combines traditional error logging with AI-powered debugging assistance. It provides intelligent error analysis using OpenAI's GPT models while maintaining production-ready reliability through graceful degradation, caching, and queue management.\n\nThe system is designed with a \"never break the application\" philosophy - all AI features are optional and the middleware will continue functioning even when external services fail.\n\n## System Architecture\n\n### Core Components\n- **Error Handling Middleware**: Express.js compatible middleware for capturing and processing errors\n- **AI Analysis Engine**: OpenAI GPT-4o integration for generating contextual debugging advice\n- **Caching Layer**: LRU cache with TTL for cost-effective AI advice storage\n- **Queue Management**: Concurrency-limited queue system for managing AI analysis requests\n- **Logging System**: Winston-based structured logging with file rotation\n\n### Technology Stack\n- **Runtime**: Node.js 18+\n- **HTTP Client**: Axios with custom retry logic and connection pooling\n- **Caching**: Custom LRU cache with time-to-live support\n- **Queue**: Denque-based double-ended queue for O(1) operations\n- **Logging**: Enhanced Winston with security-aware sanitization, performance monitoring, and structured logging\n- **Security**: HTML escaping for safe error output plus comprehensive data sanitization\n\n## Key Components\n\n### Error Processing Pipeline\n1. **Error Capture**: Middleware intercepts errors from Express applications\n2. **Unique Identification**: Generates crypto-based unique identifiers for error tracking\n3. **Context Analysis**: Extracts and processes error context including stack traces\n4. **Security Sanitization**: Removes sensitive data from logs using pattern-based detection\n5. **AI Analysis**: Queues errors for OpenAI analysis with caching and retry logic\n6. **Enhanced Logging**: Structured logging with performance monitoring and request correlation\n7. **Response Generation**: Returns structured JSON or HTML responses based on Accept headers\n\n### Configuration System\n- **Environment Variables**: 20+ configurable parameters for fine-tuning behavior\n- **Defaults**: Production-ready defaults with conservative resource limits\n- **Validation**: Built-in environment variable validation with helpful error messages\n- **Dynamic Configuration**: Runtime configuration changes without restarts\n\n### Queue and Concurrency Management\n- **Concurrency Limiting**: Configurable concurrent AI analysis requests (default: 5)\n- **Queue Management**: Bounded queue with overflow protection (default: 100 pending)\n- **Metrics**: Built-in queue health monitoring and logging\n- **Backpressure**: Graceful degradation when system is overloaded\n\n## Data Flow\n\n1. **Error Occurrence**: Application error triggers middleware\n2. **Error Enrichment**: Unique ID generation and context extraction\n3. **Immediate Response**: HTTP response sent to client without waiting for AI analysis\n4. **Background Analysis**: Error queued for AI processing with concurrency control\n5. **Cache Check**: System checks for existing advice before API call\n6. **AI Analysis**: OpenAI API call with retry logic and timeout protection\n7. **Cache Storage**: Results stored in LRU cache for future identical errors\n8. **Logging**: Structured logs written to rotating files\n\n## External Dependencies\n\n### Required Services\n- **OpenAI API**: GPT-4o model for error analysis (optional - graceful degradation when unavailable)\n\n### NPM Dependencies\n- **axios**: HTTP client with retry and connection pooling\n- **winston**: Structured logging framework\n- **winston-daily-rotate-file**: Log rotation management\n- **denque**: High-performance queue implementation\n- **lru-cache**: Memory-efficient caching with TTL support\n- **escape-html**: Security-focused HTML escaping\n- **qtests**: Testing utilities for mocking and stubbing\n\n## Deployment Strategy\n\n### Environment Configuration\n- **Development**: Verbose logging enabled, reduced cache sizes, immediate error feedback\n- **Production**: File-only logging, optimized cache settings, queue metrics monitoring\n- **High Traffic**: Increased concurrency limits, larger caches, enhanced connection pooling\n\n### Resource Management\n- **Memory**: LRU cache with configurable limits and TTL-based cleanup\n- **Network**: Connection pooling with configurable socket limits\n- **File System**: Rotating logs with size and time-based retention policies\n- **API Costs**: Intelligent caching prevents redundant OpenAI API calls\n\n### Monitoring and Observability\n- **Queue Metrics**: Periodic logging of queue depth and processing rates\n- **Error Tracking**: Unique error IDs for correlation across logs\n- **Performance Monitoring**: Built-in timing and resource usage tracking\n- **Health Checks**: Environment validation on startup with helpful warnings\n\n## Changelog\n\n```\nChangelog:\n- June 17, 2025. Initial setup\n- June 17, 2025. Enhanced error middleware with meta-error handling, headers protection, and improved fallback responses\n- June 17, 2025. Integrated comprehensive enhanced logging system with security-aware sanitization, performance monitoring, request correlation, and structured logging capabilities\n- August 10, 2025. Successfully migrated from axios-based OpenAI implementation to LangChain architecture supporting multiple AI providers (OpenAI and Google Gemini). Added Gemini 2.5 Flash-lite model support as requested by user. Updated all tests to work with new architecture. Fixed nested object sanitization logic for enhanced logging. All 146 tests now passing.\n- August 11, 2025. Successfully completed qtests module integration with ALL TESTS PASSING! Enhanced testing infrastructure with qtests utilities achieving ~30% reduction in test boilerplate code. Created lib/testUtils.js with QerrorsTestEnv and QerrorsStubbing classes. Implemented conditional qtests setup to avoid winston conflicts. Fixed 24 failing tests by resolving qtests.stubMethod compatibility issues with wrapped functions through hybrid stubbing approach. Added comprehensive analysis documentation (QTESTS_ANALYSIS.md) and demonstration tests. Created test-runner.js at root for unified test execution. Fixed verbose behavior: AI advice now prints to console by default, can be suppressed with QERRORS_VERBOSE=false. Test suite: 157/157 tests passing (100% success rate).\n```\n\n## User Preferences\n\n```\nPreferred communication style: Simple, everyday language.\nTesting policy: Don't report anything as fixed until you have tested it.\nTest suite requirement: Run the full test suite as step 3 and only after that also works report success.\n```","size_bytes":6720},"setup.js":{"content":"\n/**\n * qerrors Setup - Enhanced with qtests Integration\n * \n * This setup file configures module resolution for both qerrors stubs and qtests stubs.\n * It ensures our custom stubs work alongside qtests functionality for comprehensive testing.\n */\n\nconst path = require('path'); // use Node path to build absolute stub directory\nconst Module = require('module'); // access internal module loader to refresh paths\n\n// Configure our custom stubs directory\nconst stubsPath = path.join(__dirname, 'stubs'); // resolve location of dependency stubs\n\n// Note: qtests setup is NOT automatically enabled here due to winston conflicts\n// Instead, individual test files can import qtests utilities as needed\n// This prevents qtests from overriding our production winston configuration\n// while still allowing manual use of qtests utilities in tests\n\n// Configure NODE_PATH to include our custom stubs\nprocess.env.NODE_PATH = process.env.NODE_PATH // prepend stubs directory to module lookup\n ? `${stubsPath}${path.delimiter}${process.env.NODE_PATH}` // keep existing NODE_PATH while prioritizing stubs\n : stubsPath; // when NODE_PATH is empty ensure stubs are still used\n\nModule._initPaths(); // reinitialize resolution cache so Node picks up updated NODE_PATH\n\n","size_bytes":1253},"test_openai_functionality.js":{"content":"/**\n * Test script for OpenAI functionality in qerrors\n * \n * This script creates various types of errors to test the AI-powered\n * error analysis feature and verify it provides helpful debugging suggestions.\n */\n\nconst { qerrors, logError } = require('./index.js');\n\nasync function testOpenAIFunctionality() {\n console.log('=== Testing qerrors OpenAI Functionality ===\\n');\n \n // Test 1: Database connection error simulation\n console.log('1. Testing database connection error...');\n try {\n throw new Error('ECONNREFUSED: Connection refused to database server at localhost:5432');\n } catch (error) {\n error.stack = `Error: ECONNREFUSED: Connection refused to database server at localhost:5432\n at Database.connect (/app/database.js:45:12)\n at UserService.findUser (/app/services/user.js:23:8)\n at AuthController.login (/app/controllers/auth.js:15:5)`;\n \n console.log('Calling qerrors with database error...');\n await qerrors(error, 'Database connection test', { \n operation: 'database_connect',\n host: 'localhost',\n port: 5432,\n database: 'myapp_db'\n });\n }\n \n await new Promise(resolve => setTimeout(resolve, 2000)); // Wait for async processing\n \n // Test 2: Type error simulation\n console.log('\\n2. Testing JavaScript type error...');\n try {\n const user = null;\n user.name.toUpperCase(); // This will throw TypeError\n } catch (error) {\n console.log('Calling qerrors with type error...');\n await qerrors(error, 'User profile processing', {\n operation: 'process_user_profile',\n userId: '12345',\n expectedType: 'object',\n actualValue: null\n });\n }\n \n await new Promise(resolve => setTimeout(resolve, 2000)); // Wait for async processing\n \n // Test 3: API validation error\n console.log('\\n3. Testing API validation error...');\n try {\n const validationError = new Error('Validation failed: email is required, password must be at least 8 characters');\n validationError.name = 'ValidationError';\n validationError.details = {\n email: 'Email field is required',\n password: 'Password must be at least 8 characters long'\n };\n throw validationError;\n } catch (error) {\n console.log('Calling qerrors with validation error...');\n await qerrors(error, 'User registration API', {\n operation: 'user_registration',\n endpoint: '/api/users/register',\n method: 'POST',\n requestData: { email: '', password: '123' }\n });\n }\n \n console.log('\\n=== Test Complete ===');\n console.log('Check the logs directory for detailed AI analysis results.');\n console.log('The AI should provide debugging suggestions for each error type.');\n}\n\n// Run the test\nif (require.main === module) {\n testOpenAIFunctionality().catch(console.error);\n}\n\nmodule.exports = { testOpenAIFunctionality };","size_bytes":3032},"lib/config.js":{"content":"'use strict'; //(enable strict mode for defaults module)\n\n/**\n * Configuration defaults for qerrors module\n * \n * These defaults represent carefully balanced values for production use. Each setting\n * has been chosen based on practical testing and common deployment scenarios.\n * \n * Design rationale:\n * - Conservative defaults prevent resource exhaustion while allowing customization\n * - String values maintain consistency with environment variable parsing\n * - Values scale appropriately for both development and production environments\n * - OpenAI integration defaults balance API cost with functionality\n */\n\nconst defaults = { //default environment variable values for all qerrors configuration options\n QERRORS_CONCURRENCY: '5', //max concurrent analyses\n QERRORS_CACHE_LIMIT: '50', //LRU cache size\n QERRORS_CACHE_TTL: '86400', //seconds each cache entry remains valid //(new default ttl)\n QERRORS_QUEUE_LIMIT: '100', //max waiting analyses before rejecting //(new env default)\n QERRORS_SAFE_THRESHOLD: '1000', //upper limit for concurrency and queue //(new config default)\n\n QERRORS_RETRY_ATTEMPTS: '2', //number of API retries //(renamed env var and updated default)\n QERRORS_RETRY_BASE_MS: '100', //base delay for retries //(renamed env var and updated default)\n QERRORS_RETRY_MAX_MS: '2000', //cap wait time for exponential backoff //(new env default)\n QERRORS_TIMEOUT: '10000', //axios request timeout in ms\n QERRORS_MAX_SOCKETS: '50', //max sockets per http/https agent\n QERRORS_MAX_FREE_SOCKETS: '256', //max idle sockets per agent //(new env default)\n QERRORS_MAX_TOKENS: '2048', //max tokens for openai responses //(new env default)\n QERRORS_OPENAI_URL: 'https://api.openai.com/v1/chat/completions', //endpoint used for analysis //(new openai url default)\n\n QERRORS_LOG_MAXSIZE: String(1024 * 1024), //log file size in bytes\n QERRORS_LOG_MAXFILES: '5', //number of rotated log files\n QERRORS_LOG_MAX_DAYS: '0', //days to retain rotated logs //(0 disables time rotation)\n QERRORS_VERBOSE: 'true', //default on so AI advice prints to console, set to 'false' to suppress\n QERRORS_LOG_DIR: 'logs', //directory for rotated logs\n QERRORS_DISABLE_FILE_LOGS: '', //flag to disable file transports when set\n QERRORS_SERVICE_NAME: 'qerrors', //service identifier for logger //(new default)\n\n QERRORS_LOG_LEVEL: 'info', //logger output severity default\n\n QERRORS_METRIC_INTERVAL_MS: '30000' //interval for queue metrics in ms //(new default)\n\n};\n\n\n\nmodule.exports = defaults; //export defaults for external use\n\nfunction getEnv(name) { //return env var or default when undefined\n return process.env[name] !== undefined ? process.env[name] : defaults[name];\n}\n\nmodule.exports.getEnv = getEnv; //expose getEnv helper\n\nfunction safeRun(name, fn, fallback, info) { //utility wrapper for try/catch //(added helper)\n try { return fn(); } catch (err) { console.error(`${name} failed`, info); return fallback; } //(log and fall back)\n}\n\nmodule.exports.safeRun = safeRun; //export safeRun for env utils //(make accessible)\n\nfunction getInt(name, min = 1) { //parse env integer with minimum enforcement\n const int = parseInt(getEnv(name), 10); //attempt parse\n const defaultVal = typeof defaults[name] === 'number' ? defaults[name] : parseInt(defaults[name], 10); //(handle numeric defaults safely)\n const val = Number.isNaN(int) ? defaultVal : int; //default when NaN\n return val >= min ? val : min; //enforce allowed minimum\n}\n\nmodule.exports.getInt = getInt; //export helper for qerrors usage //(central helper)\n","size_bytes":3540},"lib/envUtils.js":{"content":"\n\n\n/**\n * Core utility for identifying missing environment variables\n * \n * This function serves as the foundation for all environment validation in qerrors.\n * It uses a functional programming approach with Array.filter for clean, readable code\n * that efficiently processes multiple variables in a single pass.\n * \n * Design rationale:\n * - Pure function design enables easy testing and reuse\n * - Filter operation is more readable than manual loop constructs \n * - Returns array format allows flexible handling by calling code\n * - Truthiness check handles both undefined and empty string cases\n * \n * @param {string[]} varArr - Array of environment variable names to check\n * @returns {string[]} Array of missing variable names (empty if all present)\n */\nfunction getMissingEnvVars(varArr) {\n const missingArr = varArr.filter(name => !process.env[name]); //identify missing environment variables using functional filter\n return missingArr; //return filtered array of missing variable names\n}\n\n/**\n * Throws an error if any required environment variables are missing\n * \n * This function implements the \"fail fast\" principle for critical configuration.\n * It's designed for variables that are absolutely required for application function.\n * The thrown error includes all missing variables to help developers fix all issues at once.\n * \n * @param {string[]} varArr - Array of required environment variable names\n * @throws {Error} If any variables are missing, with descriptive message\n * @returns {string[]} Empty array if no variables are missing (for testing purposes)\n */\nfunction throwIfMissingEnvVars(varArr) {\n const missingEnvVars = getMissingEnvVars(varArr); //reuse detection utility\n\n if (missingEnvVars.length > 0) {\n const errorMessage = `Missing required environment variables: ${missingEnvVars.join(', ')}`; //(construct descriptive error message listing all missing vars)\n console.error(errorMessage); //(log prior to throw for immediate visibility)\n const err = new Error(errorMessage); //(create error object with detailed message)\n console.error(err); //(log error instead of calling qerrors to avoid infinite recursion in error handling module)\n throw err; //(propagate failure to stop application startup when critical config missing)\n }\n\n return missingEnvVars; //(return empty array when all required vars present, useful for testing)\n}\n\n/**\n * Logs warnings for missing optional environment variables\n * \n * This function handles variables that enhance functionality but aren't strictly required.\n * It uses console.warn rather than throwing errors to allow graceful degradation.\n * The function is designed to provide helpful feedback without breaking the application.\n * \n * @param {string[]} varArr - Array of optional environment variable names to check\n * @param {string} customMessage - Custom warning message to display (optional)\n * @returns {boolean} True if all variables are present, otherwise false\n */\nfunction warnIfMissingEnvVars(varArr, customMessage = '') {\n const missingEnvVars = getMissingEnvVars(varArr); //reuse detection utility\n\n if (missingEnvVars.length > 0) {\n const warningMessage = customMessage ||\n `Warning: Optional environment variables missing: ${missingEnvVars.join(', ')}. Some features may not work as expected.`; //(construct warning message with fallback default text)\n console.warn(warningMessage); //(log warning for optional vars without breaking application flow)\n }\n\n const result = missingEnvVars.length === 0; //(determine if any vars missing, compute boolean for simpler return type)\n return result; //(inform caller if all vars present, boolean instead of array for cleaner API)\n}\n\nmodule.exports = { //(export environment validation utilities for use across qerrors module)\n getMissingEnvVars, //(core detection function for identifying missing vars)\n throwIfMissingEnvVars, //(fail-fast validation for critical configuration)\n warnIfMissingEnvVars //(graceful degradation validation for optional configuration)\n};\n","size_bytes":4196},"lib/errorTypes.js":{"content":"/**\n * Error classification and standardized handling utilities for