UNPKG

@lsendel/claude-agents

Version:

Supercharge Claude Code with specialized AI sub-agents for code review, testing, debugging, documentation & more. Now with process & standards management! Easy CLI tool to install, manage & create custom AI agents for enhanced development workflow

2,228 lines (1,793 loc) 160 kB
--- name: documentation-standard type: standard version: 1.0.0 description: Comprehensive documentation standards for applications, microservices, modules, and components author: Claude tags: [documentation, standards, readme, docusaurus, automation] related_commands: [/update-docs, /check-docs, /generate-docs] --- # Documentation Standard > Version: 1.0.0 > Last updated: 2025-01-29 > Purpose: Ensure consistent, comprehensive documentation across all levels of the codebase > Automation: Integrated with CI/CD for continuous documentation maintenance ## Overview This standard defines documentation requirements and structures for: - Applications (full stack apps) - Microservices (independent services) - Modules (npm packages, libraries) - Components (UI components, utilities) - Submodules (internal modules) ## Related Processes This standard is used by: - **[analyze-product.md](../processes/analyze-product.md)** - Creating product documentation - **[create-spec.md](../processes/create-spec.md)** - Writing feature specifications - **[feature-development.md](../processes/feature-development.md)** - Updating documentation ## Documentation Hierarchy ``` project-root/ ├── README.md # Project overview ├── docs/ # Docusaurus documentation │ ├── intro.md # Getting started │ ├── architecture/ # System design │ ├── api/ # API documentation │ ├── guides/ # How-to guides │ └── reference/ # Technical reference ├── applications/ │ └── app-name/ │ ├── README.md # App-specific docs │ └── docs/ # Detailed app docs ├── microservices/ │ └── service-name/ │ ├── README.md # Service overview │ ├── API.md # API specification │ └── docs/ # Service docs ├── packages/ │ └── package-name/ │ ├── README.md # Package docs │ └── docs/ # Usage guides └── components/ └── component-name/ ├── README.md # Component docs └── component.stories.mdx # Storybook docs ``` ## Documentation Templates ### 1. Application Documentation Template ```markdown # [Application Name] ![Build Status](https://img.shields.io/badge/build-passing-brightgreen) ![Coverage](https://img.shields.io/badge/coverage-95%25-brightgreen) ![Version](https://img.shields.io/badge/version-1.0.0-blue) ## Overview Brief description of what this application does and its purpose in the system. ## Table of Contents - [Architecture](#architecture) - [Prerequisites](#prerequisites) - [Installation](#installation) - [Configuration](#configuration) - [Development](#development) - [Testing](#testing) - [Deployment](#deployment) - [API Documentation](#api-documentation) - [Troubleshooting](#troubleshooting) - [Contributing](#contributing) ## Architecture ```mermaid graph TD A[Frontend] --> B[API Gateway] B --> C[Auth Service] B --> D[Business Logic] D --> E[Database] ``` ### Tech Stack - **Frontend**: React 18, TypeScript, TailwindCSS - **Backend**: Node.js, Express, PostgreSQL - **Infrastructure**: Docker, Kubernetes, AWS ### Key Features - Feature 1: Description - Feature 2: Description - Feature 3: Description ## Prerequisites - Node.js >= 18.0.0 - PostgreSQL >= 14 - Docker (for containerized development) - AWS CLI (for deployment) ## Installation ### Quick Start ```bash # Clone the repository git clone https://github.com/org/app-name.git cd app-name # Install dependencies npm install # Set up environment cp .env.example .env # Edit .env with your configuration # Run database migrations npm run db:migrate # Start development server npm run dev ``` ### Docker Setup ```bash # Build and run with Docker Compose docker-compose up -d # View logs docker-compose logs -f ``` ## Configuration ### Environment Variables | Variable | Description | Default | Required | |----------|-------------|---------|----------| | `NODE_ENV` | Environment (development/production) | development | Yes | | `DATABASE_URL` | PostgreSQL connection string | - | Yes | | `JWT_SECRET` | Secret for JWT tokens | - | Yes | | `API_KEY` | External API key | - | No | ### Configuration Files - `config/default.json` - Default configuration - `config/production.json` - Production overrides - `config/custom-environment-variables.json` - Environment mapping ## Development ### Project Structure ``` src/ ├── api/ # API routes and controllers ├── services/ # Business logic ├── models/ # Data models ├── utils/ # Utility functions ├── middleware/ # Express middleware ├── config/ # Configuration └── tests/ # Test files ``` ### Development Workflow 1. Create feature branch: `git checkout -b feature/your-feature` 2. Make changes and test locally 3. Run linters: `npm run lint` 4. Run tests: `npm test` 5. Commit with conventional commits: `git commit -m "feat: add new feature"` 6. Push and create PR ### Code Style - ESLint configuration: `.eslintrc.js` - Prettier configuration: `.prettierrc` - Run formatter: `npm run format` ## Testing ### Test Structure ``` tests/ ├── unit/ # Unit tests ├── integration/ # Integration tests ├── e2e/ # End-to-end tests └── fixtures/ # Test data ``` ### Running Tests ```bash # Run all tests npm test # Run unit tests only npm run test:unit # Run with coverage npm run test:coverage # Run e2e tests npm run test:e2e ``` ### Testing Strategy - Unit tests: Jest - Integration tests: Supertest - E2E tests: Cypress - Minimum coverage: 80% ## Deployment ### Environments - **Development**: https://dev.app-name.com - **Staging**: https://staging.app-name.com - **Production**: https://app-name.com ### Deployment Process ```bash # Build for production npm run build # Deploy to staging npm run deploy:staging # Deploy to production (requires approval) npm run deploy:production ``` ### CI/CD Pipeline - GitHub Actions workflow: `.github/workflows/deploy.yml` - Automated tests on PR - Deployment on merge to main ## API Documentation ### Base URL - Production: `https://api.app-name.com/v1` - Staging: `https://staging-api.app-name.com/v1` ### Authentication All API requests require authentication: ```http Authorization: Bearer <token> ``` ### Endpoints See full API documentation at: https://docs.app-name.com/api ### Example Request ```bash curl -X GET https://api.app-name.com/v1/users \ -H "Authorization: Bearer <token>" \ -H "Content-Type: application/json" ``` ## Monitoring ### Health Check - Endpoint: `GET /health` - Metrics: `GET /metrics` ### Logging - Application logs: CloudWatch - Error tracking: Sentry - APM: DataDog ## Troubleshooting ### Common Issues #### Database Connection Error ``` Error: ECONNREFUSED 127.0.0.1:5432 ``` **Solution**: Ensure PostgreSQL is running and DATABASE_URL is correct. #### Port Already in Use ``` Error: listen EADDRINUSE :::3000 ``` **Solution**: Kill the process using the port or change PORT in .env ### Debug Mode ```bash # Run with debug logging DEBUG=app:* npm run dev ``` ## Contributing See [CONTRIBUTING.md](./CONTRIBUTING.md) for contribution guidelines. ## License [MIT License](./LICENSE) ## Support - Documentation: https://docs.app-name.com - Issues: https://github.com/org/app-name/issues - Slack: #app-name-support --- Last updated: 2025-01-29 | Version: 1.0.0 ``` ### 2. Microservice Documentation Template ```markdown # [Service Name] Microservice ![Service Status](https://img.shields.io/badge/status-active-brightgreen) ![API Version](https://img.shields.io/badge/api-v2.0-blue) ![SLA](https://img.shields.io/badge/SLA-99.9%25-brightgreen) ## Overview Purpose and responsibility of this microservice in the system architecture. ## Table of Contents - [Architecture](#architecture) - [API Specification](#api-specification) - [Dependencies](#dependencies) - [Configuration](#configuration) - [Development](#development) - [Testing](#testing) - [Deployment](#deployment) - [Monitoring](#monitoring) - [SLA](#sla) ## Architecture ### Service Context ```mermaid graph LR A[API Gateway] --> B[This Service] B --> C[Database] B --> D[Cache] B --> E[Message Queue] B --> F[Other Service] ``` ### Design Patterns - Pattern: Explanation of why it's used - CQRS: Separate read/write operations - Circuit Breaker: Fault tolerance for external calls ### Data Flow 1. Request received from API Gateway 2. Authentication/Authorization check 3. Business logic processing 4. Data persistence/retrieval 5. Response formatting ## API Specification ### OpenAPI/Swagger Full specification available at: `/swagger` endpoint ### Base Information - **Base URL**: `https://api.example.com/service-name/v2` - **Protocol**: HTTPS only - **Format**: JSON (application/json) - **Authentication**: OAuth 2.0 / JWT ### Endpoints #### GET /resources Retrieve list of resources with pagination **Query Parameters:** - `page` (integer): Page number, default: 1 - `limit` (integer): Items per page, default: 20, max: 100 - `sort` (string): Sort field, default: "created_at" - `order` (string): Sort order (asc/desc), default: "desc" **Response:** ```json { "data": [ { "id": "uuid", "name": "Resource Name", "created_at": "2025-01-29T10:00:00Z" } ], "meta": { "page": 1, "limit": 20, "total": 100, "pages": 5 } } ``` #### POST /resources Create a new resource **Request Body:** ```json { "name": "Resource Name", "description": "Resource Description", "metadata": {} } ``` **Response:** 201 Created ```json { "id": "uuid", "name": "Resource Name", "created_at": "2025-01-29T10:00:00Z" } ``` ### Error Responses ```json { "error": { "code": "VALIDATION_ERROR", "message": "Validation failed", "details": [ { "field": "name", "message": "Name is required" } ] } } ``` ### Rate Limiting - 1000 requests per hour per API key - Headers: `X-RateLimit-Limit`, `X-RateLimit-Remaining`, `X-RateLimit-Reset` ## Dependencies ### External Services | Service | Purpose | SLA | Fallback | |---------|---------|-----|----------| | Auth Service | Token validation | 99.9% | Cache validated tokens | | Database | Data persistence | 99.95% | Read replicas | | Redis | Caching | 99.9% | Direct DB access | | SQS | Message queue | 99.9% | In-memory queue | ### Libraries ```json { "dependencies": { "express": "^4.18.0", "pg": "^8.11.0", "redis": "^4.6.0", "winston": "^3.11.0" } } ``` ## Configuration ### Environment Variables ```bash # Service Configuration SERVICE_NAME=service-name SERVICE_PORT=3001 NODE_ENV=production # Database DATABASE_URL=postgresql://user:pass@host:5432/db DATABASE_POOL_SIZE=20 # Redis REDIS_URL=redis://host:6379 REDIS_TTL=3600 # External Services AUTH_SERVICE_URL=https://auth.example.com AUTH_SERVICE_TIMEOUT=5000 # Monitoring SENTRY_DSN=https://key@sentry.io/project DATADOG_API_KEY=your-key ``` ### Feature Flags ```json { "features": { "new_algorithm": { "enabled": false, "rollout_percentage": 0 }, "enhanced_caching": { "enabled": true, "ttl_seconds": 300 } } } ``` ## Development ### Local Setup ```bash # Install dependencies npm install # Run database migrations npm run migrate # Seed development data npm run seed # Start development server npm run dev ``` ### Docker Development ```bash # Build image docker build -t service-name . # Run with dependencies docker-compose up ``` ### Project Structure ``` src/ ├── api/ # API routes ├── services/ # Business logic ├── repositories/ # Data access ├── models/ # Data models ├── middleware/ # Express middleware ├── utils/ # Utilities ├── config/ # Configuration └── tests/ # Tests ``` ## Testing ### Test Pyramid - Unit Tests: 70% (repositories, services, utils) - Integration Tests: 20% (API endpoints) - Contract Tests: 10% (external services) ### Running Tests ```bash # All tests npm test # Unit tests only npm run test:unit # Integration tests npm run test:integration # Contract tests npm run test:contract # Coverage report npm run test:coverage ``` ### Test Data - Fixtures: `tests/fixtures/` - Factories: `tests/factories/` - Mocks: `tests/mocks/` ## Deployment ### Build Process ```bash # Build Docker image docker build -t service-name:$VERSION . # Push to registry docker push registry.example.com/service-name:$VERSION ``` ### Kubernetes Deployment ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: service-name spec: replicas: 3 selector: matchLabels: app: service-name template: metadata: labels: app: service-name spec: containers: - name: service-name image: registry.example.com/service-name:1.0.0 ports: - containerPort: 3001 env: - name: NODE_ENV value: "production" resources: requests: memory: "256Mi" cpu: "250m" limits: memory: "512Mi" cpu: "500m" ``` ### Rollback Procedure ```bash # List deployments kubectl get deployments # Rollback to previous version kubectl rollout undo deployment/service-name # Rollback to specific version kubectl rollout undo deployment/service-name --to-revision=2 ``` ## Monitoring ### Health Checks - Liveness: `GET /health/live` - Readiness: `GET /health/ready` ### Metrics - Endpoint: `GET /metrics` - Format: Prometheus - Key metrics: - Request rate - Error rate - Response time (p50, p95, p99) - Active connections ### Alerts | Alert | Condition | Action | |-------|-----------|--------| | High Error Rate | 5xx > 1% for 5 min | Page on-call | | High Latency | p95 > 1s for 10 min | Investigate | | Low Availability | Uptime < 99.9% | Incident response | ### Dashboards - Service Overview: https://grafana.example.com/d/service-name - Business Metrics: https://grafana.example.com/d/service-name-business ## SLA ### Availability - Target: 99.9% uptime - Measurement: Successful health checks - Exclusions: Planned maintenance ### Performance - p50 response time: < 100ms - p95 response time: < 500ms - p99 response time: < 1000ms ### Support - Critical issues: 15 min response - High priority: 1 hour response - Normal priority: 1 business day ## Troubleshooting ### Common Issues #### Connection Pool Exhausted **Symptoms**: Timeouts, connection errors **Solution**: Increase `DATABASE_POOL_SIZE` or optimize queries #### Memory Leak **Symptoms**: Increasing memory usage, OOM kills **Solution**: Check for circular references, upgrade dependencies ### Debug Tools ```bash # Enable debug logging DEBUG=service:* npm start # Memory profiling node --inspect src/server.js # Performance profiling node --prof src/server.js ``` ## Migration Guide ### From v1 to v2 1. Update client libraries 2. Update authentication method 3. Migrate from `/api/v1/*` to `/v2/*` 4. Update response parsing for new format ## Contact - Team: #team-service-name - On-call: PagerDuty - Service Name Team - Email: service-name-team@example.com --- Last updated: 2025-01-29 | Version: 2.0.0 | Owners: Team Name ``` ### 3. Module/Package Documentation Template ```markdown # [Package Name] ![npm version](https://img.shields.io/npm/v/package-name.svg) ![downloads](https://img.shields.io/npm/dm/package-name.svg) ![license](https://img.shields.io/npm/l/package-name.svg) ![build status](https://img.shields.io/github/workflow/status/org/package-name/CI) > Brief description of what this package does ## Features - ✅ Feature 1 with brief description - ✅ Feature 2 with brief description - ✅ Feature 3 with brief description - ✅ TypeScript support - ✅ Zero dependencies - ✅ Tree-shakeable ## Table of Contents - [Installation](#installation) - [Quick Start](#quick-start) - [API Reference](#api-reference) - [Examples](#examples) - [Configuration](#configuration) - [TypeScript](#typescript) - [Testing](#testing) - [Contributing](#contributing) - [License](#license) ## Installation ```bash npm install package-name # or yarn add package-name # or pnpm add package-name ``` ## Quick Start ```javascript import { mainFunction } from 'package-name'; // Basic usage const result = mainFunction('input'); console.log(result); // With options const customResult = mainFunction('input', { option1: true, option2: 'value' }); ``` ## API Reference ### mainFunction(input, options?) Main function description. #### Parameters - `input` (string | number): Description of input parameter - `options` (object, optional): Configuration options - `option1` (boolean): Description of option1. Default: `false` - `option2` (string): Description of option2. Default: `'default'` - `callback` (function): Optional callback function #### Returns `Promise<Result>`: Description of return value #### Example ```javascript const result = await mainFunction('test', { option1: true, option2: 'custom' }); ``` ### helperFunction(param) Helper function description. #### Parameters - `param` (any): Description #### Returns `any`: Description ### Class: MainClass Main class description. #### Constructor ```javascript const instance = new MainClass(config); ``` ##### Parameters - `config` (object): Configuration object - `property1` (string): Description - `property2` (number): Description #### Methods ##### method1(arg) Method description. ###### Parameters - `arg` (type): Description ###### Returns `ReturnType`: Description ###### Example ```javascript const result = instance.method1('argument'); ``` ## Examples ### Basic Example ```javascript import { mainFunction } from 'package-name'; async function example() { try { const result = await mainFunction('input'); console.log('Success:', result); } catch (error) { console.error('Error:', error); } } ``` ### Advanced Example ```javascript import { MainClass, helperFunction } from 'package-name'; const instance = new MainClass({ property1: 'value', property2: 42 }); // Using with async/await async function advancedExample() { const processed = helperFunction(rawData); const result = await instance.method1(processed); return result; } // Using with promises function promiseExample() { return instance.method2() .then(result => { console.log('Result:', result); return result; }) .catch(error => { console.error('Error:', error); throw error; }); } ``` ### Real-world Use Cases #### Use Case 1: Data Processing ```javascript import { processData } from 'package-name'; const rawData = [/* ... */]; const processed = processData(rawData, { filter: item => item.active, transform: item => ({ ...item, timestamp: new Date() }), batch: 100 }); ``` #### Use Case 2: Integration Example ```javascript import express from 'express'; import { middleware } from 'package-name'; const app = express(); // Use as middleware app.use(middleware({ option1: true, option2: 'value' })); // Use in route app.get('/api/data', async (req, res) => { const result = await mainFunction(req.query.input); res.json(result); }); ``` ## Configuration ### Global Configuration ```javascript import { configure } from 'package-name'; configure({ globalOption1: 'value', globalOption2: true, debug: process.env.NODE_ENV === 'development' }); ``` ### Environment Variables | Variable | Description | Default | |----------|-------------|---------| | `PACKAGE_NAME_API_KEY` | API key for external service | - | | `PACKAGE_NAME_TIMEOUT` | Request timeout in ms | 5000 | | `PACKAGE_NAME_DEBUG` | Enable debug logging | false | ## TypeScript This package includes TypeScript definitions. ```typescript import { mainFunction, MainFunctionOptions, Result } from 'package-name'; const options: MainFunctionOptions = { option1: true, option2: 'value' }; const result: Result = await mainFunction('input', options); ``` ### Type Definitions ```typescript interface MainFunctionOptions { option1?: boolean; option2?: string; callback?: (error: Error | null, result?: Result) => void; } interface Result { success: boolean; data: any; metadata: { timestamp: Date; version: string; }; } type ProcessFunction = (input: string, options?: MainFunctionOptions) => Promise<Result>; ``` ## Testing ### Running Tests ```bash # Run all tests npm test # Run with coverage npm run test:coverage # Run in watch mode npm run test:watch ``` ### Writing Tests ```javascript import { mainFunction } from 'package-name'; describe('mainFunction', () => { it('should process input correctly', async () => { const result = await mainFunction('test'); expect(result).toEqual({ success: true, data: 'processed test' }); }); it('should handle errors', async () => { await expect(mainFunction(null)).rejects.toThrow('Invalid input'); }); }); ``` ## Benchmarks ```bash # Run benchmarks npm run benchmark ``` Results on MacBook Pro M1: - mainFunction: 1,234,567 ops/sec - helperFunction: 2,345,678 ops/sec ## Migration Guide ### From v1.x to v2.0 1. Update import statements: ```javascript // Before const packageName = require('package-name'); // After import { mainFunction } from 'package-name'; ``` 2. Update API calls: ```javascript // Before packageName.process('input', callback); // After const result = await mainFunction('input'); ``` ## Troubleshooting ### Common Issues #### Issue: "Module not found" **Solution**: Ensure you're using Node.js >= 14 and have installed the package correctly. #### Issue: "Invalid options" **Solution**: Check that your options object matches the expected schema. ## Contributing See [CONTRIBUTING.md](./CONTRIBUTING.md) for details. ## Changelog See [CHANGELOG.md](./CHANGELOG.md) for version history. ## License MIT © [Your Name] ## Related - [related-package-1](https://github.com/org/related-package-1) - Description - [related-package-2](https://github.com/org/related-package-2) - Description --- Made with ❤️ by [Your Name](https://github.com/username) ``` ### 4. Database Documentation Template ```markdown # [Database Name] Database Documentation > Database system for [application/service name] > Engine: PostgreSQL 15.x / MySQL 8.x / MongoDB 6.x ## Overview Brief description of the database purpose, data it stores, and its role in the system. ## Database Information | Property | Value | |----------|-------| | Engine | PostgreSQL 15.3 | | Host | db.example.com | | Port | 5432 | | Database Name | app_production | | Connection Pool | 20-100 connections | | Backup Schedule | Daily at 2 AM UTC | | Retention Policy | 30 days | ## Schema Design ### Entity Relationship Diagram ```mermaid erDiagram users ||--o{ orders : places users ||--o{ addresses : has orders ||--|{ order_items : contains order_items }o--|| products : includes products }o--|| categories : belongs_to users { uuid id PK string email UK string password_hash string first_name string last_name timestamp created_at timestamp updated_at boolean is_active jsonb metadata } orders { uuid id PK uuid user_id FK string status decimal total_amount timestamp ordered_at timestamp shipped_at jsonb shipping_address } products { uuid id PK string sku UK string name text description decimal price integer stock_quantity uuid category_id FK boolean is_active } ``` ### Tables #### users Primary table for user accounts. | Column | Type | Constraints | Description | |--------|------|-------------|-------------| | id | UUID | PRIMARY KEY, DEFAULT gen_random_uuid() | Unique identifier | | email | VARCHAR(255) | UNIQUE, NOT NULL | User email address | | password_hash | VARCHAR(255) | NOT NULL | Bcrypt hashed password | | first_name | VARCHAR(100) | NOT NULL | User's first name | | last_name | VARCHAR(100) | NOT NULL | User's last name | | created_at | TIMESTAMP | NOT NULL, DEFAULT NOW() | Account creation time | | updated_at | TIMESTAMP | NOT NULL, DEFAULT NOW() | Last update time | | is_active | BOOLEAN | NOT NULL, DEFAULT true | Account status | | metadata | JSONB | DEFAULT '{}' | Additional user data | **Indexes:** - `idx_users_email` ON email (unique) - `idx_users_created_at` ON created_at - `idx_users_is_active` ON is_active WHERE is_active = true **Triggers:** - `update_updated_at` - Updates updated_at on row modification #### orders Stores customer orders. | Column | Type | Constraints | Description | |--------|------|-------------|-------------| | id | UUID | PRIMARY KEY | Order identifier | | user_id | UUID | FOREIGN KEY (users.id) | Customer reference | | status | VARCHAR(50) | NOT NULL, CHECK | Order status | | total_amount | DECIMAL(10,2) | NOT NULL, CHECK > 0 | Order total | | ordered_at | TIMESTAMP | NOT NULL, DEFAULT NOW() | Order placement time | | shipped_at | TIMESTAMP | | Shipping time | | shipping_address | JSONB | NOT NULL | Denormalized address | **Indexes:** - `idx_orders_user_id` ON user_id - `idx_orders_status` ON status - `idx_orders_ordered_at` ON ordered_at DESC **Constraints:** - `chk_order_status` CHECK (status IN ('pending', 'processing', 'shipped', 'delivered', 'cancelled')) - `chk_positive_amount` CHECK (total_amount > 0) ### Views #### v_user_order_summary Aggregated view of user order statistics. ```sql CREATE VIEW v_user_order_summary AS SELECT u.id, u.email, COUNT(o.id) as total_orders, SUM(o.total_amount) as lifetime_value, MAX(o.ordered_at) as last_order_date, AVG(o.total_amount) as avg_order_value FROM users u LEFT JOIN orders o ON u.id = o.user_id GROUP BY u.id, u.email; ``` #### v_inventory_status Real-time inventory status view. ```sql CREATE VIEW v_inventory_status AS SELECT p.id, p.sku, p.name, p.stock_quantity, COALESCE(SUM(oi.quantity), 0) as pending_orders, p.stock_quantity - COALESCE(SUM(oi.quantity), 0) as available_stock FROM products p LEFT JOIN order_items oi ON p.id = oi.product_id LEFT JOIN orders o ON oi.order_id = o.id AND o.status IN ('pending', 'processing') GROUP BY p.id; ``` ### Stored Procedures & Functions #### fn_calculate_order_total Calculates order total with tax and shipping. ```sql CREATE OR REPLACE FUNCTION fn_calculate_order_total( p_order_id UUID ) RETURNS DECIMAL(10,2) AS $$ DECLARE v_subtotal DECIMAL(10,2); v_tax_rate DECIMAL(4,3) := 0.08; v_shipping DECIMAL(10,2); BEGIN -- Calculate subtotal SELECT SUM(oi.quantity * p.price) INTO v_subtotal FROM order_items oi JOIN products p ON oi.product_id = p.id WHERE oi.order_id = p_order_id; -- Calculate shipping (simplified) v_shipping := CASE WHEN v_subtotal >= 100 THEN 0 ELSE 10 END; RETURN v_subtotal + (v_subtotal * v_tax_rate) + v_shipping; END; $$ LANGUAGE plpgsql; ``` #### sp_cleanup_inactive_users Cleanup procedure for inactive users. ```sql CREATE OR REPLACE PROCEDURE sp_cleanup_inactive_users( p_days_inactive INTEGER DEFAULT 365 ) AS $$ BEGIN -- Archive users to audit table INSERT INTO users_archive SELECT *, NOW() as archived_at FROM users WHERE is_active = false AND updated_at < NOW() - INTERVAL '1 day' * p_days_inactive; -- Delete from main table DELETE FROM users WHERE is_active = false AND updated_at < NOW() - INTERVAL '1 day' * p_days_inactive; RAISE NOTICE 'Archived % inactive users', ROW_COUNT; END; $$ LANGUAGE plpgsql; ``` ### Triggers #### trg_update_updated_at Auto-update timestamp trigger. ```sql CREATE OR REPLACE FUNCTION fn_update_updated_at() RETURNS TRIGGER AS $$ BEGIN NEW.updated_at = NOW(); RETURN NEW; END; $$ LANGUAGE plpgsql; CREATE TRIGGER trg_update_updated_at BEFORE UPDATE ON users FOR EACH ROW EXECUTE FUNCTION fn_update_updated_at(); ``` #### trg_validate_order_items Validate order items before insert. ```sql CREATE OR REPLACE FUNCTION fn_validate_order_items() RETURNS TRIGGER AS $$ BEGIN -- Check product availability IF NOT EXISTS ( SELECT 1 FROM products WHERE id = NEW.product_id AND is_active = true AND stock_quantity >= NEW.quantity ) THEN RAISE EXCEPTION 'Product % is not available or insufficient stock', NEW.product_id; END IF; RETURN NEW; END; $$ LANGUAGE plpgsql; ``` ## Migration Strategy ### Version Control All database changes are managed through versioned migration files. ``` migrations/ ├── V001__initial_schema.sql ├── V002__add_user_metadata.sql ├── V003__create_order_views.sql ├── V004__add_inventory_tracking.sql └── V005__performance_indexes.sql ``` ### Migration Process ```bash # Run migrations flyway migrate # Validate migrations flyway validate # Rollback last migration flyway undo # Get migration info flyway info ``` ### Rollback Procedures Each migration should have a corresponding rollback script: ```sql -- Migration V003 CREATE VIEW v_user_order_summary AS ...; -- Rollback V003 DROP VIEW IF EXISTS v_user_order_summary; ``` ## Performance Optimization ### Indexing Strategy #### Query Pattern Analysis ```sql -- Most common queries and their indexes -- 1. User lookup by email SELECT * FROM users WHERE email = ?; -- Index: idx_users_email -- 2. Recent orders by user SELECT * FROM orders WHERE user_id = ? ORDER BY ordered_at DESC; -- Index: idx_orders_user_id_ordered_at -- 3. Orders by status SELECT * FROM orders WHERE status = 'pending'; -- Index: idx_orders_status ``` #### Index Maintenance ```sql -- Analyze index usage SELECT schemaname, tablename, indexname, idx_scan, idx_tup_read, idx_tup_fetch FROM pg_stat_user_indexes ORDER BY idx_scan; -- Find missing indexes SELECT schemaname, tablename, attname, n_distinct, most_common_vals FROM pg_stats WHERE n_distinct > 100 AND schemaname = 'public'; ``` ### Query Optimization #### Slow Query Log Analysis ```sql -- Enable slow query logging ALTER SYSTEM SET log_min_duration_statement = 1000; -- Log queries > 1s -- Analyze slow queries SELECT query, calls, total_time, mean_time, stddev_time, rows FROM pg_stat_statements WHERE mean_time > 1000 ORDER BY mean_time DESC LIMIT 20; ``` #### Query Plans ```sql -- Example: Analyzing order search query EXPLAIN (ANALYZE, BUFFERS) SELECT o.*, u.email, u.first_name, u.last_name FROM orders o JOIN users u ON o.user_id = u.id WHERE o.status = 'pending' AND o.ordered_at >= NOW() - INTERVAL '7 days' ORDER BY o.ordered_at DESC LIMIT 100; ``` ### Partitioning Strategy #### Time-based Partitioning for Orders ```sql -- Create partitioned table CREATE TABLE orders_partitioned ( LIKE orders INCLUDING ALL ) PARTITION BY RANGE (ordered_at); -- Create monthly partitions CREATE TABLE orders_2024_01 PARTITION OF orders_partitioned FOR VALUES FROM ('2024-01-01') TO ('2024-02-01'); CREATE TABLE orders_2024_02 PARTITION OF orders_partitioned FOR VALUES FROM ('2024-02-01') TO ('2024-03-01'); -- Automated partition creation CREATE OR REPLACE FUNCTION create_monthly_partition() RETURNS void AS $$ DECLARE start_date date; end_date date; partition_name text; BEGIN start_date := date_trunc('month', CURRENT_DATE); end_date := start_date + interval '1 month'; partition_name := 'orders_' || to_char(start_date, 'YYYY_MM'); EXECUTE format('CREATE TABLE IF NOT EXISTS %I PARTITION OF orders_partitioned FOR VALUES FROM (%L) TO (%L)', partition_name, start_date, end_date); END; $$ LANGUAGE plpgsql; ``` ## Security ### Access Control #### Role-Based Access ```sql -- Application roles CREATE ROLE app_read; CREATE ROLE app_write; CREATE ROLE app_admin; -- Read-only permissions GRANT SELECT ON ALL TABLES IN SCHEMA public TO app_read; GRANT SELECT ON ALL SEQUENCES IN SCHEMA public TO app_read; -- Write permissions GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA public TO app_write; GRANT USAGE ON ALL SEQUENCES IN SCHEMA public TO app_write; -- Admin permissions GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO app_admin; GRANT ALL PRIVILEGES ON ALL SEQUENCES IN SCHEMA public TO app_admin; GRANT CREATE ON SCHEMA public TO app_admin; -- Application users CREATE USER app_api WITH PASSWORD 'secure_password'; GRANT app_write TO app_api; CREATE USER app_analytics WITH PASSWORD 'secure_password'; GRANT app_read TO app_analytics; ``` #### Row-Level Security ```sql -- Enable RLS on sensitive tables ALTER TABLE users ENABLE ROW LEVEL SECURITY; -- Policy for users to see only their own data CREATE POLICY users_isolation ON users FOR ALL TO app_api USING (id = current_setting('app.current_user_id')::uuid); -- Policy for admins to see all data CREATE POLICY admin_all_access ON users FOR ALL TO app_admin USING (true); ``` ### Data Encryption #### At-Rest Encryption ```sql -- Enable transparent data encryption (TDE) -- PostgreSQL: Use filesystem encryption or pgcrypto -- Encrypt sensitive columns CREATE EXTENSION IF NOT EXISTS pgcrypto; -- Example: Encrypting PII ALTER TABLE users ADD COLUMN ssn_encrypted bytea; UPDATE users SET ssn_encrypted = pgp_sym_encrypt(ssn, 'encryption_key') WHERE ssn IS NOT NULL; -- Decrypt when needed SELECT id, pgp_sym_decrypt(ssn_encrypted, 'encryption_key') as ssn FROM users WHERE id = ?; ``` #### In-Transit Encryption ```yaml # PostgreSQL SSL configuration ssl = on ssl_cert_file = 'server.crt' ssl_key_file = 'server.key' ssl_ca_file = 'root.crt' ssl_ciphers = 'HIGH:MEDIUM:+3DES:!aNULL' ssl_prefer_server_ciphers = on ``` ### Audit Trail #### Audit Table Structure ```sql CREATE TABLE audit_log ( id BIGSERIAL PRIMARY KEY, table_name VARCHAR(100) NOT NULL, operation VARCHAR(10) NOT NULL, user_id UUID, changed_at TIMESTAMP NOT NULL DEFAULT NOW(), old_values JSONB, new_values JSONB, query TEXT, ip_address INET ); CREATE INDEX idx_audit_log_table_name ON audit_log(table_name); CREATE INDEX idx_audit_log_user_id ON audit_log(user_id); CREATE INDEX idx_audit_log_changed_at ON audit_log(changed_at); ``` #### Generic Audit Trigger ```sql CREATE OR REPLACE FUNCTION fn_audit_trigger() RETURNS TRIGGER AS $$ BEGIN INSERT INTO audit_log ( table_name, operation, user_id, old_values, new_values, query, ip_address ) VALUES ( TG_TABLE_NAME, TG_OP, current_setting('app.current_user_id', true)::uuid, CASE WHEN TG_OP IN ('UPDATE', 'DELETE') THEN to_jsonb(OLD) END, CASE WHEN TG_OP IN ('INSERT', 'UPDATE') THEN to_jsonb(NEW) END, current_query(), inet_client_addr() ); RETURN CASE WHEN TG_OP = 'DELETE' THEN OLD ELSE NEW END; END; $$ LANGUAGE plpgsql; -- Apply to sensitive tables CREATE TRIGGER trg_audit_users AFTER INSERT OR UPDATE OR DELETE ON users FOR EACH ROW EXECUTE FUNCTION fn_audit_trigger(); ``` ## Backup & Recovery ### Backup Strategy #### Automated Backups ```bash #!/bin/bash # backup.sh - Daily backup script DB_NAME="app_production" BACKUP_DIR="/backups/postgres" TIMESTAMP=$(date +%Y%m%d_%H%M%S) BACKUP_FILE="$BACKUP_DIR/$DB_NAME_$TIMESTAMP.sql.gz" # Full backup with compression pg_dump -h localhost -U postgres -d $DB_NAME | gzip > $BACKUP_FILE # Verify backup if [ $? -eq 0 ]; then echo "Backup successful: $BACKUP_FILE" # Upload to S3 aws s3 cp $BACKUP_FILE s3://backups-bucket/postgres/ else echo "Backup failed" exit 1 fi # Cleanup old backups (keep 30 days) find $BACKUP_DIR -name "*.sql.gz" -mtime +30 -delete ``` #### Point-in-Time Recovery (PITR) ```bash # PostgreSQL configuration for PITR archive_mode = on archive_command = 'test ! -f /archive/%f && cp %p /archive/%f' wal_level = replica max_wal_senders = 3 ``` ### Recovery Procedures #### Full Database Restore ```bash # Restore from backup gunzip < backup_file.sql.gz | psql -U postgres -d $DB_NAME # Restore to specific point in time pg_basebackup -h localhost -D /var/lib/postgresql/data_restore -U replicator -v -P # Recovery configuration restore_command = 'cp /archive/%f %p' recovery_target_time = '2024-01-15 14:30:00' ``` #### Table-Level Recovery ```sql -- Restore single table from backup -- 1. Create temporary database CREATE DATABASE temp_restore; -- 2. Restore backup to temp database -- gunzip < backup.sql.gz | psql -d temp_restore -- 3. Copy table to production INSERT INTO production.users SELECT * FROM temp_restore.users WHERE id NOT IN (SELECT id FROM production.users); -- 4. Cleanup DROP DATABASE temp_restore; ``` ## Monitoring & Maintenance ### Health Checks #### Database Health Queries ```sql -- Connection statistics SELECT datname, numbackends, xact_commit, xact_rollback, blks_read, blks_hit, tup_returned, tup_fetched, tup_inserted, tup_updated, tup_deleted FROM pg_stat_database WHERE datname = current_database(); -- Table bloat check SELECT schemaname, tablename, pg_size_pretty(pg_total_relation_size(schemaname||'.'||tablename)) as size, pg_size_pretty(pg_relation_size(schemaname||'.'||tablename)) as table_size, pg_size_pretty(pg_total_relation_size(schemaname||'.'||tablename) - pg_relation_size(schemaname||'.'||tablename)) as index_size FROM pg_tables WHERE schemaname = 'public' ORDER BY pg_total_relation_size(schemaname||'.'||tablename) DESC; -- Long-running queries SELECT pid, now() - pg_stat_activity.query_start AS duration, query, state FROM pg_stat_activity WHERE (now() - pg_stat_activity.query_start) > interval '5 minutes' AND state != 'idle'; ``` #### Automated Health Check Script ```python # db_health_check.py import psycopg2 import json from datetime import datetime def check_database_health(conn_string): health_report = { "timestamp": datetime.now().isoformat(), "status": "healthy", "checks": {} } try: conn = psycopg2.connect(conn_string) cur = conn.cursor() # Check connection count cur.execute(""" SELECT count(*) FROM pg_stat_activity WHERE state = 'active' """) active_connections = cur.fetchone()[0] health_report["checks"]["active_connections"] = { "value": active_connections, "status": "ok" if active_connections < 80 else "warning" } # Check replication lag cur.execute(""" SELECT client_addr, state, sent_lsn - replay_lsn AS lag_bytes FROM pg_stat_replication """) replication = cur.fetchall() health_report["checks"]["replication"] = { "replicas": len(replication), "max_lag": max([r[2] for r in replication]) if replication else 0 } # Check table bloat cur.execute(""" SELECT COUNT(*) FROM pg_stat_user_tables WHERE n_dead_tup > n_live_tup * 0.2 """) bloated_tables = cur.fetchone()[0] health_report["checks"]["bloated_tables"] = { "count": bloated_tables, "status": "ok" if bloated_tables == 0 else "warning" } except Exception as e: health_report["status"] = "unhealthy" health_report["error"] = str(e) return health_report ``` ### Maintenance Tasks #### Regular Maintenance Schedule ```sql -- Daily tasks VACUUM ANALYZE; -- Update statistics and clean up -- Weekly tasks REINDEX INDEX CONCURRENTLY idx_orders_user_id; -- Rebuild bloated indexes -- Monthly tasks VACUUM FULL large_table; -- Full vacuum for very large tables -- Quarterly tasks CLUSTER orders USING idx_orders_ordered_at; -- Physically reorder table ``` #### Automated Maintenance Script ```bash #!/bin/bash # maintenance.sh # Daily vacuum psql -U postgres -d app_production -c "VACUUM ANALYZE;" # Update table statistics psql -U postgres -d app_production -c "ANALYZE;" # Check for bloated indexes psql -U postgres -d app_production -c " SELECT schemaname, tablename, indexname, pg_size_pretty(pg_relation_size(indexrelid)) AS index_size FROM pg_stat_user_indexes JOIN pg_index ON pg_stat_user_indexes.indexrelid = pg_index.indexrelid WHERE pg_relation_size(indexrelid) > 100000000 -- 100MB ORDER BY pg_relation_size(indexrelid) DESC;" # Archive old audit logs psql -U postgres -d app_production -c " INSERT INTO audit_log_archive SELECT * FROM audit_log WHERE changed_at < NOW() - INTERVAL '90 days'; DELETE FROM audit_log WHERE changed_at < NOW() - INTERVAL '90 days';" ``` ## Troubleshooting ### Common Issues #### Connection Pool Exhaustion **Symptoms:** "FATAL: remaining connection slots are reserved" ```sql -- Check current connections SELECT pid, usename, application_name, client_addr, state, query_start, state_change FROM pg_stat_activity ORDER BY query_start; -- Terminate idle connections SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE state = 'idle' AND state_change < NOW() - INTERVAL '10 minutes'; ``` #### Lock Contention **Symptoms:** Queries hanging, deadlocks ```sql -- View current locks SELECT locktype, database, relation::regclass, mode, granted, pid FROM pg_locks WHERE NOT granted; -- Find blocking queries SELECT blocked_locks.pid AS blocked_pid, blocked_activity.usename AS blocked_user, blocking_locks.pid AS blocking_pid, blocking_activity.usename AS blocking_user, blocked_activity.query AS blocked_statement, blocking_activity.query AS blocking_statement FROM pg_catalog.pg_locks blocked_locks JOIN pg_catalog.pg_stat_activity blocked_activity ON blocked_activity.pid = blocked_locks.pid JOIN pg_catalog.pg_locks blocking_locks ON blocking_locks.locktype = blocked_locks.locktype AND blocking_locks.database IS NOT DISTINCT FROM blocked_locks.database AND blocking_locks.relation IS NOT DISTINCT FROM blocked_locks.relation AND blocking_locks.page IS NOT DISTINCT FROM blocked_locks.page AND blocking_locks.tuple IS NOT DISTINCT FROM blocked_locks.tuple AND blocking_locks.virtualxid IS NOT DISTINCT FROM blocked_locks.virtualxid AND blocking_locks.transactionid IS NOT DISTINCT FROM blocked_locks.transactionid AND blocking_locks.classid IS NOT DISTINCT FROM blocked_locks.classid AND blocking_locks.objid IS NOT DISTINCT FROM blocked_locks.objid AND blocking_locks.objsubid IS NOT DISTINCT FROM blocked_locks.objsubid AND blocking_locks.pid != blocked_locks.pid JOIN pg_catalog.pg_stat_activity blocking_activity ON blocking_activity.pid = blocking_locks.pid WHERE NOT blocked_locks.granted; ``` #### Performance Degradation **Symptoms:** Slow queries, high CPU usage ```sql -- Reset statistics SELECT pg_stat_reset(); -- Wait for some activity, then analyze SELECT schemaname, tablename, n_tup_ins, n_tup_upd, n_tup_del, n_live_tup, n_dead_tup, last_vacuum, last_autovacuum FROM pg_stat_user_tables WHERE n_dead_tup > 1000 ORDER BY n_dead_tup DESC; -- Force analyze on specific table ANALYZE verbose users; ``` ## Development Guidelines ### Naming Conventions #### Tables - Use plural, lowercase, snake_case: `users`, `order_items` - Junction tables: `users_roles`, `products_categories` #### Columns - Primary keys: `id` (UUID preferred) - Foreign keys: `<table>_id` (e.g., `user_id`, `product_id`) - Timestamps: `created_at`, `updated_at`, `deleted_at` - Booleans: `is_<state>` or `has_<feature>` (e.g., `is_active`, `has_premium`) #### Indexes - Primary key: `pk_<table>` - Foreign key: `fk_<table>_<column>` - Other indexes: `idx_<table>_<column(s)>` - Unique constraint: `uq_<table>_<column(s)>` #### Constraints - Check: `chk_<table>_<description>` - Not null: Defined inline with column ### SQL Style Guide ```sql -- Good example SELECT u.id, u.email, u.first_name, COUNT(o.id) AS order_count, SUM(o.total_amount) AS total_spent FROM users u LEFT JOIN orders o ON u.id = o.user_id WHERE u.created_at >= '2024-01-01' AND u.is_active = true GROUP BY u.id, u.email, u.first_name HAVING COUNT(o.id) > 0 ORDER BY total_spent DESC LIMIT 100; -- Use CTEs for complex queries WITH active_users AS ( SELECT id, email FROM users WHERE is_active = true AND created_at >= NOW() - INTERVAL '30 days' ), user_orders AS ( SELECT user_id, COUNT(*) as order_count, SUM(total_amount) as total_spent FROM orders WHERE status != 'cancelled' GROUP BY user_id ) SELECT au.email, COALESCE(uo.order_count, 0) as orders, COALESCE(uo.total_spent, 0) as revenue FROM active_users au LEFT JOIN user_orders uo ON au.id = uo.user_id ORDER BY revenue DESC; ``` ### Change Management #### Migration Best Practices ```sql -- Always use transactions for DDL changes BEGIN; -- Add column with default (safe) ALTER TABLE users ADD COLUMN preference JSONB DEFAULT '{}'; -- Add NOT NULL constraint (requires backfill) UPDATE users SET preference = '{}' WHERE preference IS NULL; ALTER TABLE users ALTER COLUMN preference SET NOT NULL; -- Create index concurrently (non-blocking) CREATE INDEX CONCURRENTLY idx_users_preference ON users USING gin(preference); COMMIT; -- For large tables, batch updates DO $$ DECLARE batch_size INTEGER := 1000; offset_val INTEGER := 0; BEGIN LOOP UPDATE users SET preference = '{}' WHERE id IN ( SELECT id FROM users WHERE preference IS NULL LIMIT batch_size OFFSET offset_val ); EXIT WHEN NOT FOUND; offset_val := offset_val + batch_size; -- Prevent lock buildup PERFORM pg_sleep(0.1); END LOOP; END $$; ``` ## Integration ### Connection Pooling #### PgBouncer Configuration ```ini [databases] app_production = host=localhost port=5432 dbname=app_production [pgbouncer] listen_port = 6432 listen_addr = * auth_type = md5 auth_file = /etc/pgbouncer/userlist.txt pool_mode = transaction max_client_conn = 1000 default_pool_size = 25 reserve_pool_size = 5 reserve_pool_timeout = 3 server_lifetime = 3600 server_idle_timeout = 600 ``` #### Application Configuration ```yaml # Node.js with pg-pool database: host: localhost port: 6432 # PgBouncer port database: app_production user: app_user password: ${DB_PASSWORD} max: 20 # Connection pool size idleTimeoutMillis: 30000 connectionTimeoutMillis: 2000 ``` ### ORM Configuration #### TypeORM Example ```typescript // ormconfig.ts export default { type: 'postgres', host: process.env.DB_HOST, port: parseInt(process.env.DB_PORT), username: process.env.DB_USER, password: process.env.DB_PASSWORD, database: process.env.DB_NAME, synchronize: false, // Never in production logging: process.env.NODE_ENV === 'development', entities: ['src/entities/**/*.ts'], migrations: ['src/migrations/**/*.ts'], subscribers: ['src/subscribers/**/*.ts'], cli: { entitiesDir: 'src/entities', migrationsDir: 'src/migrations', subscribersDir: 'src/subscribers' }, extra: { max: 20, // Connection pool ssl: process.env.NODE_ENV === 'production' ? { rejectUnauthorized: false } : false } }; ``` ## References ### Documentation Links - [PostgreSQL Documentation](https://www.postgresql.org/docs/) - [MySQL Documentation](https://dev.mysql.com/doc/) - [MongoDB Documentation](https://docs.mongodb.com/) ### Monitoring Tools - [pgAdmin](https://www.pgadmin.org/) - [DataDog Database Monitoring](https://www.datadoghq.com/database-monitoring/) - [Prometheus PostgreSQL Exporter](https://github.com/prometheus-community/postgres_exporter) ### Performance Resources - [PostgreSQL Performance Tuning](https://wiki.postgresql.org/wiki/Performance_Optimization) - [Use The Index, Luke](https://use-the-index-luke.com/) - [PostgreSQL Query Planner](https://www.postgresql.org/docs/current/using-explain.html) --- Last updated: 2025-01-29 | Maintained by: Database Team ``` ### 5. API Documentation Template ```markdown # [API Name] API Documentation > RESTful API / GraphQL API / gRPC Service > Version: 2.0.0 > Base URL: https://api.example.com/v2 ## Overview Brief description of the API, its purpose, and main features. ## Table of Contents - [Getting Started](#getting-started) - [Authentication](#authentication) - [Rate Limiting](#rate-limiting) - [Endpoints](#endpoints) - [Data Models](#data-models) - [Error Handling](#error-handling) - [Webhooks](#webhooks) - [GraphQL Schema](#graphql-schema) - [WebSocket Events](#websocket-events) - [SDKs & Libraries](#sdks--libraries) - [Changelog](#changelog) ## Getting Started ### Quick Start ```bash # Get your API key from https://dashboard.example.com export API_KEY="your_api_key_here" # Make your first request curl -H "Authorization: Bearer $API_KEY" \ https://api.example.com/v2/users ``` ### Base URLs - Production: `https://api.example.com/v2` - Staging: `https://staging-api.example.com/v2` - Development: `http://localhost:3000/v2` ### API Versioning This API uses URL versioning. Always include the version in your requests: - Current version: `/v2` - Previous version: `/v1` (deprecated, sunset date: 2024-12-31) ## Authentication ### API Key Authentication Include your API key in the Authorization header: ```http Authorization: Bearer YOUR_API_KEY ``` ### OAuth 2.