UNPKG

claude-flow-tbowman01

Version:

Enterprise-grade AI agent orchestration with ruv-swarm integration (Alpha Release)

1,042 lines (843 loc) 28.5 kB
# Batchtools Usage Guide for Claude-Flow SPARC Prompts ## Table of Contents 1. [Introduction to Batchtools](#introduction-to-batchtools) 2. [Common Batchtools Patterns](#common-batchtools-patterns) 3. [Implementation Examples](#implementation-examples) 4. [Performance Optimization Tips](#performance-optimization-tips) 5. [Integration with SPARC Modes](#integration-with-sparc-modes) 6. [Best Practices Checklist](#best-practices-checklist) ## Introduction to Batchtools ### What are Batchtools? Batchtools are a set of utilities and patterns that enable parallel execution of multiple operations in Claude-Flow. Instead of processing tasks sequentially, batchtools allow you to execute multiple operations concurrently, significantly improving performance and reducing execution time. ### Why Batchtools Improve Performance 1. **Parallel Execution**: Execute multiple independent operations simultaneously 2. **Reduced Latency**: Minimize waiting time between operations 3. **Resource Efficiency**: Better utilization of system resources 4. **Atomic Operations**: Group related changes together for consistency 5. **Pipeline Optimization**: Enable efficient data processing pipelines ### Core Concepts and Principles #### Parallelization - Execute independent tasks simultaneously - Use `batchtools.parallel()` for concurrent operations - Ensure operations don't have interdependencies #### Batching - Group similar operations together - Use `batchtools.createFiles()` for multiple file creations - Apply `batchtools.modifyFiles()` for bulk modifications #### Pipeline Processing - Chain operations with dependencies - Use stages for sequential requirements - Parallelize within each stage ### When to Use Parallel vs Sequential Operations **Use Parallel Operations When:** - Tasks are independent of each other - Operations access different resources - No shared state modifications - Read-only operations on multiple files - Creating multiple new files **Use Sequential Operations When:** - Tasks depend on previous results - Operations modify shared state - Order of execution matters - Complex interdependencies exist - Database transactions require consistency ## Common Batchtools Patterns ### Parallel File Operations #### Batch File Reading ```javascript // Read multiple files simultaneously const files = await batchtools.parallel([ read('/src/controllers/user.controller.ts'), read('/src/services/user.service.ts'), read('/src/models/user.model.ts'), read('/tests/user.test.ts'), ]); // Process results const [controller, service, model, tests] = files; ``` #### Batch File Creation ```javascript // Create multiple files in a single operation await batchtools.createFiles([ { path: '/src/controllers/auth.controller.ts', content: generateController('auth'), }, { path: '/src/services/auth.service.ts', content: generateService('auth'), }, { path: '/src/middleware/auth.middleware.ts', content: generateMiddleware('auth'), }, { path: '/tests/auth.test.ts', content: generateTests('auth'), }, ]); ``` #### Batch File Editing ```javascript // Modify multiple files simultaneously await batchtools.modifyFiles([ { path: '/src/config/database.ts', modifications: [ { old: 'localhost', new: process.env.DB_HOST }, { old: '5432', new: process.env.DB_PORT }, ], }, { path: '/src/config/redis.ts', modifications: [ { old: 'localhost', new: process.env.REDIS_HOST }, { old: '6379', new: process.env.REDIS_PORT }, ], }, ]); ``` ### Concurrent Searches #### Parallel Grep Operations ```javascript // Search for patterns across different directories const searchResults = await batchtools.parallel([ grep('TODO', '/src/**/*.ts'), grep('FIXME', '/src/**/*.ts'), grep('deprecated', '/src/**/*.ts'), grep('console.log', '/src/**/*.ts'), ]); // Aggregate results const issues = { todos: searchResults[0], fixmes: searchResults[1], deprecated: searchResults[2], consoleLogs: searchResults[3], }; ``` #### Parallel Glob Patterns ```javascript // Find different file types concurrently const fileGroups = await batchtools.parallel([ glob('**/*.test.ts'), glob('**/*.spec.ts'), glob('**/*.controller.ts'), glob('**/*.service.ts'), ]); const testFiles = [...fileGroups[0], ...fileGroups[1]]; const implementationFiles = [...fileGroups[2], ...fileGroups[3]]; ``` ### Batch Task Execution #### Parallel Test Execution ```javascript // Run different test suites simultaneously const testResults = await batchtools.parallel([ exec('npm run test:unit'), exec('npm run test:integration'), exec('npm run test:e2e'), exec('npm run lint'), exec('npm run typecheck'), ]); // Check all results const allPassed = testResults.every((result) => result.exitCode === 0); ``` #### Concurrent Build Process ```javascript // Build multiple components in parallel await batchtools.parallel([ buildComponent('frontend'), buildComponent('backend'), buildComponent('shared-lib'), generateDocumentation(), ]); ``` ### Pipeline Operations with Dependencies #### Multi-Stage Pipeline ```javascript // Define pipeline with parallel stages const pipeline = await batchtools.pipeline([ { name: 'analysis', parallel: [analyzeCodeQuality(), checkDependencies(), validateArchitecture()], }, { name: 'build', parallel: [buildFrontend(), buildBackend(), buildDockerImages()], }, { name: 'test', parallel: [runUnitTests(), runIntegrationTests(), runSecurityScans()], }, { name: 'deploy', sequential: [deployToStaging(), runSmokeTests(), deployToProduction()], }, ]); ``` ## Implementation Examples ### Real Code Examples from Optimized Prompts #### Architect Mode - Parallel Component Analysis ```javascript // From architect.md - Analyze multiple components simultaneously const architectureAnalysis = async (components) => { const analyses = await batchtools.parallel( components.map((component) => async () => { const [code, tests, docs] = await batchtools.parallel([ read(`/src/components/${component}/**/*.ts`), read(`/tests/${component}/**/*.test.ts`), read(`/docs/${component}.md`), ]); return { component, dependencies: analyzeDependencies(code), testCoverage: analyzeTestCoverage(tests), documentation: analyzeDocumentation(docs), }; }), ); return consolidateAnalysis(analyses); }; ``` #### TDD Mode - Parallel Test Creation ```javascript // From tdd.md - Create complete test suite in parallel const createTestSuite = async (feature) => { const testTypes = ['unit', 'integration', 'e2e', 'performance']; await batchtools.parallel( testTypes.map((type) => async () => { const tests = generateTestsForType(feature, type); const testFiles = tests.map((test) => ({ path: `/tests/${type}/${feature}/${test.name}.test.ts`, content: test.content, })); return batchtools.createFiles(testFiles); }), ); }; ``` #### Code Mode - Batch CRUD Generation ```javascript // From code.md - Generate CRUD operations for multiple entities const generateCRUD = async (entities) => { await batchtools.forEach(entities, async (entity) => { const components = await batchtools.parallel([ generateController(entity), generateService(entity), generateRepository(entity), generateDTO(entity), generateValidation(entity), ]); const files = [ { path: `/src/controllers/${entity.toLowerCase()}.controller.ts`, content: components[0] }, { path: `/src/services/${entity.toLowerCase()}.service.ts`, content: components[1] }, { path: `/src/repositories/${entity.toLowerCase()}.repository.ts`, content: components[2] }, { path: `/src/dto/${entity.toLowerCase()}.dto.ts`, content: components[3] }, { path: `/src/validators/${entity.toLowerCase()}.validator.ts`, content: components[4] }, ]; await batchtools.createFiles(files); }); }; ``` ### Before/After Comparisons #### Before: Sequential File Creation ```javascript // Slow sequential approach - ~5 seconds for 5 files await write('/src/controllers/user.controller.ts', userController); await write('/src/services/user.service.ts', userService); await write('/src/models/user.model.ts', userModel); await write('/src/validators/user.validator.ts', userValidator); await write('/tests/user.test.ts', userTests); ``` #### After: Parallel File Creation ```javascript // Fast parallel approach - ~1 second for 5 files await batchtools.createFiles([ { path: '/src/controllers/user.controller.ts', content: userController }, { path: '/src/services/user.service.ts', content: userService }, { path: '/src/models/user.model.ts', content: userModel }, { path: '/src/validators/user.validator.ts', content: userValidator }, { path: '/tests/user.test.ts', content: userTests }, ]); ``` #### Before: Sequential Analysis ```javascript // Slow sequential analysis - ~10 seconds for 4 operations const codeQuality = await analyzeCodeQuality(); const dependencies = await checkDependencies(); const security = await runSecurityScan(); const performance = await analyzePerformance(); ``` #### After: Parallel Analysis ```javascript // Fast parallel analysis - ~3 seconds for 4 operations const [codeQuality, dependencies, security, performance] = await batchtools.parallel([ analyzeCodeQuality(), checkDependencies(), runSecurityScan(), analyzePerformance(), ]); ``` ### Common Pitfalls and How to Avoid Them #### Pitfall 1: Race Conditions ```javascript // BAD: Modifying shared state in parallel let counter = 0; await batchtools.parallel([ async () => { counter += await getCount1(); }, async () => { counter += await getCount2(); }, ]); // GOOD: Collect results then aggregate const [count1, count2] = await batchtools.parallel([getCount1(), getCount2()]); const counter = count1 + count2; ``` #### Pitfall 2: Dependency Conflicts ```javascript // BAD: Creating file and reading it in parallel await batchtools.parallel([ write('/config.json', config), read('/config.json'), // May fail - file might not exist yet ]); // GOOD: Use sequential operations for dependencies await write('/config.json', config); const configData = await read('/config.json'); ``` #### Pitfall 3: Resource Exhaustion ```javascript // BAD: Too many parallel operations const results = await batchtools.parallel(thousandFiles.map((file) => processLargeFile(file))); // GOOD: Use batching with limits const results = await batchtools.batchProcess(thousandFiles, (file) => processLargeFile(file), { batchSize: 10, concurrency: 5, }); ``` ## Performance Optimization Tips ### Resource Management Strategies #### Memory Management ```javascript // Process large datasets in chunks const processLargeDataset = async (files) => { const chunkSize = 100; const results = []; for (let i = 0; i < files.length; i += chunkSize) { const chunk = files.slice(i, i + chunkSize); const chunkResults = await batchtools.parallel(chunk.map((file) => processFile(file))); results.push(...chunkResults); // Allow garbage collection between chunks await new Promise((resolve) => setImmediate(resolve)); } return results; }; ``` #### CPU Optimization ```javascript // Balance CPU-intensive tasks const optimalConcurrency = os.cpus().length; await batchtools.parallel(tasks, { concurrency: optimalConcurrency, scheduler: 'round-robin', }); ``` ### Optimal Batch Sizes #### Dynamic Batch Sizing ```javascript const calculateOptimalBatchSize = (totalItems, itemSize) => { const availableMemory = os.freemem(); const maxBatchMemory = availableMemory * 0.7; // Use 70% of available const optimalSize = Math.floor(maxBatchMemory / itemSize); return Math.min(optimalSize, totalItems, 1000); // Cap at 1000 }; const batchSize = calculateOptimalBatchSize(files.length, avgFileSize); ``` ### Handling Errors in Parallel Operations #### Error Collection Pattern ```javascript const safeParallel = async (operations) => { const results = await batchtools.parallel( operations.map((op) => op.catch((err) => ({ error: err }))), ); const errors = results.filter((r) => r.error); const successes = results.filter((r) => !r.error); if (errors.length > 0) { console.error(`${errors.length} operations failed:`, errors); } return { successes, errors }; }; ``` #### Retry Pattern ```javascript const parallelWithRetry = async (operations, maxRetries = 3) => { const retry = async (op, attempts = 0) => { try { return await op(); } catch (error) { if (attempts < maxRetries) { await new Promise((r) => setTimeout(r, Math.pow(2, attempts) * 1000)); return retry(op, attempts + 1); } throw error; } }; return batchtools.parallel(operations.map((op) => () => retry(op))); }; ``` ### Monitoring and Debugging Batch Operations #### Progress Tracking ```javascript const trackProgress = async (operations, onProgress) => { let completed = 0; const total = operations.length; const wrappedOps = operations.map((op, index) => async () => { const result = await op(); completed++; onProgress({ completed, total, percentage: (completed / total) * 100 }); return result; }); return batchtools.parallel(wrappedOps); }; // Usage await trackProgress(operations, ({ completed, total, percentage }) => { console.log(`Progress: ${completed}/${total} (${percentage.toFixed(1)}%)`); }); ``` #### Performance Monitoring ```javascript const measureBatchPerformance = async (operations) => { const startTime = Date.now(); const startMemory = process.memoryUsage(); const results = await batchtools.parallel(operations); const endTime = Date.now(); const endMemory = process.memoryUsage(); return { results, metrics: { duration: endTime - startTime, memoryDelta: endMemory.heapUsed - startMemory.heapUsed, throughput: operations.length / ((endTime - startTime) / 1000), }, }; }; ``` ## Integration with SPARC Modes ### How Each SPARC Mode Benefits from Batchtools #### Specification Mode - **Parallel Requirements Analysis**: Analyze multiple requirement documents simultaneously - **Concurrent Validation**: Validate specifications against multiple criteria in parallel - **Batch Documentation**: Generate specification documents for multiple features at once ```javascript // Parallel specification validation const validateSpecifications = async (specs) => { const validations = await batchtools.parallel([ checkCompleteness(specs), validateConsistency(specs), verifyTestability(specs), assessFeasibility(specs), ]); return consolidateValidationResults(validations); }; ``` #### Architecture Mode - **Component Analysis**: Analyze multiple system components concurrently - **Diagram Generation**: Create multiple architecture diagrams in parallel - **Dependency Mapping**: Map dependencies across services simultaneously ```javascript // Parallel architecture artifact generation const generateArchitectureArtifacts = async (system) => { await batchtools.parallel([ generateSystemDiagram(system), generateComponentDiagram(system), generateDeploymentDiagram(system), generateDataFlowDiagram(system), generateSecurityModel(system), ]); }; ``` #### Code Mode - **Module Generation**: Generate multiple modules simultaneously - **API Creation**: Create endpoints and handlers in parallel - **Service Implementation**: Implement multiple services concurrently ```javascript // Parallel service implementation const implementServices = async (services) => { await batchtools.forEach(services, async (service) => { const implementations = await batchtools.parallel([ implementInterface(service), implementBusinessLogic(service), implementDataAccess(service), implementValidation(service), implementErrorHandling(service), ]); await batchtools.createFiles( implementations.map((impl) => ({ path: impl.path, content: impl.content, })), ); }); }; ``` #### TDD Mode - **Test Generation**: Create multiple test suites in parallel - **Test Execution**: Run different test types concurrently - **Coverage Analysis**: Analyze coverage across modules simultaneously ```javascript // Parallel TDD cycle const tddCycle = async (feature) => { // Red phase - Create all failing tests await batchtools.parallel([ createUnitTests(feature), createIntegrationTests(feature), createE2ETests(feature), ]); // Green phase - Implement in parallel await batchtools.parallel([ implementFeatureLogic(feature), implementAPIEndpoints(feature), implementDataLayer(feature), ]); // Verify - Run all tests in parallel const results = await batchtools.parallel([ runUnitTests(feature), runIntegrationTests(feature), runE2ETests(feature), ]); return results.every((r) => r.passed); }; ``` #### Integration Mode - **Service Wiring**: Connect multiple services simultaneously - **Health Checks**: Verify multiple endpoints in parallel - **Configuration**: Apply configurations across services concurrently ```javascript // Parallel integration validation const validateIntegration = async (services) => { const checks = await batchtools.parallel([ ...services.map((s) => checkServiceHealth(s)), ...services.map((s) => validateAPIContracts(s)), ...services.map((s) => testServiceCommunication(s)), verifyDataConsistency(services), checkSecurityPolicies(services), ]); return aggregateValidationResults(checks); }; ``` ### Mode-Specific Optimization Strategies #### Development Modes ```javascript // Optimize for rapid development const developmentOptimizations = { architect: { parallel: ['analysis', 'design', 'documentation'], cache: true, hotReload: true, }, code: { parallel: ['generation', 'formatting', 'validation'], incremental: true, watch: true, }, tdd: { parallel: ['test-creation', 'test-execution'], continuous: true, coverage: true, }, }; ``` #### Quality Assurance Modes ```javascript // Optimize for thorough analysis const qaOptimizations = { debug: { parallel: ['log-analysis', 'trace-collection', 'state-inspection'], detailed: true, snapshot: true, }, security: { parallel: ['static-analysis', 'dynamic-testing', 'dependency-scan'], comprehensive: true, report: true, }, optimization: { parallel: ['performance-profiling', 'memory-analysis', 'bundle-size'], benchmark: true, compare: true, }, }; ``` ### Cross-Mode Parallel Execution Patterns #### Pipeline Pattern ```javascript // Execute multiple modes in a pipeline const sparcPipeline = async (feature) => { const stages = [ { name: 'specification', modes: ['spec-pseudocode'], parallel: false, }, { name: 'design', modes: ['architect'], parallel: false, }, { name: 'implementation', modes: ['code', 'tdd'], parallel: true, }, { name: 'quality', modes: ['debug', 'security-review', 'optimization'], parallel: true, }, { name: 'integration', modes: ['integration'], parallel: false, }, ]; for (const stage of stages) { if (stage.parallel) { await batchtools.parallel(stage.modes.map((mode) => executeMode(mode, feature))); } else { for (const mode of stage.modes) { await executeMode(mode, feature); } } } }; ``` #### Swarm Pattern ```javascript // Coordinate multiple modes for complex tasks const swarmExecution = async (task) => { const swarmConfig = { frontend: { modes: ['architect', 'code', 'tdd'], agents: 3, parallel: true, }, backend: { modes: ['architect', 'code', 'tdd', 'security-review'], agents: 4, parallel: true, }, infrastructure: { modes: ['devops', 'security-review'], agents: 2, parallel: true, }, documentation: { modes: ['docs-writer'], agents: 1, parallel: false, }, }; await batchtools.parallel( Object.entries(swarmConfig).map(([component, config]) => executeSwarm(component, config, task)), ); }; ``` ## Best Practices Checklist ### When to Use Batch Operations **Use Batch Operations When:** - [ ] Creating multiple files with no interdependencies - [ ] Reading multiple files for analysis - [ ] Running independent test suites - [ ] Performing multiple searches or validations - [ ] Generating documentation for multiple components - [ ] Analyzing code across multiple modules - [ ] Building multiple independent components - [ ] Deploying to multiple environments **Avoid Batch Operations When:** - [ ] Operations depend on previous results - [ ] Modifying shared state or resources - [ ] Database transactions require strict ordering - [ ] File operations have dependencies - [ ] Memory constraints are tight - [ ] Operations require sequential validation - [ ] Error in one operation should stop others ### Security Considerations #### Secure Parallel Execution ```javascript // Validate inputs before parallel execution const secureParallelExecution = async (operations, validators) => { // Validate all operations first const validations = await batchtools.parallel(operations.map((op, i) => validators[i](op))); if (validations.some((v) => !v.valid)) { throw new Error('Validation failed for one or more operations'); } // Execute with security constraints return batchtools.parallel(operations, { timeout: 30000, maxMemory: '1GB', sandbox: true, }); }; ``` #### Resource Isolation ```javascript // Isolate resources in parallel operations const isolatedExecution = async (operations) => { return batchtools.parallel( operations.map((op) => ({ execute: op, context: createIsolatedContext(), permissions: getOperationPermissions(op), })), ); }; ``` ### Error Handling Strategies #### Graceful Degradation ```javascript const gracefulBatchExecution = async (operations) => { const results = { successful: [], failed: [], partial: [], }; const outcomes = await batchtools.parallel( operations.map((op) => op() .then((result) => ({ status: 'success', result })) .catch((error) => ({ status: 'error', error })), ), ); outcomes.forEach((outcome, index) => { if (outcome.status === 'success') { results.successful.push({ index, ...outcome }); } else { results.failed.push({ index, ...outcome }); } }); return results; }; ``` #### Circuit Breaker Pattern ```javascript class BatchCircuitBreaker { constructor(threshold = 5, timeout = 60000) { this.failureCount = 0; this.threshold = threshold; this.timeout = timeout; this.state = 'CLOSED'; this.nextAttempt = Date.now(); } async execute(operations) { if (this.state === 'OPEN' && Date.now() < this.nextAttempt) { throw new Error('Circuit breaker is OPEN'); } try { const results = await batchtools.parallel(operations); this.onSuccess(); return results; } catch (error) { this.onFailure(); throw error; } } onSuccess() { this.failureCount = 0; this.state = 'CLOSED'; } onFailure() { this.failureCount++; if (this.failureCount >= this.threshold) { this.state = 'OPEN'; this.nextAttempt = Date.now() + this.timeout; } } } ``` ### Testing Parallel Code #### Unit Testing Batch Operations ```javascript describe('Batch Operations', () => { it('should execute operations in parallel', async () => { const startTime = Date.now(); const operations = [ () => delay(100).then(() => 'A'), () => delay(100).then(() => 'B'), () => delay(100).then(() => 'C'), ]; const results = await batchtools.parallel(operations); const duration = Date.now() - startTime; expect(results).toEqual(['A', 'B', 'C']); expect(duration).toBeLessThan(150); // Should be ~100ms, not 300ms }); it('should handle partial failures', async () => { const operations = [ () => Promise.resolve('success'), () => Promise.reject(new Error('fail')), () => Promise.resolve('success'), ]; const results = await gracefulBatchExecution(operations); expect(results.successful).toHaveLength(2); expect(results.failed).toHaveLength(1); }); }); ``` #### Integration Testing ```javascript describe('Parallel Integration', () => { it('should integrate multiple services concurrently', async () => { const services = ['auth', 'user', 'notification']; const results = await batchtools.parallel(services.map((service) => integrateService(service))); // Verify all services integrated successfully results.forEach((result) => { expect(result.status).toBe('integrated'); expect(result.healthCheck).toBe('passing'); }); // Verify cross-service communication const communicationTest = await testServiceCommunication(services); expect(communicationTest.allConnected).toBe(true); }); }); ``` ### Performance Benchmarking ```javascript const benchmarkBatchOperations = async () => { const operations = generateTestOperations(1000); console.log('Benchmarking Sequential vs Parallel Execution...\n'); // Sequential benchmark const sequentialStart = Date.now(); for (const op of operations) { await op(); } const sequentialTime = Date.now() - sequentialStart; // Parallel benchmark with different concurrency levels const concurrencyLevels = [5, 10, 20, 50, 100]; const results = {}; for (const concurrency of concurrencyLevels) { const parallelStart = Date.now(); await batchtools.parallel(operations, { concurrency }); const parallelTime = Date.now() - parallelStart; results[concurrency] = { time: parallelTime, speedup: sequentialTime / parallelTime, throughput: operations.length / (parallelTime / 1000), }; } console.log(`Sequential: ${sequentialTime}ms`); console.log('\nParallel Results:'); Object.entries(results).forEach(([concurrency, metrics]) => { console.log( ` Concurrency ${concurrency}: ${metrics.time}ms (${metrics.speedup.toFixed(2)}x speedup, ${metrics.throughput.toFixed(0)} ops/sec)`, ); }); }; ``` ## Conclusion Batchtools are a powerful feature of Claude-Flow that can significantly improve the performance of your SPARC development workflow. By following the patterns and best practices outlined in this guide, you can: 1. **Reduce execution time** by parallelizing independent operations 2. **Improve resource utilization** through efficient batching 3. **Enhance code quality** with comprehensive parallel testing 4. **Scale your development** process across multiple components 5. **Maintain reliability** with proper error handling and monitoring Remember to always consider the nature of your operations, manage resources appropriately, and handle errors gracefully when implementing batchtools in your projects. For more information and updates, refer to the [Claude-Flow documentation](https://github.com/ruvnet/claude-code-flow) and the SPARC methodology guides.