UNPKG

arela

Version:

AI-powered CTO with multi-agent orchestration, code summarization, visual testing (web + mobile) for blazing fast development.

954 lines (799 loc) 27.1 kB
# Arela Codebase Overview & Architecture ## Executive Summary Arela v3.7.0 is an **AI-powered CTO platform** currently in Phase 1 (Foundation) with a vision to transform into a complete codebase analysis and autonomous refactoring system. The codebase is well-structured with clear separation of concerns, using TypeScript, SQLite for graph storage, and a modular architecture supporting multi-agent orchestration. **Current Status:** Phase 1 Complete | Phase 2 (v3.8.0) In Progress - Slice Detection --- ## 1. Overall Structure of src/ Directory ``` src/ ├── analyze/ # Phase 1 - Architecture analysis (multi-repo, coupling, cohesion) ├── architecture.ts # Main architecture analyzer ├── coupling.ts # Coupling score calculation ├── cohesion.ts # Cohesion score calculation ├── multi-repo.ts # Cross-repo analysis ├── patterns.ts # Pattern detection ├── types.ts # Architecture types & interfaces └── reporter.ts # Output formatting ├── ingest/ # Phase 1 - Codebase ingestion (files → graph) ├── index.ts # Main orchestrator ├── types.ts # Data structures for ingestion ├── file-scanner.ts # Find files in repo ├── static-analyzer.ts # TypeScript AST analysis ├── universal-analyzer.ts # Language-agnostic regex parsing ├── graph-builder.ts # Build dependency graph └── storage.ts # SQLite GraphDB interface ├── memory/ # Phase 1 - Tri-Memory System ├── index.ts # Unified memory interface ├── types.ts # Memory data structures ├── graph.ts # Graph memory (queries dependency graph) ├── vector.ts # Vector memory (RAG wrapper) ├── audit.ts # Governance log (SQLite) └── cli.ts # Memory CLI commands ├── agents/ # Agent orchestration & discovery ├── discovery.ts # Find installed AI agents ├── orchestrate.ts # Run tickets with multi-agent support ├── dispatch.ts # Route work to appropriate agent └── status.ts # Show ticket execution status ├── flow/ # Flow analysis & tracing (v3.5.0) ├── analyzer.ts # Analyze function call flows ├── tracer.ts # Trace execution flows ├── standards.ts # Flow patterns └── reporter.ts # Flow analysis reporting ├── analyze/ # Old analysis module (legacy) └── [files from Phase 1] ├── tickets/ # Ticket parsing & management ├── parser.ts # Parse .md/.yaml tickets ├── schema.ts # Ticket validation └── auto-generate.ts # Generate tickets from prompts ├── rag/ # Semantic search (vector DB wrapper) ├── index.ts # Main RAG interface └── server.ts # RAG server ├── utils/ # Utilities ├── progress.ts # Progress bars ├── schema.ts # Zod schemas ├── cli-personality.ts # CLI messaging styles ├── structure-validator.ts # Project validation ├── auto-index.ts # Incremental RAG indexing └── ragignore.ts # RAG ignore patterns ├── mcp/ # Model Context Protocol (Windsurf integration) └── server.ts # MCP server implementation ├── persona/ # AI persona/rules system ├── loader.ts # Load persona rules └── templates/ # Rule templates ├── run/ # Test execution (web & mobile) ├── web.ts # Web app testing ├── mobile.ts # Mobile app testing ├── flows.ts # User flow definitions ├── reporter.ts # Test reporting └── pilot.ts # AI pilot (autonomous testing) ├── generate/ # Code generation └── flow-generator.ts # Generate test flows from AI ├── cli.ts # Main CLI (Commander.js) ├── index.ts # Programmatic exports └── types.ts # Core type definitions ``` ### Directory Organization Summary | Directory | Purpose | Phase | Status | |-----------|---------|-------|--------| | `ingest/` | Parse files Graph DB | Phase 1 | Complete | | `memory/` | Tri-Memory System (Vector/Graph/Audit) | Phase 1 | Complete | | `analyze/` | Architecture analysis (coupling/cohesion) | Phase 1 | Complete | | `flow/` | Function call flow analysis | Phase 1 | Complete | | `agents/` | Multi-agent orchestration | Phase 1 | Complete | | `run/` | Test execution (web/mobile) | Phase 1 | Complete | | `detect/` | **Slice detection (NEW)** | Phase 2 | 🚧 To Build | --- ## 2. Graph DB Implementation (SQLite) ### Current Status **Fully Implemented** in `src/ingest/storage.ts` ### Database Location ``` .arela/memory/graph.db # SQLite database (WAL mode) ``` ### Database Schema (8 Tables) ```sql 1. files - id, path, repo, type, lines - index: repo, type - tracks: every file in the codebase 2. functions - id, name, file_id, line_start, line_end, is_exported - index: file_id, name - tracks: function definitions 3. imports - id, from_file_id, to_file_id, to_module, import_type, imported_names, line_number - index: from_file_id, to_file_id - tracks: import relationships 4. function_calls - id, caller_function_id, callee_function_id, callee_name, line_number - index: caller_function_id, callee_function_id - tracks: function→function calls 5. api_endpoints - id, method, path, file_id, function_id, line_number - index: file_id, method+path - tracks: REST endpoints defined 6. api_calls - id, method, url, file_id, line_number - index: file_id, method+url - tracks: API calls made 7. audit_log (governance) - id, timestamp, agent, action, result, metadata - tracks: decision audit trail 8. vector_index (wrapped by vector.ts) - Semantic embeddings for RAG ``` ### GraphDB Class Interface ```typescript class GraphDB { // Initialization constructor(dbPath: string) private initSchema() // File operations addFile(file: FileNode): number getFileId(filePath: string): number | null // Function operations addFunction(fileId: number, func: FunctionNode): number getFunctionId(fileId: number, functionName: string): number | null // Dependency tracking addImport(fromFileId, toFileId, toModule, importType, names, line) addFunctionCall(callerFunctionId, calleeId, calleeName, line) // API tracking addApiEndpoint(endpoint: ApiEndpoint) addApiCall(fileId: number, call: ApiCall) // Querying query(sql: string, params: any[]): any[] getSummary(): { filesCount, functionsCount, importsCount, ... } // Transactions beginTransaction() commit() rollback() // Utilities close() clear() } ``` ### Example Queries Available ```typescript // Find files that depend on a file SELECT f.path FROM imports i JOIN files f ON f.id = i.from_file_id WHERE i.to_file_id = ? // Find functions called by a function SELECT f.name, f2.path FROM function_calls fc JOIN functions f ON f.id = fc.caller_function_id JOIN functions f2 ON f2.id = fc.callee_function_id WHERE fc.caller_function_id = ? // Find API endpoints in a file SELECT method, path FROM api_endpoints WHERE file_id = ? ``` ### Data Types Stored ```typescript // From src/ingest/types.ts FileNode { path: string // relative path repoPath: string // absolute repo path type: FileType // component|service|controller|util|hook|type|config|other lines: number imports?: ImportInfo[] exports?: ExportInfo[] functions?: FunctionNode[] apiCalls?: ApiCall[] } ImportInfo { from: string // module/file names: string[] // [imports] type: 'default'|'named'|'namespace' line: number } FunctionNode { name: string fileId: number lineStart: number lineEnd: number isExported: boolean calls?: number[] // IDs of called functions calledBy?: number[] } ApiCall { method: string // GET, POST, etc. url: string line: number } ``` --- ## 3. CLI Structure & Command Registration ### Framework - **Library:** `commander.js` v12.0.0 - **File:** `src/cli.ts` (700+ lines) - **Pattern:** Commander.js program with subcommands ### CLI Commands Registered ``` arela agents # List discovered AI agents arela init # Initialize project setup arela orchestrate # Run tickets with multi-agent arela run [web|mobile] # Test execution arela status # Show ticket status arela doctor # Validate project structure arela index # Build RAG index arela auto-index # Incremental RAG update arela install-hook # Install git auto-index hook arela uninstall-hook # Remove git hook arela analyze [flow|architecture] # Code analysis arela ingest [codebase] # Ingest files → graph arela mcp # Start MCP server arela memory [init|query|impact|audit|status] # Tri-Memory commands ``` ### Command Registration Pattern ```typescript // Example from cli.ts program .command("agents") .description("Discover and list available AI agents") .option("--verbose", "Show detailed information", false) .action(async (opts) => { const agents = await discoverAgents(); // Handle command }); // For complex commands with dynamic handlers: export class UnsupportedPlatformError extends Error { constructor(public readonly platform: string) { super(`Platform "${platform}" not supported.`); } } export function buildRunCommand( programInstance: Command, handler: RunCommandHandler = handleRunCommand ): Command { return programInstance .command("run") .description("Run and test your app") .argument("<platform>", "Platform: web or mobile") .option("--url <url>", "URL for web apps") .action(async (platformArg, opts) => { try { await handler(platformArg, opts); } catch (error) { if (error instanceof UnsupportedPlatformError) { console.error(error.message); process.exit(1); } } }); } ``` ### Programmatic Usage All CLI commands are exported for programmatic use: ```typescript // Exported from src/index.ts export * from './agents/discovery.js' export * from './agents/orchestrate.js' export * from './tickets/parser.js' export * from './memory/index.js' ``` Usage example: ```typescript import { discoverAgents, orchestrate } from 'arela'; const agents = await discoverAgents(); await orchestrate({ cwd: process.cwd(), parallel: true, tickets: ['CODEX-001', 'CLAUDE-002'] }); ``` --- ## 4. Types & Interfaces ### Core Domain Types ```typescript // src/types.ts - Core types export type AgentName = 'codex' | 'claude' | 'deepseek' | 'ollama' | 'cascade' export type TicketPriority = 'low' | 'medium' | 'high' | 'critical' export type TicketComplexity = 'simple' | 'medium' | 'complex' export type TicketStatus = 'pending' | 'in_progress' | 'completed' | 'failed' | 'blocked' export interface Ticket { id: string title: string description: string agent: AgentName priority: TicketPriority complexity: TicketComplexity status: TicketStatus estimatedTime?: string estimatedCost?: number dependencies?: string[] tags?: string[] files?: Array<{ path: string; action: 'create'|'modify'|'delete' }> acceptance?: Array<{ id: string; description: string; status: 'pending'|'passed'|'failed' }> } export interface DiscoveredAgent { name: string command: string available: boolean type: 'cloud' | 'local' | 'ide' version?: string } ``` ### Graph & Import Types ```typescript // src/ingest/types.ts export interface FileNode { path: string repoPath: string type: FileType // component|service|controller|util|hook|type|config|other lines: number imports?: ImportInfo[] exports?: ExportInfo[] functions?: FunctionNode[] apiCalls?: ApiCall[] } export interface ImportInfo { from: string names: string[] type: 'default' | 'named' | 'namespace' line: number } export interface FunctionNode { id?: number name: string fileId?: number filePath?: string isExported: boolean lineStart: number lineEnd: number calls?: number[] calledBy?: number[] } export interface FunctionCall { callerId: number calleeId: number line: number } export interface ApiEndpoint { method: string // GET, POST, PUT, DELETE path: string fileId: number functionId?: number line: number } export interface ApiCall { method: string url: string line: number filePath?: string } export interface CodebaseMap { summary: { filesScanned: number importsFound: number functionsDefined: number apiCallsFound: number } stats: { modules: number components: number services: number apiEndpoints: number } dbPath: string duration: number } ``` ### Memory System Types ```typescript // src/memory/types.ts export interface DependencyEdge { file: string reason: string // e.g., "imports, default export" weight: number // number of import relationships } export interface ImpactAnalysis { file: string exists: boolean upstream: DependencyEdge[] // files that import this downstream: DependencyEdge[] // files this imports fanIn: number // total upstream dependency weight fanOut: number // total downstream dependency weight } export interface GraphStats { ready: boolean files: number imports: number functions: number functionCalls: number apiEndpoints: number apiCalls: number dbPath: string } export interface AuditEntry { timestamp?: string agent: string action: string inputHash?: string outputHash?: string result: 'success' | 'failure' | 'pending' metadata?: Record<string, unknown> commitHash?: string ticketId?: string policyViolations?: unknown[] } export interface MemoryQueryResult { question: string semantic: SemanticResult[] // from vector DB (RAG) relatedFiles: string[] // from graph DB timestamp: string } export interface TriMemoryStats { vector: VectorStats graph: GraphStats audit: AuditStats } ``` ### Architecture Analysis Types ```typescript // src/analyze/types.ts export type ArchitectureType = 'horizontal' | 'vertical' | 'hybrid' export interface ArchitectureScore { horizontal: number // 0-100 vertical: number // 0-100 } export interface CouplingCohesionScores { coupling: number // 0-100 (lower is better) cohesion: number // 0-100 (higher is better) } export interface RepoAnalysis { name: string path: string architecture: ArchitectureType scores: ArchitectureScore metrics: CouplingCohesionScores directories: DirectoryAnalysis[] issues: ArchitectureIssue[] } export interface DirectoryAnalysis { path: string type: DirectoryType // layer|feature|module|other fileCount: number internalImports: number // within directory externalImports: number // from outside importedBy: number // reverse imports } export interface ArchitectureIssue { severity: 'critical' | 'warning' | 'info' title: string description: string affectedFiles?: number affectedDirs?: string[] recommendation?: string } export interface ArchitectureReport { timestamp: string repositories: RepoAnalysis[] overallArchitecture: ArchitectureType overallScores: ArchitectureScore globalMetrics: CouplingCohesionScores issues: ArchitectureIssue[] apiDrift: ApiDrift[] recommendations: string[] effort?: { estimated: string // "8-12 weeks" breakeven: string // "14 months" roi3Year: number // 380 for 380% } } ``` --- ## 5. Database System & Storage ### Primary Database: SQLite with better-sqlite3 **Package:** `better-sqlite3@^11.0.0` **Location:** `.arela/memory/graph.db` **Configuration:** ```typescript this.db.pragma("journal_mode = WAL"); // Write-Ahead Logging (faster) this.db.pragma("foreign_keys = ON"); // Enable foreign keys ``` ### Storage Layers ``` ┌─────────────────────────────────────────┐ Arela Memory System ├─────────────────────────────────────────┤ TriMemory (src/memory/index.ts) ├── VectorMemory RAG index ├── GraphMemory graph.db └── AuditMemory audit.db ├─────────────────────────────────────────┤ .arela/memory/ ├── graph.db (SQLite) ├── audit.db (SQLite) └── vector/ (RAG embeddings) └─────────────────────────────────────────┘ ``` ### Ingest Pipeline ``` Codebase scanDirectory (find files) analyzeFile (universal-analyzer or static-analyzer) ├── Extract imports ├── Extract functions ├── Extract API calls └── Extract exports buildGraph (graph-builder.ts) ├── Add files to DB ├── Add functions to DB ├── Resolve imports ├── Resolve function calls └── Add API endpoints/calls GraphDB (storage.ts) └── .arela/memory/graph.db ``` ### Query Interface ```typescript // Example: Find all files that import a specific file const dependents = await queryGraph( repoPath, `SELECT f.path FROM imports i JOIN files f ON f.id = i.from_file_id WHERE i.to_file_id = (SELECT id FROM files WHERE path = ?)`, ['src/auth/login.ts'] ); // Example: Analyze impact of a file const impact = await triMemory.impact('src/auth/login.ts'); // Returns: { // upstream: [{file: 'x', weight: 3, reason: 'imports'}, ...], // downstream: [{file: 'y', weight: 2, reason: 'imports'}, ...], // fanIn: 5, fanOut: 2 // } ``` --- ## 6. Phase 1 Implementation Status ✅ ### Completed Features #### 1.1 Multi-Repo Architecture Analyzer - **File:** `src/analyze/architecture.ts` - **Status:** Complete - **Capabilities:** - Analyzes single & multiple repositories - Detects architecture type (horizontal/vertical/hybrid) - Calculates coupling/cohesion scores (0-100) - Identifies architectural issues - Recommends VSA migration path - Calculates effort/ROI estimates - **CLI:** ```bash arela analyze architecture /mobile /backend ``` #### 1.2 Universal Codebase Ingestion - **Files:** `src/ingest/*` - **Status:** Complete - **Capabilities:** - Supports 15+ programming languages - Language-agnostic regex parsing - Extracts imports, functions, API calls - Builds complete dependency graph - Stores in SQLite GraphDB - Fast: 3,585 files in 3.91s - **Languages:** JS/TS, Python, Go, Rust, Ruby, PHP, Java, C#, C++, Swift, Kotlin - **CLI:** ```bash arela ingest codebase arela ingest codebase --refresh --analyze ``` #### 1.3 Tri-Memory System - **Files:** `src/memory/*` - **Status:** Complete - **Three Types:** 1. **Vector Memory** - Semantic RAG search 2. **Graph Memory** - Structural dependencies 3. **Audit Memory** - Governance log - **CLI:** ```bash arela memory init arela memory query "authentication logic" arela memory impact src/auth/login.ts arela memory audit --ticket CODEX-001 arela memory status ``` #### 1.4 Flow Analysis - **Files:** `src/flow/*` - **Status:** Complete - **Capabilities:** - Analyze function call flows - Trace execution paths - Identify flow patterns - **CLI:** ```bash arela analyze flow main --verbose ``` #### 1.5 Agent Discovery & Orchestration - **Files:** `src/agents/*` - **Status:** Complete - **Capabilities:** - Discover installed AI agents (Claude, Codex, Ollama, etc.) - Route work to appropriate agent - Execute tickets with multi-agent orchestration - Show execution status - **CLI:** ```bash arela agents --verbose arela orchestrate --parallel --max-parallel 5 ``` --- ## 7. Phase 2 Development Plan (v3.8.0) 🚧 ### Feature 2.1: Autonomous Slice Boundary Detection (YOUR TASK) **Location:** `src/detect/` (to be created) **Ticket:** `.arela/tickets/claude/CLAUDE-001-v3.8.0-slice-detection.md` **Algorithm:** Louvain Community Detection - O(n log n) complexity - Maximizes modularity - Produces hierarchical communities - Works on weighted graphs **Expected Files:** ``` src/detect/ ├── slices.ts # Main detector interface ├── clustering.ts # Louvain algorithm implementation ├── scoring.ts # Cohesion/coupling scoring └── review.ts # Human approval workflow ``` **Expected CLI:** ```bash arela detect slices arela detect slices --repo /path/to/repo arela detect slices /path/to/mobile /path/to/backend arela detect slices --json slices.json arela review slices ``` **Expected Output:** ``` Detected 4 optimal vertical slices: 1. authentication (23 files, cohesion: 87%) 2. workout (45 files, cohesion: 82%) 3. nutrition (31 files, cohesion: 79%) 4. social (28 files, cohesion: 75%) ``` **Data Required from Phase 1:** - Graph DB (imports, functions, dependencies) - File nodes and relationships - Coupling/cohesion calculation helpers ### Feature 2.2: API Contract Generator **Location:** `src/generate/contract.ts` (to be created) **Capabilities:** - Generate OpenAPI specs from code - Validate specs against implementation - Detect schema drift **CLI:** ```bash arela generate contract --from-code src/api/ arela validate contract openapi.yaml --against src/api/ ``` --- ## 8. Existing Infrastructure Supporting Slice Detection ### Available Graph Data The Graph DB already contains: - All file nodes with types (component, service, util, etc.) - Import relationships with weights - Function definitions and calls - API endpoints and calls - File-to-file dependency paths ### Available Analysis Functions ```typescript // From src/memory/graph.ts - Can be reused: await graphMemory.impact(filePath) // Gets fan-in/fan-out await graphMemory.findSlice(identifier) // Finds related files // From src/ingest/storage.ts - Can be reused: db.query(sql, params) // Direct SQL access db.getSummary() // Graph statistics // From src/analyze/coupling.ts - Can be reused: calculateCoupling() // Coupling metrics calculateCohesion() // Cohesion metrics ``` ### Utility Functions ```typescript // Available utilities: - picocolors for CLI output - better-sqlite3 for database - ts-morph for type extraction (already imported) - fast-glob for file scanning - fs-extra for file operations ``` ### Pattern from Existing Code The ingest module shows the pattern for handling graph data: ```typescript // From src/ingest/graph-builder.ts function buildGraph(analyses: FileAnalysis[], repoPath: string, db: GraphDB) { // Phase 1: Add all files // Phase 2: Resolve imports // Phase 3: Resolve function calls // Phase 4: Analyze relationships } ``` Similar pattern should be used for slice detection. --- ## 9. Key Dependencies ```json { "better-sqlite3": "^11.0.0", // Graph DB "ts-morph": "^21.0.0", // TypeScript AST "commander": "^12.0.0", // CLI "picocolors": "^1.1.0", // CLI colors "fast-glob": "^3.3.0", // File scanning "fs-extra": "^11.2.0", // File operations "zod": "^3.23.0", // Schema validation "yaml": "^2.4.0" // YAML parsing } ``` No external ML libraries are used—implement Louvain yourself or use a lightweight algorithm. --- ## 10. Development Checklist ### For Phase 2 (Slice Detection Implementation) - [ ] **Create slice detection module** (`src/detect/`) - [ ] `slices.ts` - Main public API - [ ] `clustering.ts` - Louvain algorithm - [ ] `scoring.ts` - Slice quality metrics - [ ] `types.ts` - Slice data structures - [ ] **Extend CLI** (`src/cli.ts`) - [ ] `arela detect slices` command - [ ] `arela review slices` command - [ ] Support for `--json` export - [ ] Multi-repo support - [ ] **Extend types** (`src/types.ts`) - [ ] Slice interfaces - [ ] Detection options - [ ] Recommendation types - [ ] **Add integration tests** - [ ] Test on real codebase - [ ] Test multi-repo detection - [ ] Validate clustering quality - [ ] **Update exports** (`src/index.ts`) - [ ] Export slice detection functions ### Building on Phase 1 The slice detection feature can directly leverage: 1. **GraphDB queries** - Import/dependency data already available 2. **Impact analysis** - Fan-in/fan-out calculations 3. **File nodes** - Types and metadata 4. **Coupling/cohesion** - Scoring algorithms from analyze/ 5. **CLI patterns** - Command registration in cli.ts --- ## Summary Arela is a well-structured Node.js/TypeScript application with: 1. **Clear Module Separation:** - Ingestion (files graph) - Memory (3-tier: vector/graph/audit) - Analysis (architecture, coupling, cohesion) - CLI (Commander.js) 2. **Solid Foundation (Phase 1 Complete):** - SQLite Graph DB with 8 tables - Universal codebase ingestion (15+ languages) - Tri-Memory system with 3 storage types - Multi-repo architecture analysis 3. **Ready for Phase 2:** - Graph data available for clustering - Coupling/cohesion algorithms exist - CLI patterns established - Type system in place 4. **Technology Stack:** - TypeScript with strict types - SQLite (better-sqlite3) for persistence - Commander.js for CLI - ts-morph for AST analysis - Zod for validation **The codebase is production-ready and well-organized for the slice detection feature development.**