shipdeck/tasks/002_dag_workflow_engine.md

Ship MVPs in 48 hours. Fix bugs in 30 seconds. The command deck for developers who ship.

# Task 002: DAG-Based Workflow Engine

## 1. Task Overview

### Task Title
**Title:** Build DAG workflow engine for parallel task orchestration

### Goal Statement
**Goal:** Create a Directed Acyclic Graph (DAG) based workflow engine that enables parallel execution of independent tasks while respecting dependencies. This is core to achieving the 48-hour guarantee through efficient parallelization.

---

## 2. Strategic Analysis

### Problem Context
To deliver MVPs in 48 hours, we need maximum parallelization. A DAG engine allows us to execute independent tasks simultaneously while ensuring dependent tasks wait for their prerequisites.

### Recommendation
Build a lightweight DAG executor that models the entire MVP process as nodes (tasks) and edges (dependencies), enabling optimal parallel execution.

---

## 3. Technical Requirements

### Functional Requirements
- Define workflows as DAG with nodes and dependencies
- Execute independent nodes in parallel
- Track node execution status (pending, running, completed, failed)
- Support node retry on failure
- Persist workflow state for resume capability
- Handle node outputs as inputs to dependent nodes

### Non-Functional Requirements
- **Performance:** Minimal overhead (<100ms per node transition)
- **Reliability:** State persistence for crash recovery
- **Scalability:** Handle 100+ nodes per workflow
- **Observability:** Real-time progress tracking

---

## 4. Implementation Plan

### Phase 1: Core DAG Structure
- [ ] Create lib/ultimate/dag/workflow.js
- [ ] Implement Node class with status tracking
- [ ] Implement Edge class for dependencies
- [ ] Create DAG validation (cycle detection)

### Phase 2: Execution Engine
- [ ] Create lib/ultimate/dag/executor.js
- [ ] Implement topological sort for execution order
- [ ] Add parallel execution for independent nodes
- [ ] Implement shared context for data passing

### Phase 3: State Management
- [ ] Add state persistence to filesystem/database
- [ ] Implement workflow resume from checkpoint
- [ ] Add progress tracking and reporting
- [ ] Create workflow status API

### Phase 4: Integration
- [ ] Integrate with agent-executor
- [ ] Create workflow templates for common patterns
- [ ] Add workflow visualization (optional)
- [ ] Test with multi-agent workflows

---

## 5. Success Criteria
- [ ] Can define and validate a DAG workflow
- [ ] Independent tasks execute in parallel
- [ ] Dependencies are respected (no race conditions)
- [ ] Workflow can resume after interruption
- [ ] 3x speed improvement vs sequential execution

---

## 6. Dependencies
- Task 001 (Anthropic API Integration) completed
- State storage solution (filesystem or Supabase)

---

## 7. Estimated Effort
**Priority:** P0 (Critical - Week 1)
**Complexity:** High
**Duration:** 3-4 days