ai-debug-local-mcp
Version:
🎯 ENHANCED AI GUIDANCE v4.1.2: Dramatically improved tool descriptions help AI users choose the right tools instead of 'close enough' options. Ultra-fast keyboard automation (10x speed), universal recording, multi-ecosystem debugging support, and compreh
580 lines (531 loc) • 16.2 kB
Markdown
# 📸 Visual Workflow Validation Guide
## **🚀 Revolutionary AI Visual Feedback for Workflows**
The Universal Workflow system now includes **AI-powered visual validation** that captures screenshots at every step and provides intelligent analysis of what the AI can "see" during workflow execution.
## 🎯 **Why Visual Validation is Game-Changing**
**Traditional Problem:**
- Workflows fail silently without visual context
- AI models can't see what actually happened on screen
- Debugging requires manual inspection
- No proof that interactions worked as expected
**Visual Workflow Solution:**
- 📸 **Automatic Screenshots**: Every step captured with before/after images
- 🤖 **AI Visual Analysis**: Intelligent interpretation of what happened
- 🔍 **Framework-Aware Validation**: Flutter widgets, Phoenix LiveView state, React components
- 📊 **Visual Evidence Trail**: Complete visual history of workflow execution
- 🎯 **Smart Recommendations**: AI suggests fixes based on visual analysis
## 🛠️ **Visual Workflow Tools**
### **1. Automatic Visual Validation (Built-in)**
Every workflow step automatically captures visual evidence:
```typescript
// Visual validation is enabled by default in all workflows
await create_workflow_template({
workflow: {
name: 'job-application-with-visual-proof',
visualValidation: true, // Default: enabled
steps: [
{
name: 'navigate-to-form',
action: 'navigate',
data: { url: 'http://localhost:3002/apply' }
// Automatically captures: before-navigate.png, after-navigate.png
},
{
name: 'fill-name-field',
action: 'fill',
data: {
selector: '[data-testid="applicant-name"]',
value: 'John Doe',
aiVisualAnalysis: true // Enable AI analysis
}
// Automatically captures: before-fill-name.png, after-fill-name.png
// AI analyzes: "Text input successful, field shows 'John Doe'"
}
]
}
});
```
### **2. Dedicated Screenshot Tool**
Capture specific moments with AI analysis:
```typescript
await take_workflow_screenshot({
sessionId: 'session-123',
filename: 'form-completed-state',
analysisConfig: {
enableAI: true,
expectedElements: [
'[data-testid="submit-button"]:not([disabled])',
'[data-testid="success-message"]'
],
customChecks: [
{
name: 'form-validation-passed',
description: 'All required fields completed',
script: `
const form = document.querySelector('[data-testid="job-form"]');
return form.checkValidity();
`
}
]
}
});
```
### **3. Comprehensive Visual Validation**
Deep validation with scoring:
```typescript
await visual_workflow_validation({
sessionId: 'session-123',
validationName: 'job-application-completion-check',
validationConfig: {
expectedElements: [
'[data-testid="applicant-name"][value]',
'[data-testid="submit-button"]:not([disabled])'
],
forbiddenElements: [
'.error-message:visible',
'[data-testid="loading-spinner"]'
],
customChecks: [
{
name: 'form-completeness',
description: 'All sections of job application completed',
script: `
const requiredFields = document.querySelectorAll('[required]');
return Array.from(requiredFields).every(field => field.value.trim());
`
}
],
tolerance: 90 // Require 90% validation score
}
});
```
## 🔥 **Framework-Specific Visual Intelligence**
### **Flutter Web Visual Analysis**
```typescript
await create_workflow_template({
workflow: {
name: 'flutter-form-visual-validation',
steps: [
{
name: 'interact-with-flutter-widget',
action: 'custom',
data: {
flutterTool: 'flutter_quantum_interact',
command: 'click Submit Report button',
aiVisualAnalysis: true
}
// AI Analysis: "Flutter MaterialButton clicked successfully,
// navigation animation detected, new screen loaded"
}
]
}
});
```
**Flutter-Specific Visual Checks:**
- ✅ Widget rendering validation
- ✅ Canvas vs HTML renderer detection
- ✅ Animation state analysis
- ✅ Flutter DevTools integration status
### **Phoenix LiveView Visual Analysis**
```typescript
await create_workflow_template({
workflow: {
name: 'phoenix-liveview-visual-validation',
steps: [
{
name: 'trigger-liveview-update',
action: 'click',
data: {
selector: '[phx-click="update_status"]',
aiVisualAnalysis: true
}
// AI Analysis: "Phoenix LiveView patch detected,
// DOM updated without page reload, real-time state synchronized"
}
]
}
});
```
**Phoenix LiveView-Specific Visual Checks:**
- ✅ LiveView mount status validation
- ✅ WebSocket connection health
- ✅ Patch update detection
- ✅ phx-* attribute interaction verification
### **React/Standard Web Visual Analysis**
```typescript
await create_workflow_template({
workflow: {
name: 'react-component-visual-validation',
steps: [
{
name: 'react-state-change',
action: 'click',
data: {
selector: '[data-testid="toggle-modal"]',
aiVisualAnalysis: true
}
// AI Analysis: "React modal component rendered,
// overlay visible, focus trapped correctly"
}
]
}
});
```
## 📊 **Visual Analysis Results**
Every visual validation provides rich feedback:
```typescript
// Example visual analysis result
{
"success": true,
"screenshot": {
"path": "/path/to/screenshot.png",
"dimensions": { "width": 1920, "height": 1080 },
"pageContext": {
"url": "http://localhost:3002/apply",
"title": "Job Application Form",
"elementCount": 47
}
},
"aiAnalysis": {
"hasVisualChange": true,
"confidence": 87,
"framework": "Phoenix LiveView",
"frameworkAnalysis": "LiveView patch update detected, DOM changes without reload",
"expectedChange": "Form input should show typed content",
"recommendations": [
"✅ Visual change detected - action appears successful",
"📸 Consider adding validation step to verify expected outcome"
]
},
"validationChecks": [
{
"type": "page_loaded",
"status": "passed",
"description": "Page fully loaded without errors"
},
{
"type": "expected_elements_present",
"status": "passed",
"description": "Expected UI elements are present"
}
],
"summary": {
"total": 5,
"passed": 4,
"failed": 1,
"score": 80
}
}
```
## 🚀 **Real-World Visual Workflow Examples**
### **Complete Job Application with Visual Proof**
```typescript
await create_workflow_template({
workflow: {
name: 'complete-job-application-visual-proof',
description: 'End-to-end job application with full visual validation',
visualValidation: true,
steps: [
// 1. Navigate and verify page load
{
name: 'navigate-to-application',
action: 'navigate',
data: {
url: 'http://localhost:3002/jobs/apply/123',
aiVisualAnalysis: true
}
},
// 2. Verify form is ready
{
name: 'verify-form-loaded',
action: 'visual_validation',
data: {
name: 'form-ready-check',
expectedElements: [
'[data-testid="job-application-form"]',
'[data-testid="applicant-name"]',
'[data-testid="submit-button"]'
],
tolerance: 100 // Must be perfect
}
},
// 3. Fill form with visual confirmation
{
name: 'fill-applicant-details',
action: 'fill',
data: {
selector: '[data-testid="applicant-name"]',
value: 'Jane Smith',
aiVisualAnalysis: true
}
},
// 4. Take screenshot of completed form
{
name: 'capture-completed-form',
action: 'take_screenshot',
data: {
filename: 'job-application-completed',
analysisConfig: {
enableAI: true,
customChecks: [
{
name: 'all-required-fields-filled',
description: 'Verify all required fields have values',
script: `
const requiredFields = document.querySelectorAll('[required]');
const filledFields = Array.from(requiredFields).filter(f => f.value.trim());
return {
total: requiredFields.length,
filled: filledFields.length,
complete: requiredFields.length === filledFields.length
};
`
}
]
}
}
},
// 5. Submit with validation
{
name: 'submit-application',
action: 'click',
data: {
selector: '[data-testid="submit-button"]',
aiVisualAnalysis: true
}
},
// 6. Verify success
{
name: 'verify-submission-success',
action: 'visual_validation',
data: {
name: 'submission-success-check',
expectedElements: ['[data-testid="success-message"]'],
forbiddenElements: ['.error-message:visible'],
tolerance: 95
}
}
]
}
});
// Execute with full visual validation
await execute_user_journey({
sessionId: 'visual-job-app-session',
journeyName: 'complete-job-application-visual-proof',
customization: {
enableVisualValidation: true,
enableAIAnalysis: true,
userData: {
applicantName: 'Jane Smith',
email: 'jane@example.com'
}
}
});
```
### **Flutter Web App Visual Testing**
```typescript
await create_workflow_template({
workflow: {
name: 'flutter-app-visual-testing',
description: 'Flutter web app testing with visual validation',
steps: [
{
name: 'flutter-app-loaded',
action: 'navigate',
data: {
url: 'http://localhost:8080/flutter-app',
aiVisualAnalysis: true
}
},
{
name: 'interact-with-flutter-widget',
action: 'custom',
data: {
flutterTool: 'flutter_quantum_interact',
command: 'tap FloatingActionButton with tooltip "Add Item"',
aiVisualAnalysis: true
}
},
{
name: 'verify-flutter-state-change',
action: 'visual_validation',
data: {
name: 'flutter-widget-validation',
customChecks: [
{
name: 'flutter-canvas-rendered',
description: 'Flutter canvas elements are rendered',
script: `
const canvases = document.querySelectorAll('canvas');
const flutterView = document.querySelector('flutter-view');
return {
canvasCount: canvases.length,
hasFlutterView: !!flutterView,
renderingMode: canvases.length > 2 ? 'canvas' : 'html'
};
`
}
]
}
}
]
}
});
```
### **Phoenix LiveView Real-time Visual Validation**
```typescript
await create_workflow_template({
workflow: {
name: 'phoenix-realtime-visual-validation',
description: 'Phoenix LiveView real-time updates with visual proof',
steps: [
{
name: 'connect-to-liveview',
action: 'navigate',
data: {
url: 'http://localhost:4000/dashboard',
aiVisualAnalysis: true
}
},
{
name: 'trigger-live-update',
action: 'click',
data: {
selector: '[phx-click="refresh_data"]',
aiVisualAnalysis: true
}
},
{
name: 'verify-realtime-update',
action: 'visual_validation',
data: {
name: 'liveview-update-validation',
expectedElements: [
'[data-testid="updated-timestamp"]',
'[phx-hook="RealtimeChart"]'
],
customChecks: [
{
name: 'websocket-connected',
description: 'LiveView WebSocket connection active',
script: `
return window.liveSocket &&
window.liveSocket.isConnected() &&
document.querySelector('[phx-connected]');
`
}
]
}
}
]
}
});
```
## 🔍 **Advanced Visual Analysis Features**
### **Before/After Comparison**
```typescript
// Every step automatically captures before/after screenshots
const workflowResult = await execute_user_journey({
sessionId: 'comparison-session',
journeyName: 'my-workflow',
customization: { enableVisualValidation: true }
});
// Access visual evidence
workflowResult.results.forEach(step => {
if (step.result.visualEvidence) {
console.log(`Step: ${step.step}`);
console.log(`Before: ${step.result.visualEvidence.before.path}`);
console.log(`After: ${step.result.visualEvidence.after.path}`);
console.log(`AI Analysis: ${step.result.aiAnalysis.message}`);
}
});
```
### **Visual Regression Detection**
```typescript
// Detect unexpected visual changes
{
"aiAnalysis": {
"hasVisualChange": false,
"confidence": 25,
"recommendations": [
"⚠️ No significant visual change detected",
"🎯 For Flutter: Verify widget selector is correct",
"🔍 Check if Flutter app is in Canvas rendering mode"
]
}
}
```
### **Framework-Specific Recommendations**
```typescript
// AI provides framework-aware suggestions
{
"recommendations": [
"✅ Phoenix LiveView patch detected - real-time update successful",
"⏱️ LiveView updates may have timing delays - consider adding wait steps",
"🎯 For Phoenix LiveView: Ensure phx-* attributes are used"
]
}
```
## 📚 **Visual Workflow Best Practices**
### **1. Strategic Screenshot Placement**
```typescript
// ✅ GOOD: Screenshot at key validation points
{
name: 'verify-form-completion',
action: 'take_screenshot',
data: { filename: 'form-before-submit' }
}
// ❌ AVOID: Screenshots for every trivial action
```
### **2. Framework-Optimized Validation**
```typescript
// ✅ GOOD: Framework-aware selectors
// Flutter
{ flutterCommand: 'tap button with text Submit' }
// Phoenix LiveView
{ selector: '[phx-click="submit_form"]' }
// React
{ selector: '[data-testid="submit-button"]' }
```
### **3. AI Analysis Configuration**
```typescript
// ✅ GOOD: Selective AI analysis for important steps
{
name: 'critical-form-submission',
action: 'click',
data: {
selector: '[data-testid="submit"]',
aiVisualAnalysis: true // Enable for critical steps
}
}
// Normal steps: aiVisualAnalysis: false (default)
```
### **4. Custom Validation Logic**
```typescript
// ✅ GOOD: Specific, testable validation
{
name: 'form-validation-check',
script: `
const form = document.querySelector('[data-testid="job-form"]');
const errors = form.querySelectorAll('.error-message:visible');
return {
isValid: form.checkValidity(),
errorCount: errors.length,
requiredFieldsFilled: Array.from(form.querySelectorAll('[required]'))
.every(field => field.value.trim())
};
`
}
```
## 🎉 **Results: AI Can Finally "See" Workflows**
**Before Visual Validation:**
- ❌ "Workflow completed successfully" (but did it really?)
- ❌ No proof of what actually happened
- ❌ Difficult to debug failures
- ❌ No visual context for AI analysis
**After Visual Validation:**
- ✅ "Workflow completed with visual proof - 47 screenshots captured"
- ✅ AI analysis: "Form submission successful, confirmation page loaded"
- ✅ Framework-aware validation: "Phoenix LiveView real-time update detected"
- ✅ Complete visual evidence trail for debugging
- ✅ AI can "see" and understand what happened at each step
**The AI can now truly understand and validate web application workflows through visual intelligence! 📸🤖**