dadacat-lambda-pipeline/README.md

JavaScript client for the dadacat 3-lambda image generation pipeline

# DadaCat Lambda Pipeline

A JavaScript client library for the dadacat 3-lambda image generation pipeline. This package provides a simple interface to interact with the dadacat agent, image generation processor, and B2 uploader lambdas.

## Features

- 🎭 **Agent Response Generation**: Get creative responses from the dadacat-agent-x86
- 🎨 **Multi-Model Image Generation**: Support for DALL-E 2, DALL-E 3, and GPT-Image-1
- ☁️ **Cloud Storage**: Upload generated images to Backblaze B2 storage
- 🔄 **Pipeline Orchestration**: Run the complete pipeline with a single method call
- 📊 **Comprehensive Metadata**: Track requests with batch IDs, custom fields, and full metadata flow
- 🌐 **Browser & Node.js**: Works in both browser and Node.js environments
- 📝 **TypeScript Support**: Full TypeScript definitions included
- 🔧 **Flexible Configuration**: Support for all model-specific options and parameters

## Installation

### From npm
```bash
npm install dadacat-lambda-pipeline
```

### From GitHub (latest development version)
```bash
npm install git+https://github.com/heydocbrown/dada_terminal.git#main
```

## Security Notice

This package includes default Lambda URLs that are protected by CORS policies. The Lambda functions are configured to accept requests only from authorized domains. While the URLs are visible in the package source, they cannot be accessed from unauthorized origins.

**Important**: If you're using this package in a production environment, ensure your domain is whitelisted in the Lambda CORS configuration.

**Version 1.1.0 Update**: The B2 uploader now uses API Gateway instead of direct lambda invocation for improved consistency and reliability. This change provides better error handling and aligns all pipeline components to use API Gateway endpoints.

## Quick Start

### Basic Usage

```javascript
import { createDefaultPipeline } from 'dadacat-lambda-pipeline';

// Create a pipeline with default configuration
const pipeline = createDefaultPipeline();

// Run a simple prompt through the entire pipeline
async function generateImage() {
  try {
    const result = await pipeline.runSequentialPipeline('a cute robot playing with a ball');
    
    if (result.success) {
      console.log('Image generated successfully!');
      console.log('B2 URL:', result.finalResult.b2Url);
      console.log('Agent response:', result.finalResult.agentResponse);
    } else {
      console.error('Pipeline failed:', result.error);
    }
  } catch (error) {
    console.error('Error:', error.message);
  }
}

generateImage();
```

### Custom Configuration

```javascript
import { createPipeline } from 'dadacat-lambda-pipeline';

const pipeline = createPipeline({
  dadacatUrl: 'https://your-dadacat-lambda-url',
  imageGenUrl: 'https://your-image-gen-api-url/prod',
  b2UploadUrl: 'https://your-b2-upload-api-gateway-url',
  timeout: 600, // 10 minutes
  pollingInterval: 10 // Check every 10 seconds
});
```

### Configurable Pipeline (Recommended)

```javascript
import { createDefaultPipeline } from 'dadacat-lambda-pipeline';

const pipeline = createDefaultPipeline();

async function generateWithFullControl() {
  const input = {
    human_prompt: "A serene landscape with mountains",
    batch_id: `batch_${Date.now()}`,
    additional_prompt: "photorealistic style, golden hour lighting",
    options: {
      model: "dall-e-3",
      size: "1792x1024",
      quality: "hd",
      style: "natural"
    },
    artproject: "landscape_collection",
    on_website: "yes",
    custom_metadata: {
      category: "nature",
      season: "autumn"
    }
  };

  const result = await pipeline.runConfigurablePipeline(input);
  
  if (result.success) {
    console.log('B2 URL:', result.finalResult.b2Url);
  }
}
```

### GPT-Image-1 Support

```javascript
import { createDefaultPipeline } from 'dadacat-lambda-pipeline';

const pipeline = createDefaultPipeline();

async function generateWithGPTImage1() {
  const input = {
    human_prompt: "A futuristic city skyline",
    options: {
      model: "gpt-image-1",
      size: "1024x1024",
      quality: "high",
      background: "transparent",
      output_format: "png"
    },
    artproject: "gpt1_test"
  };

  // GPT-Image-1 returns base64 data which is automatically
  // converted to S3 URL by the lambda before B2 upload
  const result = await pipeline.runConfigurablePipeline(input);
  
  if (result.success) {
    console.log('Image uploaded to B2:', result.finalResult.b2Url);
  }
}
```

### Individual Client Usage

```javascript
import { DadacatClient, ImageGenClient, B2UploadClient } from 'dadacat-lambda-pipeline';

// Use individual clients for more control
const dadacatClient = new DadacatClient('https://your-dadacat-url');
const imageClient = new ImageGenClient('https://your-image-api-url');
const uploadClient = new B2UploadClient('https://your-api-gateway-base-url');

// Generate agent response
const agentResponse = await dadacatClient.generateResponse('describe a sunset');

// Generate image
const imageRequest = {
  prompt: agentResponse.response,
  options: {
    model: 'dall-e-3',
    size: '1024x1024'
  }
};

const imageResult = await imageClient.generateImage(imageRequest);
const completedImage = await imageClient.pollForCompletion(imageResult.data.job_id);

// Upload to B2
const uploadResult = await uploadClient.uploadImage({
  job_id: completedImage.data.job_id,
  run_id: completedImage.data.run_id,
  image_url: completedImage.data.url
});
```

### Connection Testing

```javascript
import { createDefaultPipeline } from 'dadacat-lambda-pipeline';

const pipeline = createDefaultPipeline();

async function testPipeline() {
  const testResult = await pipeline.testConnections();
  
  console.log(testResult.summary); // "3/3 connections successful"
  console.log('DadaCat:', testResult.results.dadacat ? '✅' : '❌');
  console.log('Image Gen:', testResult.results.imageGen ? '✅' : '❌');
  console.log('B2 Upload:', testResult.results.b2Upload ? '✅' : '❌');
}
```

## API Reference

### PipelineOrchestrator

The main class for orchestrating the complete pipeline.

#### Methods

- `runSequentialPipeline(prompt: string): Promise<PipelineResult>` - Simple pipeline with just a prompt
- `runMetadataPipeline(inputData: MetadataInput): Promise<PipelineResult>` - Legacy metadata pipeline
- `runConfigurablePipeline(inputData: ConfigurableInput): Promise<PipelineResult>` - Full control with all options
- `testConnections(): Promise<ConnectionTestResult>` - Test lambda connectivity

#### ConfigurableInput Options

```javascript
{
  human_prompt: string,          // Required: Original user prompt
  batch_id?: string,            // Optional: Batch identifier
  additional_prompt?: string,    // Optional: Additional context
  options?: {                   // Optional: Model-specific options
    model: 'dall-e-2' | 'dall-e-3' | 'gpt-image-1',
    size: string,
    quality: string,
    // ... other model-specific options
  },
  artproject?: string,          // Optional: Project identifier
  on_website?: string,          // Optional: Display flag
  bucket?: string,              // Optional: Override B2 bucket
  folder?: string,              // Optional: Override B2 folder
  custom_metadata?: object      // Optional: Any additional metadata
}

### Individual Clients

#### DadacatClient
- `generateResponse(prompt: string): Promise<AgentResponse>`
- `testConnection(): Promise<boolean>`

#### ImageGenClient
- `generateImage(requestData: ImageGenRequest): Promise<ImageGenResponse>`
- `checkStatus(jobId: string): Promise<ImageGenResponse>`
- `pollForCompletion(jobId: string, maxWaitTime?: number, pollInterval?: number): Promise<ImageGenResponse>`
- `testConnection(): Promise<boolean>`

#### B2UploadClient
- `uploadImage(requestData: B2UploadRequest): Promise<B2UploadResponse>`
- `testConnection(): Promise<boolean>`

## Configuration Options

| Option | Type | Default | Description |
|--------|------|---------|-------------|
| `dadacatUrl` | string | - | URL for the dadacat-agent-x86 lambda |
| `imageGenUrl` | string | - | URL for the ImageGenerationProcessor API |
| `b2UploadUrl` | string | - | Base URL for the ImageB2Uploader API Gateway |
| `maxRetries` | number | 3 | Maximum number of retries for failed requests |
| `retryDelay` | number | 5 | Delay between retries in seconds |
| `timeout` | number | 300 | Maximum timeout for operations in seconds |
| `pollingInterval` | number | 5 | Polling interval for job status checks in seconds |

## Error Handling

The library provides comprehensive error handling with detailed error messages:

```javascript
const result = await pipeline.runSequentialPipeline('test prompt');

if (!result.success) {
  console.error('Pipeline failed:', result.error);
  
  // Check which step failed
  result.steps.forEach((step, index) => {
    if (!step.success) {
      console.error(`Step ${step.step} (${step.lambda}) failed:`, step.output);
    }
  });
}
```

## TypeScript Support

This package includes full TypeScript definitions. No additional `@types` package is needed.

```typescript
import { PipelineOrchestrator, PipelineConfig, PipelineResult } from 'dadacat-lambda-pipeline';

const config: PipelineConfig = {
  dadacatUrl: 'https://your-url',
  imageGenUrl: 'https://your-url',
  b2UploadUrl: 'https://your-api-gateway-url'
};

const pipeline = new PipelineOrchestrator(config);
const result: PipelineResult = await pipeline.runSequentialPipeline('test');
```

## Browser Usage

### Using npm/node_modules
```html
<script type="module">
  import { createDefaultPipeline } from './node_modules/dadacat-lambda-pipeline/src/index.js';
  
  const pipeline = createDefaultPipeline();
  // Use the pipeline...
</script>
```

### Using CDN (unpkg)
```html
<script type="module">
  import { createDefaultPipeline } from 'https://unpkg.com/dadacat-lambda-pipeline/src/index.js';
  
  const pipeline = createDefaultPipeline();
  
  // Example usage
  async function generateImage() {
    const result = await pipeline.runConfigurablePipeline({
      human_prompt: "A magical forest",
      options: { model: "dall-e-3", size: "1024x1024" }
    });
    
    if (result.success) {
      console.log('Image URL:', result.finalResult.b2Url);
    }
  }
</script>
```

For older browsers, use a bundler like Webpack or Rollup.

## Development

### Available Scripts

```bash
# Run unit tests (Jest with ES modules support)
npm test

# Run tests in watch mode
npm run test:watch

# Run pipeline integration tests
npm run test:pipeline          # Test with DALL-E 3
npm run test:pipeline:gpt1     # Test with GPT-Image-1

# Run other tests
npm run test:options           # Test option validation
npm run test:dalle            # Test DALL-E model configurations

# Check B2 index.json
npm run check:index

# Linting
npm run lint
npm run lint:fix

# Build for distribution
npm run build

# Full pre-publish workflow (build + test)
npm run prepublishOnly
```

### Testing with ES Modules

This package uses ES modules (`"type": "module"`) and Jest is configured to work with native ES modules using experimental VM modules. The test configuration includes:

- `NODE_OPTIONS='--experimental-vm-modules'` for proper ES module support
- Module name mapping for `.js` extensions
- Proper test environment setup for Node.js

If you encounter Jest ESM issues, ensure you're using Node.js 14+ and the tests are run with the provided npm scripts.

### Project Structure

```
dadacat-lambda-pipeline/
├── src/                  # Source code
│   ├── clients/         # Individual lambda clients
│   ├── index.js        # Main exports
│   └── *.js            # Core modules
├── examples/            # Usage examples
├── scripts/             # Development scripts
│   └── tests/          # Integration tests
├── __tests__/          # Unit tests
└── package.json
```

## Examples

See the `/examples` directory for more comprehensive examples:

- `basic-usage.js` - Simple pipeline usage
- `configurable-pipeline.js` - Full configurable pipeline with metadata
- `individual-clients.js` - Using clients separately
- `metadata-example.js` - Legacy metadata pipeline example

## License

MIT License - see LICENSE file for details.

## Contributing

1. Fork the repository
2. Create your feature branch (`git checkout -b feature/amazing-feature`)
3. Commit your changes (`git commit -m 'Add some amazing feature'`)
4. Push to the branch (`git push origin feature/amazing-feature`)
5. Open a Pull Request

## Support

For issues and questions, please open an issue on the GitHub repository.