@viji-dev/core/README.md

Universal execution engine for Viji Creative scenes

# Viji Core Package (`@viji-dev/core`)

**Universal execution engine for Viji Creative scenes**

A powerful, secure, and feature-rich JavaScript/TypeScript library that provides the foundation for creative scene execution across all Viji platform contexts. The core offers identical IFrame + WebWorker execution with comprehensive parameter management, audio/video analysis, user interaction handling, and performance optimization.

## 🚀 Features

### ✅ **Core Execution Engine**
- **Secure IFrame + WebWorker Architecture**: Complete isolation with controlled communication
- **Multi-Instance Support**: Concurrent instances for main scenes and previews
- **Automatic Resource Management**: Memory leak prevention and cleanup

### ✅ **Parameter System**
- **Declarative Parameter Definition**: Define parameters once with automatic UI generation
- **Proxy-Based Access**: Fast parameter access in render loops
- **Category-Based Organization**: Audio, video, interaction, and general parameters
- **Real-time Validation**: Type safety and range checking
- **Capability-Aware UI**: Parameters shown based on active features

### ✅ **Audio Analysis**
- **Real-time Audio Processing**: Volume, frequency analysis, and beat detection
- **Custom Frequency Bands**: Bass, mid, treble, and custom band analysis
- **Multiple Input Sources**: Microphone, audio files, and screen capture
- **Audio-Reactive Scenes**: Make scenes respond to audio input

### ✅ **Video Analysis**
- **Real-time Video Processing**: Frame analysis in separate WebWorker
- **Multiple Input Sources**: Camera, video files, and screen capture
- **Video-Reactive Scenes**: Make scenes respond to video motion and brightness
- **Frame Data Access**: Raw video frame data for custom analysis

### ✅ **User Interaction**
- **Mouse Tracking**: Position, buttons, movement, and scroll wheel
- **Keyboard Input**: Key states, modifiers, and event handling
- **Touch Support**: Multi-touch with gesture detection
- **Canvas-Coordinate Mapping**: Accurate input positioning

### ✅ **Performance Optimization**
- **Configurable Frame Rates**: Full (60fps) or half (30fps) modes
- **Resolution Scaling**: Fractional or explicit canvas dimensions
- **Adaptive Performance**: Automatic optimization based on hardware
- **Memory Management**: Efficient resource pooling and cleanup

## 📦 Installation

```bash
npm install @viji-dev/core
```

## 🎯 Quick Start

### Basic Scene Creation

```typescript
import { VijiCore } from '@viji-dev/core';

// Artist scene code
const sceneCode = `
// Define parameters using helper functions
const color = viji.color('#ff6b6b', {
  label: 'Shape Color',
  description: 'Color of the animated shape',
  group: 'appearance'
});

const size = viji.slider(50, {
  min: 10,
  max: 150,
  step: 5,
  label: 'Shape Size',
  description: 'Size of the animated shape',
  group: 'appearance'
});

const speed = viji.slider(1.0, {
  min: 0.1,
  max: 3.0,
  step: 0.1,
  label: 'Animation Speed',
  description: 'Speed of the animation',
  group: 'animation'
});

// Main render function
function render(viji) {
  const ctx = viji.useContext('2d');
  
  // Clear canvas
  ctx.fillStyle = '#2c3e50';
  ctx.fillRect(0, 0, viji.width, viji.height);
  
  // Animated shape
  const time = viji.time * speed.value;
  const x = viji.width / 2 + Math.sin(time) * 100;
  const y = viji.height / 2 + Math.cos(time) * 100;
  
  ctx.fillStyle = color.value;
  ctx.beginPath();
  ctx.arc(x, y, size.value / 2, 0, Math.PI * 2);
  ctx.fill();
}
`;

// Create core instance
const core = new VijiCore({
  hostContainer: document.getElementById('scene-container'),
  sceneCode: sceneCode,
  frameRateMode: 'full',
  allowUserInteraction: true
});

// Initialize and start rendering
await core.initialize();
console.log('Scene is running!');
```

## 🔧 Integration API

### Core Configuration

The `VijiCoreConfig` interface defines all available configuration options:

```typescript
interface VijiCoreConfig {
  // Required configuration
  hostContainer: HTMLElement;           // Container element for the scene
  sceneCode: string;                    // Artist JavaScript code with render function
  
  // Performance configuration
  frameRateMode?: 'full' | 'half';      // 'full' = 60fps, 'half' = 30fps
  autoOptimize?: boolean;               // Enable automatic performance optimization
  
  // Input streams
  audioStream?: MediaStream;            // Audio input for analysis
  videoStream?: MediaStream;            // Video input for analysis
  
  // Audio analysis configuration
  analysisConfig?: {
    fftSize?: number;                   // FFT size for frequency analysis (default: 2048)
    smoothing?: number;                 // Smoothing factor 0-1 (default: 0.8)
    frequencyBands?: FrequencyBand[];   // Custom frequency bands
    beatDetection?: boolean;            // Enable beat detection
    onsetDetection?: boolean;           // Enable onset detection
  };

  // Parameter system
  parameters?: ParameterGroup[];        // Initial parameter values
  
  // Feature toggles
  noInputs?: boolean;                   // Disable all input processing
  allowUserInteraction?: boolean;       // Enable mouse/keyboard/touch events
}
```

### Instance Management

#### Creation and Initialization

```typescript
// Create core instance
const core = new VijiCore({
  hostContainer: document.getElementById('scene-container'),
  sceneCode: sceneCode,
  frameRateMode: 'full',
  allowUserInteraction: true
});

// Initialize the core (required before use)
await core.initialize();

// Check if core is ready for operations
  if (core.ready) {
  console.log('Core is ready for use');
}

// Get current configuration
const config = core.configuration;
console.log('Current frame rate mode:', config.frameRateMode);
```

#### Performance Control

```typescript
// Frame rate control
await core.setFullFrameRate();          // Set to 60fps mode
await core.setHalfFrameRate();          // Set to 30fps mode

// Resolution control
await core.setResolution(0.75);         // Set to 75% of container size
await core.setResolution(0.5);          // Set to 50% for performance
await core.updateResolution();          // Auto-detect container size changes

// Get performance statistics
const stats = core.getStats();
console.log('Current FPS:', stats.frameRate.effectiveRefreshRate);
console.log('Canvas size:', stats.resolution);
console.log('Scale factor:', stats.scale);
console.log('Parameter count:', stats.parameterCount);
```

#### Debug and Development

```typescript
// Enable debug logging
core.setDebugMode(true);

// Check debug mode status
const isDebugEnabled = core.getDebugMode();

// Debug mode provides detailed logging for:
// - Initialization process
// - Communication between components
// - Parameter system operations
// - Audio/video stream processing
// - Performance statistics
```

### Parameter Management

The parameter system provides a powerful way to create interactive scenes with automatic UI generation.

#### Parameter Definition and Access

```typescript
// Listen for parameter definitions from artist code
core.onParametersDefined((groups) => {
  console.log('Parameters available:', groups);
  
  // Each group contains:
  // - groupName: string
  // - category: 'audio' | 'video' | 'interaction' | 'general'
  // - description: string
  // - parameters: Record<string, ParameterDefinition>
  
  // Generate UI based on parameter groups
  generateParameterUI(groups);
});

// Set individual parameter values
await core.setParameter('color', '#ff0000');
await core.setParameter('size', 75);
await core.setParameter('enabled', true);

// Set multiple parameters efficiently
await core.setParameters({
  'color': '#00ff00',
  'size': 100,
  'speed': 2.0,
  'enabled': false
});

// Get current parameter values
const values = core.getParameterValues();
const color = core.getParameter('color');

// Listen for parameter changes
core.onParameterChange('size', (value) => {
  console.log('Size parameter changed to:', value);
});

// Listen for parameter errors
core.onParameterError((error) => {
  console.error('Parameter error:', error.message);
  console.error('Error code:', error.code);
});
```

#### Capability-Aware Parameters

```typescript
// Get all parameter groups
const allGroups = core.getParameterGroups();

// Get parameter groups filtered by active capabilities
const visibleGroups = core.getVisibleParameterGroups();

// Check current capabilities
const capabilities = core.getCapabilities();
console.log('Audio available:', capabilities.hasAudio);
console.log('Video available:', capabilities.hasVideo);
console.log('Interaction enabled:', capabilities.hasInteraction);

// Check if specific parameter category is active
const isAudioActive = core.isCategoryActive('audio');
const isVideoActive = core.isCategoryActive('video');

// Parameters are automatically categorized:
// - 'audio': Only shown when audio stream is connected
// - 'video': Only shown when video stream is connected  
// - 'interaction': Only shown when user interaction is enabled
// - 'general': Always available
```

### Audio and Video Integration

#### Audio Stream Management

```typescript
// Set audio stream for analysis
const audioStream = await navigator.mediaDevices.getUserMedia({ 
  audio: {
    echoCancellation: false,
    noiseSuppression: false,
    autoGainControl: false
  } 
});
await core.setAudioStream(audioStream);

// Configure audio analysis
await core.setAudioAnalysisConfig({
  fftSize: 2048,        // Higher values = better frequency resolution
  smoothing: 0.8        // 0 = no smoothing, 1 = maximum smoothing
});

// Get current audio stream
const currentStream = core.getAudioStream();

// Disconnect audio
await core.setAudioStream(null);
```

#### Video Stream Management

```typescript
// Set video stream for analysis
const videoStream = await navigator.mediaDevices.getUserMedia({ 
  video: {
    width: { ideal: 640 },
    height: { ideal: 480 },
    frameRate: { ideal: 30 }
  } 
});
await core.setVideoStream(videoStream);

// Video analysis includes:
// - Real-time frame processing
// - Frame data access for custom analysis
// - Brightness and motion detection
// - Custom computer vision processing

// Disconnect video
await core.setVideoStream(null);
```

#### Interaction Management

```typescript
// Enable or disable user interactions at runtime
await core.setInteractionEnabled(true);  // Enable mouse, keyboard, and touch
await core.setInteractionEnabled(false); // Disable all interactions

// Get current interaction state
const isInteractionEnabled = core.getInteractionEnabled();

// Interaction state affects:
// - Mouse, keyboard, and touch event processing
// - Parameter visibility (interaction category parameters)
// - Scene behavior that depends on user input

// Note: Interaction state is separate from initialization config
// You can toggle interactions regardless of initial allowUserInteraction value
// The interaction system is always available for runtime control
```

#### Capability Change Monitoring

```typescript
// Listen for capability changes
core.onCapabilitiesChange((capabilities) => {
  console.log('Capabilities updated:', capabilities);
  
  // Update UI based on new capabilities
  if (capabilities.hasAudio) {
    showAudioControls();
  } else {
    hideAudioControls();
  }
  
  if (capabilities.hasVideo) {
    showVideoControls();
  } else {
    hideVideoControls();
  }
  
  if (capabilities.hasInteraction) {
    showInteractionControls();
  } else {
    hideInteractionControls();
  }
});
```

### Event Handling and Lifecycle

#### Core Lifecycle Events

```typescript
// Core is ready for operations
if (core.ready) {
  // All systems initialized and running
  console.log('Core is fully operational');
}

// Check if parameters are initialized
if (core.parametersReady) {
  // Parameter system is ready
  console.log('Parameters are available');
  }
```

#### Cleanup and Resource Management

```typescript
// Destroy instance and clean up all resources
await core.destroy();

// This automatically:
// - Stops all rendering loops
// - Disconnects audio/video streams
// - Cleans up WebWorker and IFrame
// - Releases all event listeners
// - Clears parameter system
// - Frees memory resources
```

## 🎨 Artist API

The artist API provides a comprehensive set of tools for creating interactive, audio-reactive, and video-responsive scenes.

### Canvas and Rendering

```typescript
function render(viji) {
  // Get canvas contexts
  const ctx = viji.useContext('2d');     // 2D rendering context
  const gl = viji.useContext('webgl');   // WebGL rendering context
  
  // Canvas properties
  viji.canvas;        // OffscreenCanvas object
  viji.width;         // Canvas width in pixels
  viji.height;        // Canvas height in pixels
  viji.pixelRatio;    // Device pixel ratio for crisp rendering
  
  // Example: Draw a responsive circle
  const centerX = viji.width / 2;
  const centerY = viji.height / 2;
  const radius = Math.min(viji.width, viji.height) * 0.1;
  
  ctx.fillStyle = '#ff6b6b';
  ctx.beginPath();
  ctx.arc(centerX, centerY, radius, 0, Math.PI * 2);
  ctx.fill();
}
```

### Timing Information

The timing system provides FPS-independent timing data for smooth animations:

```typescript
function render(viji) {
  // Timing data (FPS independent)
  viji.time;          // Elapsed time in seconds since scene start
  viji.deltaTime;     // Time since last frame in seconds
  viji.frameCount;    // Total number of frames rendered
  viji.fps;           // Current frames per second
  
  // Example: Smooth animation regardless of frame rate
  const animationSpeed = 2.0; // rotations per second
  const rotation = (viji.time * animationSpeed * Math.PI * 2) % (Math.PI * 2);
  
  ctx.save();
  ctx.translate(viji.width / 2, viji.height / 2);
  ctx.rotate(rotation);
  ctx.fillRect(-25, -25, 50, 50);
  ctx.restore();
}
```

### Parameter System

The parameter system allows artists to define interactive parameters that automatically generate UI controls.

#### Parameter Definition

```typescript
// Define parameters (call once outside render loop)
const color = viji.color('#ff6b6b', {
  label: 'Primary Color',
  description: 'Main color for shapes',
  group: 'appearance',
  category: 'general'
});

const size = viji.slider(50, {
  min: 10,
  max: 150,
  step: 5,
  label: 'Shape Size',
  description: 'Size of shapes in pixels',
  group: 'appearance',
  category: 'general'
});

const speed = viji.slider(1.0, {
  min: 0.1,
  max: 3.0,
  step: 0.1,
  label: 'Animation Speed',
  description: 'Speed of animation in rotations per second',
  group: 'animation',
  category: 'general'
});

const useAudio = viji.toggle(false, {
  label: 'Audio Reactive',
  description: 'Make shapes react to audio input',
  group: 'audio',
  category: 'audio'
});

const shapeType = viji.select('circle', {
  options: ['circle', 'square', 'triangle', 'star'],
  label: 'Shape Type',
  description: 'Type of shape to draw',
  group: 'appearance',
  category: 'general'
});

const title = viji.text('My Scene', {
  label: 'Scene Title',
  description: 'Title displayed in the scene',
  group: 'text',
  category: 'general',
  maxLength: 50
});

const particleCount = viji.number(5, {
  min: 1,
  max: 20,
  step: 1,
  label: 'Particle Count',
  description: 'Number of particles to render',
  group: 'animation',
  category: 'general'
});
```

#### Parameter Usage in Render Loop

```typescript
function render(viji) {
  const ctx = viji.useContext('2d');
  
  // Fast parameter access (proxy-based)
  ctx.fillStyle = color.value;           // Get current color value
  const radius = size.value / 2;         // Get current size value
  const animationSpeed = speed.value;    // Get current speed value
  
  // Clear canvas
  ctx.fillStyle = '#2c3e50';
  ctx.fillRect(0, 0, viji.width, viji.height);
  
  // Draw title
  ctx.fillStyle = 'white';
  ctx.font = '20px Arial';
  ctx.textAlign = 'center';
  ctx.fillText(title.value, viji.width / 2, 30);
  
  // Draw particles
  for (let i = 0; i < particleCount.value; i++) {
    const angle = (i / particleCount.value) * Math.PI * 2 + (viji.time * animationSpeed);
    const x = viji.width / 2 + Math.cos(angle) * 100;
    const y = viji.height / 2 + Math.sin(angle) * 100;
    
    ctx.fillStyle = color.value;
    ctx.beginPath();
    
    switch (shapeType.value) {
      case 'circle':
        ctx.arc(x, y, radius, 0, Math.PI * 2);
        break;
      case 'square':
        ctx.rect(x - radius, y - radius, radius * 2, radius * 2);
        break;
      case 'triangle':
        ctx.moveTo(x, y - radius);
        ctx.lineTo(x - radius, y + radius);
        ctx.lineTo(x + radius, y + radius);
        ctx.closePath();
        break;
    }
    
    ctx.fill();
  }
}
```

### Audio Analysis

The audio system provides real-time analysis of audio input with comprehensive frequency and volume data.

#### Audio API Overview

```typescript
function render(viji) {
  const audio = viji.audio;
  
  if (audio.isConnected) {
    // Volume analysis
    const volume = audio.volume.rms;     // 0-1 RMS volume (true volume)
    const peak = audio.volume.peak;      // 0-1 peak volume (maximum amplitude)
    
    // Frequency bands (0-1 values)
    const bass = audio.bands.bass;       // 60-250 Hz
    const mid = audio.bands.mid;         // 500-2000 Hz
    const treble = audio.bands.treble;   // 2000-20000 Hz
    
    // Extended frequency bands
    const subBass = audio.bands.subBass;     // 20-60 Hz
    const lowMid = audio.bands.lowMid;       // 250-500 Hz
    const highMid = audio.bands.highMid;     // 2000-4000 Hz
    const presence = audio.bands.presence;   // 4000-6000 Hz
    const brilliance = audio.bands.brilliance; // 6000-20000 Hz
    
    // Beat detection
    if (audio.beat?.isKick) {
      // Kick drum detected
      console.log('Kick detected!');
    }
    
    // Raw frequency data (0-255 values)
    const frequencyData = audio.getFrequencyData();
    
    // Example: Audio-reactive animation
    const scale = 1 + (volume * 2); // Scale based on volume
    const hue = (bass * 360); // Color based on bass
    
    ctx.save();
    ctx.translate(viji.width / 2, viji.height / 2);
    ctx.scale(scale, scale);
    ctx.fillStyle = `hsl(${hue}, 70%, 60%)`;
    ctx.fillRect(-25, -25, 50, 50);
    ctx.restore();
  }
}
```

#### Audio-Reactive Scene Example

```typescript
// Define audio-reactive parameters
const audioReactive = viji.toggle(true, {
  label: 'Audio Reactive',
  description: 'Make shapes react to audio',
  group: 'audio',
  category: 'audio'
});

const volumeSensitivity = viji.slider(1.0, {
  min: 0.1,
  max: 5.0,
  step: 0.1,
  label: 'Volume Sensitivity',
  description: 'How sensitive shapes are to volume',
  group: 'audio',
  category: 'audio'
});

const bassReactivity = viji.slider(1.0, {
  min: 0,
  max: 3.0,
  step: 0.1,
  label: 'Bass Reactivity',
  description: 'How much shapes react to bass',
  group: 'audio',
  category: 'audio'
});

function render(viji) {
  const ctx = viji.useContext('2d');
  const audio = viji.audio;
  
  // Clear canvas
  ctx.fillStyle = '#2c3e50';
  ctx.fillRect(0, 0, viji.width, viji.height);
  
  if (audioReactive.value && audio.isConnected) {
    // Audio-reactive animation
    const volume = audio.volume.rms * volumeSensitivity.value;
    const bass = audio.bands.bass * bassReactivity.value;
    
    // Scale based on volume
    const scale = 1 + volume;
    
    // Color based on bass
    const hue = 200 + (bass * 160); // Blue to purple range
    
    // Position based on frequency distribution
    const x = viji.width * (audio.bands.mid + audio.bands.treble) / 2;
    const y = viji.height * (1 - audio.bands.bass);
    
    ctx.save();
    ctx.translate(x, y);
    ctx.scale(scale, scale);
    ctx.fillStyle = `hsl(${hue}, 80%, 60%)`;
    ctx.beginPath();
    ctx.arc(0, 0, 30, 0, Math.PI * 2);
    ctx.fill();
    ctx.restore();
  }
}
```

### Video Analysis

The video system provides real-time video frame analysis with frame data access for custom processing.

#### Video API Overview

```typescript
function render(viji) {
  const video = viji.video;
  
  if (video.isConnected) {
    // Video properties
    const frameWidth = video.frameWidth;
    const frameHeight = video.frameHeight;
    const frameRate = video.frameRate;
    
    // Current video frame (OffscreenCanvas)
    if (video.currentFrame) {
      // Draw video frame as background
      ctx.globalAlpha = 0.3;
      ctx.drawImage(video.currentFrame, 0, 0, viji.width, viji.height);
      ctx.globalAlpha = 1.0;
    }
    
    // Frame data for custom analysis
    const frameData = video.getFrameData();
    
    // Example: Custom video analysis
    if (frameData) {
      // Access raw pixel data for custom processing
      const imageData = frameData.data;
      const width = frameData.width;
      const height = frameData.height;
      
      // Example: Calculate average brightness
      let totalBrightness = 0;
      for (let i = 0; i < imageData.length; i += 4) {
        const r = imageData[i];
        const g = imageData[i + 1];
        const b = imageData[i + 2];
        totalBrightness += (r + g + b) / 3;
      }
      const averageBrightness = totalBrightness / (imageData.length / 4);
      
      // Use brightness for effects
      const brightness = averageBrightness / 255; // Normalize to 0-1
      
      // Create brightness-reactive animation
      ctx.fillStyle = `rgba(255, 255, 255, ${brightness * 0.5})`;
      ctx.fillRect(0, 0, viji.width, viji.height);
    }
  }
}
```

#### Video-Reactive Scene Example

```typescript
// Define video-reactive parameters
const videoReactive = viji.toggle(true, {
  label: 'Video Reactive',
  description: 'Make shapes react to video',
  group: 'video',
  category: 'video'
});

const motionSensitivity = viji.slider(1.0, {
  min: 0.1,
  max: 3.0,
  step: 0.1,
  label: 'Motion Sensitivity',
  description: 'How sensitive shapes are to video changes',
  group: 'video',
  category: 'video'
});

function render(viji) {
  const ctx = viji.useContext('2d');
  const video = viji.video;
  
  if (videoReactive.value && video.isConnected) {
    // Video-reactive animation using frame data
    const frameData = video.getFrameData();
    
    if (frameData) {
      // Simple motion detection (compare with previous frame)
      // This is a basic example - you can implement more sophisticated analysis
      const imageData = frameData.data;
      let motionEnergy = 0;
      
      // Calculate motion energy (simplified)
      for (let i = 0; i < imageData.length; i += 4) {
        const brightness = (imageData[i] + imageData[i + 1] + imageData[i + 2]) / 3;
        motionEnergy += brightness;
      }
      
      const normalizedMotion = (motionEnergy / (imageData.length / 4)) / 255;
      const scale = 1 + (normalizedMotion * motionSensitivity.value);
      
      // Create motion-reactive shapes
      ctx.save();
      ctx.translate(viji.width / 2, viji.height / 2);
      ctx.scale(scale, scale);
      ctx.fillStyle = `hsl(${normalizedMotion * 360}, 70%, 60%)`;
      ctx.beginPath();
      ctx.arc(0, 0, 30, 0, Math.PI * 2);
      ctx.fill();
      ctx.restore();
    }
  }
}
```

### User Interaction

The interaction system provides comprehensive support for mouse, keyboard, and touch input.

#### Mouse Interaction

```typescript
function render(viji) {
  const mouse = viji.mouse;
  
  // Mouse position (canvas coordinates)
  if (mouse.isInCanvas) {
    const x = mouse.x;                   // Current X coordinate
    const y = mouse.y;                   // Current Y coordinate
    
    // Mouse movement
    const deltaX = mouse.deltaX;         // X movement since last frame
    const deltaY = mouse.deltaY;         // Y movement since last frame
    const velocity = mouse.velocity;     // Smoothed velocity { x, y }
    
    // Mouse buttons
    const isPressed = mouse.isPressed;   // Any button currently pressed
    const leftButton = mouse.leftButton; // Left button state
    const rightButton = mouse.rightButton; // Right button state
    const middleButton = mouse.middleButton; // Middle button state
    
    // Frame-based events
    const wasPressed = mouse.wasPressed; // Button was pressed this frame
    const wasReleased = mouse.wasReleased; // Button was released this frame
    const wasMoved = mouse.wasMoved;     // Mouse moved this frame
    
    // Scroll wheel
    const wheelDelta = mouse.wheelDelta; // Combined wheel delta
    const wheelX = mouse.wheelX;         // Horizontal wheel delta
    const wheelY = mouse.wheelY;         // Vertical wheel delta
    
    // Example: Mouse-reactive animation
    ctx.fillStyle = leftButton ? 'red' : 'blue';
    ctx.beginPath();
    ctx.arc(x, y, 20 + Math.abs(velocity.x + velocity.y), 0, Math.PI * 2);
    ctx.fill();
  }
}
```

#### Keyboard Interaction

```typescript
function render(viji) {
  const keyboard = viji.keyboard;
  
  // Key state queries
  if (keyboard.isPressed('w')) {
    // W key is currently pressed
    console.log('W key is held down');
  }
  
  if (keyboard.wasPressed('space')) {
    // Space was pressed this frame
    console.log('Space was pressed!');
  }
  
  if (keyboard.wasReleased('escape')) {
    // Escape was released this frame
    console.log('Escape was released!');
  }
  
  // Active key tracking
  const activeKeys = keyboard.activeKeys;        // Set of currently pressed keys
  const pressedThisFrame = keyboard.pressedThisFrame; // Set of keys pressed this frame
  const releasedThisFrame = keyboard.releasedThisFrame; // Set of keys released this frame
  
  // Modifier keys
  const shift = keyboard.shift;          // Shift key is held
  const ctrl = keyboard.ctrl;            // Ctrl key is held
  const alt = keyboard.alt;              // Alt key is held
  const meta = keyboard.meta;            // Meta/Cmd key is held
  
  // Recent activity
  const lastKeyPressed = keyboard.lastKeyPressed;   // Last key that was pressed
  const lastKeyReleased = keyboard.lastKeyReleased; // Last key that was released
  
  // Example: Keyboard-controlled movement
  let moveX = 0;
  let moveY = 0;
  
  if (keyboard.isPressed('w') || keyboard.isPressed('W')) moveY -= 5;
  if (keyboard.isPressed('s') || keyboard.isPressed('S')) moveY += 5;
  if (keyboard.isPressed('a') || keyboard.isPressed('A')) moveX -= 5;
  if (keyboard.isPressed('d') || keyboard.isPressed('D')) moveX += 5;
  
  // Apply movement
  ctx.save();
  ctx.translate(moveX, moveY);
  ctx.fillStyle = 'green';
  ctx.fillRect(0, 0, 50, 50);
  ctx.restore();
}
```

#### Touch Interaction

```typescript
function render(viji) {
  const touches = viji.touches;
  
  // Touch points
  for (const touch of touches.points) {
    const x = touch.x;                   // Touch X coordinate
    const y = touch.y;                   // Touch Y coordinate
    const pressure = touch.pressure;     // Pressure (0-1)
    const radius = touch.radius;         // Touch radius
    const id = touch.id;                 // Unique touch ID
    
    // Movement
    const deltaX = touch.deltaX;         // X movement since last frame
    const deltaY = touch.deltaY;         // Y movement since last frame
    const velocity = touch.velocity;     // Movement velocity { x, y }
    
    // Lifecycle
    const isNew = touch.isNew;           // Touch started this frame
    const isActive = touch.isActive;     // Touch is currently active
    const isEnding = touch.isEnding;     // Touch ending this frame
    
    // Draw touch point
    ctx.fillStyle = isNew ? 'red' : isEnding ? 'yellow' : 'blue';
    ctx.beginPath();
    ctx.arc(x, y, radius * 2, 0, Math.PI * 2);
    ctx.fill();
  }
  
  // Touch events
  const started = touches.started;       // Touches that started this frame
  const moved = touches.moved;           // Touches that moved this frame
  const ended = touches.ended;           // Touches that ended this frame
  
  // Primary touch (first touch point)
  const primary = touches.primary;       // Primary touch point or null
  
  // Touch gestures
  const gestures = touches.gestures;
  
  if (gestures.isPinching) {
    const scale = gestures.pinchScale;       // Current pinch scale
    const delta = gestures.pinchDelta;       // Scale change since last frame
    
    // React to pinch gesture
    ctx.save();
    ctx.scale(scale, scale);
    ctx.fillStyle = 'purple';
    ctx.fillRect(0, 0, 100, 100);
    ctx.restore();
  }
  
  if (gestures.isRotating) {
    const angle = gestures.rotationAngle;    // Current rotation angle
    const delta = gestures.rotationDelta;    // Rotation change since last frame
    
    // React to rotation gesture
    ctx.save();
    ctx.rotate(angle);
    ctx.fillStyle = 'orange';
    ctx.fillRect(-25, -25, 50, 50);
    ctx.restore();
  }
  
  if (gestures.isPanning) {
    const panDelta = gestures.panDelta;      // Pan movement { x, y }
    
    // React to pan gesture
    ctx.save();
    ctx.translate(panDelta.x, panDelta.y);
    ctx.fillStyle = 'cyan';
    ctx.fillRect(0, 0, 50, 50);
    ctx.restore();
  }
  
  if (gestures.isTapping) {
    const tapCount = gestures.tapCount;      // Number of taps
    const tapPosition = gestures.tapPosition; // { x, y } tap position
    
    // React to tap gesture
    if (tapPosition) {
      ctx.fillStyle = 'lime';
      ctx.beginPath();
      ctx.arc(tapPosition.x, tapPosition.y, 30, 0, Math.PI * 2);
      ctx.fill();
    }
  }
}
```

## 🏗️ Architecture

### Security Model

The core implements a comprehensive security model to ensure safe execution of artist code:

- **IFrame Isolation**: Complete separation from host environment with sandboxed execution
- **WebWorker Sandboxing**: Artist code runs with controlled API access only
- **Blob URL Creation**: Secure worker and IFrame creation from blob URLs
- **Resource Protection**: Memory leaks and errors cannot affect main application
- **Controlled Communication**: Optimized message passing with validation

### Performance Features

The core provides extensive performance optimization capabilities:

- **Configurable Frame Rates**: Full (60fps) or half (30fps) modes for performance tuning
- **Resolution Scaling**: Fractional (0.1-1.0) or explicit canvas dimensions
- **Adaptive Optimization**: Automatic performance tuning based on hardware capabilities
- **Efficient Communication**: Optimized message passing between components
- **Memory Management**: Automatic resource cleanup and memory leak prevention

### Multi-Instance Support

The core supports multiple concurrent instances for complex applications:

```typescript
// Main scene with full features
const mainCore = new VijiCore({
  hostContainer: document.getElementById('main-scene'),
  resolution: { width: 1920, height: 1080 },
  frameRateMode: 'full',
  allowUserInteraction: true,
  audioStream: sharedAudioStream,
  videoStream: sharedVideoStream
});

// Preview instance with reduced features
const previewCore = new VijiCore({
  hostContainer: document.getElementById('preview'),
  resolution: 0.25, // 25% resolution for performance
  frameRateMode: 'half',
  noInputs: true,
  allowUserInteraction: false,
  audioStream: sharedAudioStream // Shared efficiently across instances
});

// Thumbnail instance for gallery view
const thumbnailCore = new VijiCore({
  hostContainer: document.getElementById('thumbnail'),
  resolution: 0.1, // 10% resolution
  frameRateMode: 'half',
  noInputs: true,
  allowUserInteraction: false
});

// To change scenes, create a new instance and destroy the old one
const newCore = new VijiCore({
  hostContainer: document.getElementById('scene-host'),
  sceneCode: newSceneCode,
  audioStream: sharedAudioStream,
  videoStream: sharedVideoStream
});

// Automatic comprehensive cleanup when destroyed
await oldCore.destroy();
```

## 🔍 Error Handling

The core provides comprehensive error handling with detailed error information:

```typescript
import { VijiCoreError } from '@viji-dev/core';

try {
  const core = new VijiCore(config);
  await core.initialize();
} catch (error) {
  if (error instanceof VijiCoreError) {
    console.error(`Core error [${error.code}]:`, error.message);
    console.error('Error context:', error.context);
    
    // Handle specific error types
    switch (error.code) {
      case 'INVALID_CONFIG':
        console.error('Configuration is invalid:', error.context);
        break;
      case 'INITIALIZATION_ERROR':
        console.error('Failed to initialize core:', error.context);
        break;
      case 'CORE_NOT_READY':
        console.error('Core is not ready for operations');
        break;
      case 'INSTANCE_DESTROYED':
        console.error('Core instance has been destroyed');
        break;
      case 'PARAMETERS_NOT_INITIALIZED':
        console.error('Parameter system not yet initialized');
        break;
      case 'UNKNOWN_PARAMETER':
        console.error('Parameter not found:', error.context);
        break;
    }
  } else {
    console.error('Unexpected error:', error);
  }
}
```

**Common Error Codes:**
- `INVALID_CONFIG` - Configuration validation failed
- `INITIALIZATION_ERROR` - Failed to initialize core components
- `CORE_NOT_READY` - Operation attempted before ready
- `INSTANCE_DESTROYED` - Operation attempted after destroy
- `PARAMETERS_NOT_INITIALIZED` - Parameters not yet available
- `UNKNOWN_PARAMETER` - Parameter not found
- `CONCURRENT_INITIALIZATION` - Multiple initialization attempts
- `MANAGER_NOT_READY` - Internal component not available

## 🧪 Development

```bash
# Install dependencies
npm install

# Build the package
npm run build

# Run tests
npm test

# Development build (watch mode)
npm run dev

# Type checking
npm run type-check

# Linting
npm run lint
```

## 📚 Examples

The package includes comprehensive examples in the `/example` directory:

- **Basic Scene Creation**: Simple animated shapes with parameters
- **Audio-Reactive Scenes**: Scenes that respond to audio input
- **Video-Reactive Scenes**: Scenes that respond to video analysis
- **Interactive Scenes**: Mouse, keyboard, and touch interaction
- **Parameter System**: Complete parameter definition and UI generation
- **Multi-Instance**: Multiple concurrent scene instances

## 🎯 Use Cases

### Platform Integration
The core integrates seamlessly with the Viji platform, providing scene execution while the platform handles UI, user management, and social features.

### SDK Development
The core serves as the execution foundation for the Viji SDK, ensuring identical behavior between development and platform environments.

### Standalone Applications
Use the core directly in custom applications for creative scene rendering with full feature support.

## 📄 License

MIT License - see LICENSE file for details.

## 🤝 Contributing

The Viji Core package is part of the larger Viji Creative SDK ecosystem. For contribution guidelines and development setup, please refer to the main Viji project documentation.

---

**Viji Core** - Universal execution engine for creative scenes across all contexts.