@varia-bly/variably-sdk/README.md

Official JavaScript/TypeScript SDK for Variably feature flags, experimentation, LLM experiments with React hooks, and real-time dynamic configurations

# Variably JavaScript/TypeScript SDK

Official JavaScript/TypeScript SDK for Variably feature flags and experimentation platform.

## Installation

```bash
npm install @varia-bly/variably-sdk
# or
yarn add @varia-bly/variably-sdk
```

## Quick Start

```typescript
import { VariablyClient } from '@varia-bly/variably-sdk';

// Initialize the client
const client = new VariablyClient({
  apiKey: 'your-api-key',
  baseUrl: 'https://graphql.variably.tech', // optional, defaults to production GraphQL endpoint
  environment: 'production' // optional
});

// Evaluate a boolean feature flag
const userContext = {
  userId: 'user-123',
  email: 'user@example.com',
  country: 'US'
};

const isFeatureEnabled = await client.evaluateFlagBool(
  'new-checkout-flow',
  false, // default value
  userContext
);

if (isFeatureEnabled) {
  // Show new checkout flow
}

// Evaluate a feature gate
const hasAccess = await client.evaluateGate('premium-features', userContext);

// Track events
await client.track({
  name: 'button_clicked',
  userId: 'user-123',
  properties: {
    button_name: 'checkout',
    page: 'product-detail'
  }
});
```

## Configuration

```typescript
interface VariablyConfig {
  /** API key for authentication */
  apiKey: string;
  /** Base URL for the GraphQL API (default: https://graphql.variably.tech) */
  baseUrl?: string;
  /** Environment (development, staging, production) */
  environment?: string;
  /** Request timeout in milliseconds (default: 5000) */
  timeout?: number;
  /** Number of retry attempts (default: 3) */
  retryAttempts?: number;
  /** Enable analytics tracking (default: true) */
  enableAnalytics?: boolean;
  /** Cache configuration */
  cache?: {
    /** Cache TTL in milliseconds (default: 300000 = 5 minutes) */
    ttl?: number;
    /** Maximum cache size (default: 1000) */
    maxSize?: number;
    /** Enable cache (default: true) */
    enabled?: boolean;
  };
  /** Real-time Feature Gate updates via WebSocket */
  realTimeUpdates?: {
    /** Enable real-time updates (default: false) */
    enabled?: boolean;
    /** Project ID to subscribe to (required if enabled) */
    projectId?: string;
    /** Auto-invalidate cache on updates (default: true) */
    autoInvalidateCache?: boolean;
  };
}
```

## Real-Time Feature Gate Updates

**NEW**: The SDK now supports real-time Feature Gate updates via WebSocket subscriptions. When enabled, your application receives instant notifications when Feature Gates are updated in the Variably UI, and the SDK automatically invalidates its cache to ensure fresh evaluations.

### How It Works

```
UI Update → Backend → Valkey PubSub → GraphQL Subscription → WebSocket → SDK → Cache Invalidation
```

When you update a Feature Gate in the Variably UI:
1. Backend publishes the update to Valkey
2. GraphQL server receives the update and broadcasts to subscribed clients
3. SDK receives the WebSocket message
4. Cache is automatically invalidated for the updated gate
5. Next `evaluateGate()` call fetches fresh data

### Usage

```typescript
import { VariablyClient } from '@varia-bly/variably-sdk';

const client = new VariablyClient({
  apiKey: 'your-api-key',
  baseUrl: 'http://localhost:4000', // or your GraphQL endpoint

  // Enable real-time updates
  realTimeUpdates: {
    enabled: true,
    projectId: 'your-project-id', // Required: The project to subscribe to
    autoInvalidateCache: true      // Optional: Auto-clear cache on updates (default: true)
  }
});

// Now when you evaluate a gate, it will automatically receive real-time updates
const hasAccess = await client.evaluateGate('premium-features', userContext);

// When the gate is updated in the UI, the cache is automatically invalidated
// and the next evaluation will fetch fresh data
```

### Benefits

- **Instant Updates**: Changes in the Variably UI are reflected immediately in your application
- **Zero Configuration**: Just enable and provide your project ID
- **Automatic Cache Invalidation**: No manual cache management required
- **Graceful Degradation**: If WebSocket connection fails, SDK falls back to normal HTTP with caching
- **Auto-Reconnection**: Built-in reconnection logic with exponential backoff
- **Production Ready**: Comprehensive error handling and logging

### Advanced Usage

```typescript
import { VariablyClient } from '@varia-bly/variably-sdk';

const client = new VariablyClient({
  apiKey: 'your-api-key',
  baseUrl: 'http://localhost:4000',
  realTimeUpdates: {
    enabled: true,
    projectId: 'your-project-id',
    autoInvalidateCache: true
  }
});

// Check if real-time sync is active
if (client.realTimeSync?.isActive()) {
  console.log('Real-time updates are active');
  console.log('Connection state:', client.realTimeSync.getConnectionState());
}

// Stop real-time updates (e.g., during cleanup)
client.realTimeSync?.stop();
```

### Requirements

- GraphQL endpoint must support WebSocket subscriptions
- Project ID must be valid and have Feature Gates configured
- Network must allow WebSocket connections (ws:// or wss://)

## LLM Experiments

**NEW in v2.2.0**: The SDK now includes comprehensive support for LLM experiments with A/B testing, prompt optimization, and React hooks.

```typescript
import { VariablyLLMClient, useLLMPrompt } from '@varia-bly/variably-sdk';

// Basic LLM experiment
const llmClient = new VariablyLLMClient({
  apiKey: 'your-api-key'
});

const response = await llmClient.executePrompt({
  experimentId: 'exp_abc123',
  userContext: { userId: 'user_123' },
  variables: { topic: 'AI' }
});

console.log('Response:', response.content);
console.log('Variant:', response.variant.name);

// React Hook for LLM experiments
function MyComponent() {
  const { execute, response, loading } = useLLMPrompt({
    experimentId: 'exp_abc123',
    apiKey: 'your-api-key'
  });

  return (
    <div>
      <button onClick={() => execute({
        userContext: { userId: 'user_123' }
      })}>
        Generate
      </button>
      {loading && <p>Loading...</p>}
      {response && <p>{response.content}</p>}
    </div>
  );
}
```

### Context-Aware Evaluation (Better RAG Quality) — v2.5.0+

For RAG chatbots, passing conversation history and retrieved chunks enables **groundedness scoring, hallucination detection, and conversational coherence** — dimensions impossible to evaluate in isolation.

The `evaluationContext` is **not sent to the LLM** — only used by Variably's evaluator.

#### REST Client (Managed Execution)

```typescript
import { VariablyRESTClient, EvaluationContext } from '@varia-bly/variably-sdk';

const client = new VariablyRESTClient({ apiKey: 'your-api-key' });

const evaluationContext: EvaluationContext = {
  // Retrieved RAG chunks — enables groundedness & hallucination scoring
  referenceMaterials: [
    {
      id: 'chunk-001',
      content: 'Unhealthy diets high in refined sugars, fats...',
      source: 'Kenya National Clinical Guidelines',
      type: 'chunk',
      relevanceScore: 0.89,
    },
    {
      id: 'chunk-002',
      content: 'Modifiable risk factors include obesity...',
      source: 'Kenya National Clinical Guidelines',
      type: 'chunk',
      relevanceScore: 0.82,
    },
  ],
  // Conversation history — enables coherence & consistency scoring
  workflowHistory: [
    { role: 'user', content: 'What causes diabetes?' },
    { role: 'assistant', content: 'Key factors include genetics, diet...' },
    { role: 'user', content: 'What about potatoes?' },
  ],
  // The query sent to the retriever (optional)
  retrievalQuery: 'potato consumption glycemic index diabetes risk',
};

// Non-streaming
const response = await client.evaluatePrompt(
  'rag-prompt-experiment',
  { topic: 'potatoes', context: contextText },
  { userId: 'user_123' },
  { evaluationContext }
);

// Streaming (SSE)
const controller = client.evaluatePromptStream(
  'rag-prompt-experiment',
  { topic: 'potatoes', context: contextText },
  { userId: 'user_123' },
  { onToken: (t) => process.stdout.write(t.content) },
  { evaluationContext }
);
```

#### LLM Client

```typescript
const response = await llmClient.executePrompt({
  experimentId: 'rag-prompt-experiment',
  userContext: { userId: 'user_123' },
  variables: { query: 'What about potatoes?', context: contextText },
  evaluationContext: {
    referenceMaterials: [
      { id: 'chunk-001', content: 'Unhealthy diets...', source: 'Guidelines', relevanceScore: 0.89 },
    ],
    workflowHistory: [
      { role: 'user', content: 'What causes diabetes?' },
      { role: 'assistant', content: 'Key factors include genetics, diet...' },
    ],
    retrievalQuery: 'potato consumption glycemic index diabetes risk',
  },
});
```

#### Async Scoring (BYOR)

```typescript
await client.submitScoringRequest({
  experimentId: 'exp-123',
  variantId: 'variant-a',
  projectId: 'proj-456',
  prompt: executedPrompt,
  response: llmResponse,
  evaluationContext: {
    referenceMaterials: retrievedChunks,
    workflowHistory: conversationHistory,
  },
});
```

**What this enables:**

| Dimension | Description | Requires |
|-----------|-------------|----------|
| `faithfulness` | % of claims grounded in retrieved chunks | `referenceMaterials` |
| `hallucinationRate` | % of claims with no source in context | `referenceMaterials` |
| `contextUtilization` | % of relevant chunks actually used | `referenceMaterials` |
| `conversationConsistency` | No contradictions with prior turns | `workflowHistory` |
| `contextRetention` | Maintains topic awareness across turns | `workflowHistory` |
| `transparency` | Discloses when going beyond source material | `referenceMaterials` |

#### Type Reference

```typescript
interface EvaluationContext {
  referenceMaterials?: ReferenceMaterial[];  // RAG chunks / source documents
  workflowHistory?: WorkflowStep[];         // Conversation turns
  retrievalQuery?: string;                  // Query sent to retriever
}

interface ReferenceMaterial {
  id: string;           // Chunk ID
  content: string;      // Chunk text
  source?: string;      // Document source URL/name
  type?: string;        // e.g. "chunk", "document"
  relevanceScore?: number; // Retriever similarity score
}

interface WorkflowStep {
  role?: string;    // "user" | "assistant"
  step?: string;    // Step name in workflow
  input?: string;   // Step input
  output?: string;  // Step output
  content?: string; // Message content
}
```

See [Context-Aware RAG Evaluation](../../docs/concepts/context-aware-rag-evaluation.md) for the full concept doc with architecture diagrams and integration examples.

**📖 Full Documentation**: See [LLM_EXPERIMENTS.md](./LLM_EXPERIMENTS.md) for comprehensive documentation including:
- React Hooks (`useLLMPrompt`, `useLLMStreaming`, `useConversation`, `useLLMEvaluation`)
- Streaming responses
- Response evaluation and quality scoring
- Business metrics tracking
- Batch execution
- Advanced features and best practices

## Advanced Usage

### Environment Variables

You can create a client using environment variables:

```typescript
import { createClientFromEnv } from '@varia-bly/variably-sdk';

// Uses these environment variables:
// VARIABLY_API_KEY
// VARIABLY_BASE_URL
// VARIABLY_ENVIRONMENT
// VARIABLY_TIMEOUT
// VARIABLY_ENABLE_ANALYTICS

const client = createClientFromEnv();
```

### Different Flag Types

```typescript
// Boolean flags
const boolValue = await client.evaluateFlagBool('feature-enabled', false, userContext);

// String flags
const stringValue = await client.evaluateFlagString('theme', 'light', userContext);

// Number flags
const numberValue = await client.evaluateFlagNumber('max-items', 10, userContext);

// JSON flags
const jsonValue = await client.evaluateFlagJSON('config', { timeout: 5000 }, userContext);

// Get full evaluation details
const result = await client.evaluateFlag('feature-flag', 'default', userContext);
console.log(result); // { key, value, reason, cacheHit, evaluatedAt, error? }
```

### Batch Evaluation

```typescript
const flags = await client.evaluateFlags([
  'feature-a',
  'feature-b', 
  'feature-c'
], userContext);

console.log(flags['feature-a'].value);
```

### Event Tracking

```typescript
// Single event
await client.track({
  name: 'purchase_completed',
  userId: 'user-123',
  properties: {
    amount: 99.99,
    currency: 'USD',
    items: ['item-1', 'item-2']
  }
});

// Batch events
await client.trackBatch([
  { name: 'page_view', userId: 'user-123', properties: { page: '/home' } },
  { name: 'button_click', userId: 'user-123', properties: { button: 'cta' } }
]);
```

### Cache Management

```typescript
// Clear cache
client.clearCache();

// Get cache stats
const stats = client.cache.getStats();
console.log(stats); // { size, maxSize, hitRate, enabled }
```

### Metrics

```typescript
// Get SDK metrics
const metrics = client.getMetrics();
console.log(metrics);
// {
//   apiCalls: number,
//   cacheHits: number,
//   cacheMisses: number,
//   errors: number,
//   averageLatency: number,
//   cacheHitRate: number,
//   errorRate: number,
//   flagsEvaluated: number,
//   gatesEvaluated: number,
//   eventsTracked: number,
//   startTime: Date
// }
```

### Custom Logger

```typescript
import { VariablyClient, createLogger } from '@varia-bly/variably-sdk';

const logger = createLogger({
  level: 'debug',
  type: 'structured' // 'console', 'silent', 'structured'
});

// Or use a custom logging function
const customLogger = createLogger({
  level: 'info',
  custom: (level, message, meta) => {
    // Send to your logging service
    console.log(`${level}: ${message}`, meta);
  }
});
```

## Browser Usage

The SDK works in both Node.js and browser environments:

```html
<script type="module">
import { VariablyClient } from 'https://unpkg.com/@varia-bly/variably-sdk@latest/dist/index.esm.js';

const client = new VariablyClient({
  apiKey: 'your-api-key',
  baseUrl: 'https://api.variably.com'
});

// Use the client...
</script>
```

## Error Handling

```typescript
import { 
  VariablyError, 
  NetworkError, 
  AuthenticationError, 
  ValidationError 
} from '@varia-bly/variably-sdk';

try {
  const result = await client.evaluateFlag('my-flag', false, userContext);
} catch (error) {
  if (error instanceof AuthenticationError) {
    console.error('Invalid API key');
  } else if (error instanceof NetworkError) {
    console.error('Network error:', error.statusCode);
  } else if (error instanceof ValidationError) {
    console.error('Validation error:', error.field);
  } else {
    console.error('Unknown error:', error.message);
  }
}
```

## TypeScript Support

The SDK is written in TypeScript and includes full type definitions:

```typescript
import { VariablyClient, UserContext, FlagResult } from '@varia-bly/variably-sdk';

const userContext: UserContext = {
  userId: 'user-123',
  email: 'user@example.com',
  attributes: {
    plan: 'premium',
    signupDate: '2023-01-01'
  }
};

const result: FlagResult = await client.evaluateFlag('feature', false, userContext);
```

## Development

### Building

```bash
npm run build
```

### Testing

```bash
npm test
```

### Linting

```bash
npm run lint
```

## License

MIT License - see LICENSE file for details.