promptforge/README.md

Adaptive Prompt Intelligence & Orchestration SDK - Manage, optimize, and serve prompts for LLMs with versioning, feedback loops, and multi-provider support

# PromptForge

**Adaptive Prompt Intelligence & Orchestration SDK**

[![npm version](https://img.shields.io/npm/v/promptforge.svg)](https://www.npmjs.com/package/promptforge)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![TypeScript](https://img.shields.io/badge/TypeScript-5.4-blue.svg)](https://www.typescriptlang.org/)

PromptForge is a production-ready TypeScript SDK and CLI toolkit that helps developers design, optimize, version, evaluate, and serve prompts for Large Language Models (LLMs) intelligently. It combines prompt engineering best practices with reinforcement learning feedback loops and embedding-based optimization.

## 🎯 Vision

PromptForge acts as the **core prompt brain** for any AI application — managing the full lifecycle of a prompt from creation → versioning → optimization → evaluation → deployment. It becomes an intelligent layer between your app and the LLMs, learning over time which prompts work best for each task and dynamically improving them based on usage and outcomes.

**Think of it as:** LangChain + PromptLayer + ML feedback system, unified into one SDK.

## ✨ Core Features

### 🔄 Prompt Versioning & Registry
- Maintain a centralized registry of all prompts with complete metadata
- Semantic diff for prompt versions (compare embeddings)
- Version pinning for reproducibility
- Full audit trail with ownership and tags

### 📝 Prompt Templates with Variables
- Support variable interpolation `{{user_input}}`, `{{context}}`, etc.
- Dynamic runtime context injection
- JSON/YAML-based template storage
- System prompts and few-shot examples

### 📊 Prompt Performance Tracking
- Each execution logged with metadata: model, latency, tokens, cost, feedback
- Store all prompt results for analytics
- Aggregate metrics and percentile calculations
- Per-provider performance benchmarking

### 🔁 Feedback & Reinforcement System
- Collect user or system feedback (thumbs up/down, quality scores, reward signals)
- Use reinforcement learning principles to update prompt scoring
- Auto-promote top-performing prompts and deprecate weak ones
- Composite scoring: `Score = α * UserFeedback + β * PerformanceMetric + γ * CostEfficiency`

### 🧠 Embedding-Based Optimization
- Generate embeddings for all prompts and responses
- Semantic similarity and clustering
- Suggest prompt rewrites using LLM self-evaluation
- Identify redundant or overlapping prompts

### 🔌 Cross-Model Compatibility
- **Supported Providers:** OpenAI, Anthropic, Google (Gemini), Mistral, Ollama
- **Automatic Fallback:** If one provider fails, automatically fallback to backup
- **Cost Optimization:** Choose models based on quality/cost tradeoffs
- **Local Models:** Full support for self-hosted models via Ollama

### 🎯 Prompt Evaluation Harness
- Benchmark prompts on predefined datasets or user-defined examples
- Metrics: accuracy, consistency, coherence, relevance, semantic similarity
- Auto-report leaderboard of best-performing prompt+model pairs
- A/B testing between prompt versions

### ⚡ Semantic Prompt Caching
- Cache previous results using embeddings (semantic caching)
- Reuse LLM responses if query is similar enough (saves cost & latency)
- Configurable similarity threshold and TTL

### 🔐 Security & Compliance
- Secure key management
- PII filtering and redaction
- Complete audit logs for traceability
- Role-based access control ready

## 🚀 Quick Start

### Installation

```bash
# Install globally
npm install -g promptforge

# Or use npx
npx promptforge init my-project
```

### Initialize Project

```bash
forge init my-project
cd my-project

# Configure API keys in .env
code .env
```

### Create Your First Prompt

```yaml
# prompts/greeting.yaml
name: greeting
description: A friendly greeting prompt
variables:
  - name
  - language
content: |
  Hello {{name}}! Welcome to PromptForge.
  Please greet me in {{language}}.
systemPrompt: You are a friendly assistant that provides warm greetings.
```

### Push Prompt to Registry

```bash
forge push greeting --template ./prompts/greeting.yaml
```

### Execute Prompt

```bash
forge execute greeting --input '{"name": "Alice", "language": "Spanish"}'
```

### List All Prompts

```bash
forge list
forge stats
```

## 📚 SDK Usage

### Basic Example

```typescript
import { PromptForge, LLMProvider } from 'promptforge';

// Initialize
const forge = new PromptForge({
  projectName: 'my-app',
  database: {
    type: 'postgresql',
    url: process.env.DATABASE_URL,
  },
  defaultProvider: LLMProvider.OPENAI,
  fallbackProviders: [LLMProvider.ANTHROPIC, LLMProvider.GOOGLE],
});

// Create a prompt
const prompt = await forge.createPrompt({
  name: 'summarize',
  content: 'Summarize the following text: {{text}}',
  owner: 'team-ai',
  tags: ['summarization', 'production'],
  template: {
    variables: ['text'],
  },
});

// Execute prompt
const result = await forge.executePrompt({
  promptName: 'summarize',
  input: {
    text: 'Long article content...',
  },
  llmConfig: {
    provider: LLMProvider.OPENAI,
    model: 'gpt-4o-mini',
    temperature: 0.3,
  },
});

console.log(result.output);
console.log('Cost:', result.metrics.cost);
console.log('Latency:', result.metrics.latencyMs);

// Track feedback
await forge.trackFeedback({
  executionId: result.id,
  promptId: prompt.id,
  type: FeedbackType.THUMBS_UP,
  score: 0.9,
  comment: 'Great summary!',
});
```

### Evaluation Example

```typescript
// Evaluate prompt against a dataset
const evaluation = await forge.evaluatePrompt({
  promptId: prompt.id,
  examples: [
    {
      input: { text: 'Sample article 1...' },
      expectedOutput: 'Expected summary 1',
    },
    {
      input: { text: 'Sample article 2...' },
      expectedOutput: 'Expected summary 2',
    },
  ],
});

console.log('Overall Score:', evaluation.overallScore);
console.log('Accuracy:', evaluation.metrics.accuracy);
console.log('Consistency:', evaluation.metrics.consistency);
```

### Template Engine Example

```typescript
import { TemplateEngine } from 'promptforge';

const engine = new TemplateEngine();

const template = {
  id: '...',
  promptId: '...',
  content: 'Translate "{{text}}" to {{language}}',
  variables: ['text', 'language'],
  format: 'text',
};

const rendered = engine.render(template, {
  text: 'Hello world',
  language: 'French',
});

console.log(rendered); // Translate "Hello world" to French
```

## 🏗️ Architecture

```
promptforge/
├── src/
│   ├── core/
│   │   ├── forge.ts           # Main SDK class
│   │   ├── registry.ts        # Prompt versioning & storage
│   │   ├── template-engine.ts # Template parsing & rendering
│   │   ├── metrics.ts         # Performance tracking
│   │   ├── llm-adapters.ts    # Multi-provider integration
│   │   ├── cache.ts           # Semantic caching
│   │   ├── feedback.ts        # Feedback & scoring
│   │   └── evaluation.ts      # Evaluation engine
│   ├── cli/
│   │   └── index.ts           # CLI commands
│   ├── utils/
│   │   ├── logger.ts          # Logging utility
│   │   └── config-loader.ts   # Configuration management
│   └── types.ts               # TypeScript types & schemas
├── tests/
├── examples/
├── docs/
└── migrations/
```

## 📖 CLI Commands

| Command | Description |
|---------|-------------|
| `forge init [name]` | Initialize a new PromptForge project |
| `forge push <name>` | Create or update a prompt |
| `forge list` | List all prompts in registry |
| `forge execute <name>` | Execute a prompt with inputs |
| `forge eval <name>` | Evaluate prompt performance |
| `forge optimize` | Optimize prompts based on feedback |
| `forge stats` | Show registry statistics |

## 🎯 Use Cases

- **Customer Support Bots**: Version and optimize support prompts based on satisfaction scores
- **Content Generation**: A/B test different prompt variations for blog posts or marketing copy
- **RAG Systems**: Manage and version retrieval-augmented generation prompts
- **Multi-Agent Systems**: Coordinate prompts across multiple AI agents
- **Enterprise LLM Ops**: Centralized prompt management for teams with audit trails

## 🔧 Configuration

### Environment Variables

```bash
# LLM Provider API Keys
OPENAI_API_KEY=sk-...
ANTHROPIC_API_KEY=sk-ant-...
GOOGLE_API_KEY=...
MISTRAL_API_KEY=...

# Database
DATABASE_URL=postgresql://localhost:5432/promptforge
REDIS_URL=redis://localhost:6379

# Vector Database (Pinecone)
PINECONE_API_KEY=...
PINECONE_ENVIRONMENT=us-west1-gcp
PINECONE_INDEX_NAME=promptforge-embeddings

# Features
ENABLE_SEMANTIC_CACHE=true
ENABLE_AUTO_OPTIMIZATION=true
```

### Configuration File (`promptforge.json`)

```json
{
  "projectName": "my-app",
  "version": "1.0.0",
  "defaultProvider": "openai",
  "fallbackProviders": ["anthropic", "google"],
  "optimization": {
    "enabled": true,
    "autoPromote": false,
    "scoreThreshold": 0.8
  },
  "telemetry": {
    "enabled": true
  }
}
```

## 🧪 Testing

```bash
npm test
npm run test:coverage
```

## 📦 Building

```bash
npm run build
npm run watch
```

## 🤝 Contributing

Contributions are welcome! Please see [CONTRIBUTING.md](./CONTRIBUTING.md) for guidelines.

## 📄 License

MIT © [Yash Gupta](https://github.com/gyash1512)

## 🔗 Links

- **GitHub**: https://github.com/gyash1512/PromptForge
- **npm**: https://www.npmjs.com/package/promptforge
- **Documentation**: [Full API Docs](./docs/API.md)
- **Examples**: [Example Projects](./examples)

## 🙏 Acknowledgments

Built with ❤️ by [Yash Gupta](https://github.com/gyash1512)

Inspired by LangChain, PromptLayer, and the amazing AI community.

---

**Ready to forge better prompts?** ⚒️

```bash
npx promptforge init my-project
```