oneie
Version:
Build apps, websites, and AI agents in English. Zero-interaction setup for AI agents (Claude Code, Cursor, Windsurf). Download to your computer, run in the cloud, deploy to the edge. Open source and free forever.
1,035 lines (863 loc) • 26 kB
Markdown
title: Convex Agents
dimension: things
category: plans
tags: agent, ai, ai-agent, architecture
related_dimensions: events, people
scope: global
created: 2025-11-03
updated: 2025-11-03
version: 1.0.0
ai_context: |
This document is part of the things dimension in the plans category.
Location: one/things/plans/convex-agents.md
Purpose: Documents convex agents implementation plan
Related dimensions: events, people
For AI agents: Read this to understand convex agents.
# Convex Agents Implementation Plan
**Version:** 1.0.0
**Status:** Planning
**Integration:** AI SDK + AI Elements
**Target Platform:** Convex Cloud + Astro + React 19
## Architecture: Effect.ts Wrapping Agent Component
Convex Agents component provides infrastructure (thread management, streaming, retries). We wrap it with **Effect.ts services** for type-safe business logic composition:
```typescript
// Tagged error types for agent operations
class AgentExecutionError extends Data.TaggedError("AgentExecutionError")<{
agentName: string;
threadId: string;
cause: unknown;
}> {}
class ThreadNotFoundError extends Data.TaggedError("ThreadNotFoundError")<{
threadId: string;
}> {}
// Define AgentService with Context.Tag
class AgentService extends Context.Tag("AgentService")<
AgentService,
{
readonly createThread: (
ctx: ActionCtx,
agentId: string,
initialMessage: string,
) => Effect.Effect<{ threadId: string }, AgentExecutionError>;
readonly continueThread: (
ctx: ActionCtx,
threadId: string,
message: string,
) => Effect.Effect<
{ response: string; threadId: string },
AgentExecutionError | ThreadNotFoundError
>;
readonly getThreadMessages: (
threadId: string,
) => Effect.Effect<Message[], ThreadNotFoundError>;
}
>() {}
// Implement using Agent component
const AgentServiceLive = Layer.effect(
AgentService,
Effect.gen(function* () {
const agent = new Agent(components.agent, {
name: "Main Agent",
chat: openai("gpt-4-turbo"),
instructions: "You are a helpful AI assistant.",
tools: {
/* tools defined separately */
},
});
return {
createThread: (ctx, agentId, initialMessage) =>
Effect.gen(function* () {
const result = yield* Effect.tryPromise({
try: () =>
agent.createThread(ctx, {
messages: [{ role: "user", content: initialMessage }],
}),
catch: (error) =>
new AgentExecutionError({
agentName: "Main Agent",
threadId: "unknown",
cause: error,
}),
});
return { threadId: result.threadId };
}),
continueThread: (ctx, threadId, message) =>
Effect.gen(function* () {
const { thread } = yield* Effect.tryPromise({
try: () => agent.continueThread(ctx, { threadId }),
catch: (error) => new ThreadNotFoundError({ threadId }),
});
const response = yield* Effect.tryPromise({
try: () => thread.generateText({ prompt: message }),
catch: (error) =>
new AgentExecutionError({
agentName: "Main Agent",
threadId,
cause: error,
}),
});
return { response: response.text, threadId };
}),
getThreadMessages: (threadId) =>
Effect.tryPromise({
try: () => agent.getMessages(threadId),
catch: (error) => new ThreadNotFoundError({ threadId }),
}),
};
}),
);
```
## Overview
**Convex Agents** is a Convex component that manages AI agent workflows. It provides:
- Thread-based conversation management
- Automatic message persistence
- Tool call orchestration
- Real-time streaming via WebSockets
- Built-in RAG support for knowledge retrieval
- Multi-tenant isolation (per `groupId`)
- Usage tracking for billing
### Why Convex Agents?
- **Persistence**: All conversations automatically saved in Convex database
- **Scalability**: Handles thousands of concurrent agent threads
- **Real-time**: WebSocket streaming for instant client updates
- **Type-safe**: Full TypeScript support with auto-generated types
- **Integrated**: Works seamlessly with Convex functions, auth, and storage
## Architecture: Convex Agents in the Stack
```
AI Elements UI (Chat Interface)
↓
React Components + useChat Hook
↓
Convex Mutations (Agent Functions)
↓
Convex Agents Runtime
├─ Thread Management
├─ Message Persistence
├─ Tool Execution
├─ Streaming Pipeline
└─ RAG Integration
↓
AI SDK (generateText, streamText)
↓
LLM Providers (OpenAI, Anthropic, etc.)
```
## Installation & Setup (Cycle 1-10)
### Install Package
```bash
npm install @convex-dev/agent
```
### Add to Schema
```typescript
// backend/convex/schema.ts
import { defineSchema, defineTable } from "convex/server";
import { v } from "convex/values";
import { agentTables } from "@convex-dev/agent";
export default defineSchema({
// Existing tables...
groups: defineTable({...}),
things: defineTable({...}),
connections: defineTable({...}),
events: defineTable({...}),
knowledge: defineTable({...}),
// AI-specific tables
aiCalls: defineTable({
groupId: v.id("groups"),
model: v.string(),
provider: v.string(),
prompt: v.string(),
result: v.string(),
tokensUsed: v.optional(v.object({ input: v.number(), output: v.number() })),
duration: v.number(),
timestamp: v.number(),
}).index("by_groupId", ["groupId"]),
// Agent tables (provided by Convex Agents component)
...agentTables(),
});
```
### Initialize Agent Runtime
```typescript
// backend/convex/agent.ts
import { defineAgent } from "@convex-dev/agent";
import { internal } from "./_generated/server";
import { openai } from "@ai-sdk/openai";
export const agentRuntime = defineAgent(async (ctx, params) => {
const { groupId, agentId, message, threadId } = params;
// Get agent configuration from things table
const agent = await ctx.db.get(agentId);
if (!agent || agent.type !== "agent") {
throw new Error("Agent not found");
}
// Set up model based on agent configuration
const model = openai(agent.properties.model || "gpt-4-turbo");
return {
model,
systemPrompt: agent.properties.systemPrompt,
tools: await loadAgentTools(ctx, groupId, agentId),
};
});
async function loadAgentTools(ctx: any, groupId: string, agentId: string) {
// Load tool definitions from agent configuration
// Tools are defined in agent.properties.tools array
return {};
}
```
## Thread Management (Cycle 11-20)
### Create Agent Thread
```typescript
// backend/convex/mutations/agentThreads.ts
import { mutation } from "./_generated/server";
import { v } from "convex/values";
export const createThread = mutation({
args: {
groupId: v.id("groups"),
agentId: v.id("things"),
title: v.optional(v.string()),
},
handler: async (ctx, args) => {
// Create a new thread in aiThreads table
const threadId = await ctx.db.insert("aiThreads", {
groupId: args.groupId,
agentId: args.agentId,
title: args.title || "New Conversation",
status: "active",
metadata: {
createdBy: ctx.auth?.getUserIdentity()?.tokenIdentifier,
},
createdAt: Date.now(),
updatedAt: Date.now(),
});
// Log event
await ctx.db.insert("events", {
groupId: args.groupId,
type: "agent_thread_created",
actorId: ctx.auth?.getUserIdentity()?.tokenIdentifier,
targetId: threadId,
timestamp: Date.now(),
metadata: { agentId: args.agentId },
});
return threadId;
},
});
export const listThreads = query({
args: { groupId: v.id("groups"), agentId: v.id("things") },
handler: async (ctx, args) => {
return await ctx.db
.query("aiThreads")
.withIndex("by_groupId", (q) => q.eq("groupId", args.groupId))
.filter((q) => q.eq(q.field("agentId"), args.agentId))
.order("desc")
.collect();
},
});
export const getThread = query({
args: { threadId: v.id("aiThreads") },
handler: async (ctx, args) => {
return await ctx.db.get(args.threadId);
},
});
export const archiveThread = mutation({
args: { threadId: v.id("aiThreads") },
handler: async (ctx, args) => {
const thread = await ctx.db.get(args.threadId);
if (!thread) throw new Error("Thread not found");
await ctx.db.patch(args.threadId, {
status: "archived",
updatedAt: Date.now(),
});
return args.threadId;
},
});
```
### Message Management (via Convex Agents)
Convex Agents automatically manages messages within threads. Messages are:
- Stored in a Convex Agents internal table
- Keyed by `threadId`
- Accessible via `getMessages()` helper
- Automatically included in agent context
```typescript
// backend/convex/queries/agentMessages.ts
import { query } from "./_generated/server";
import { v } from "convex/values";
import { getMessages } from "@convex-dev/agent";
export const getThreadMessages = query({
args: { threadId: v.string() },
handler: async (ctx, args) => {
// Retrieve all messages for a thread
const messages = await getMessages(ctx, {
threadId: args.threadId,
});
return messages;
},
});
export const getThreadContext = query({
args: { threadId: v.string(), limit: v.optional(v.number()) },
handler: async (ctx, args) => {
const messages = await getMessages(ctx, {
threadId: args.threadId,
limit: args.limit || 10,
});
// Return formatted for LLM context
return messages.map((msg) => ({
role: msg.role,
content: msg.content,
timestamp: msg.createdAt,
}));
},
});
```
## Tool Integration (Cycle 21-30)
### Define Agent Tools
```typescript
// backend/convex/agent/tools.ts
import { tool } from "ai";
import { z } from "zod";
export const agentTools = {
searchKnowledge: tool({
description: "Search the group's knowledge base for relevant information",
parameters: z.object({
query: v.string().describe("Search query"),
limit: z.number().default(5),
}),
execute: async (input) => {
// Implement RAG search against knowledge table
return "Search results...";
},
}),
createTask: tool({
description: "Create a new task thing",
parameters: z.object({
title: z.string(),
description: z.string().optional(),
priority: z.enum(["low", "medium", "high"]),
}),
execute: async (input) => {
// Create thing with type: "task"
return { taskId: "task_123", created: true };
},
}),
getContext: tool({
description: "Get information about the current context",
parameters: z.object({
contextType: z.enum(["group", "user", "agent"]),
}),
execute: async (input) => {
// Return context information
return { context: "context_data" };
},
}),
callHuman: tool({
description: "Escalate to human support",
parameters: z.object({
reason: z.string().describe("Reason for escalation"),
}),
execute: async (input) => {
// Create connection to human agent
// Send notification
return { escalationId: "esc_123", status: "waiting_for_human" };
},
}),
};
```
### Register Tools with Agent
```typescript
// backend/convex/agent.ts (updated)
import { agentTools } from "./agent/tools";
export const agentRuntime = defineAgent(async (ctx, params) => {
const { groupId, agentId, message, threadId } = params;
const agent = await ctx.db.get(agentId);
if (!agent || agent.type !== "agent") {
throw new Error("Agent not found");
}
const model = openai(agent.properties.model || "gpt-4-turbo");
// Filter tools based on agent.properties.tools array
const allowedTools = agent.properties.tools || [];
const tools = Object.fromEntries(
Object.entries(agentTools).filter(([name]) => allowedTools.includes(name)),
);
return {
model,
systemPrompt: agent.properties.systemPrompt,
tools,
groupId,
threadId,
};
});
```
### Execute Tool Calls
Convex Agents automatically executes tool calls and manages the agentic loop. When the agent calls a tool:
1. Tool execution is triggered
2. Result is stored in thread context
3. Agent continues with result included
4. Client sees streamed updates
## Streaming Setup (Cycle 31-40)
### Streaming Mutation
```typescript
// backend/convex/mutations/agentChat.ts
import { mutation } from "./_generated/server";
import { v } from "convex/values";
import { streamText } from "ai";
import { agentRuntime } from "../agent";
export const sendMessage = mutation({
args: {
groupId: v.id("groups"),
agentId: v.id("things"),
threadId: v.string(),
message: v.string(),
},
handler: async (ctx, args) => {
// Add message to thread
const agent = await ctx.db.get(args.agentId);
if (!agent) throw new Error("Agent not found");
// Get agent configuration
const config = await agentRuntime(ctx, {
groupId: args.groupId,
agentId: args.agentId,
threadId: args.threadId,
message: args.message,
});
// Get conversation context
const messages = await getMessages(ctx, {
threadId: args.threadId,
limit: 20,
});
// Stream text response
const { textStream, usage } = await streamText({
model: config.model,
system: config.systemPrompt,
tools: config.tools,
messages: [...messages, { role: "user", content: args.message }],
});
// Log AI call
await ctx.db.insert("aiCalls", {
groupId: args.groupId,
model: agent.properties.model,
provider: "openai",
prompt: args.message,
result: "", // Will be populated later
tokensUsed: { input: 0, output: 0 },
duration: 0,
timestamp: Date.now(),
});
// Update thread
await ctx.db.patch(args.threadId, {
updatedAt: Date.now(),
});
return {
threadId: args.threadId,
stream: textStream,
usage,
};
},
});
```
### WebSocket Streaming (Client)
```typescript
// web/src/lib/hooks/useAgentStream.ts
import { useMutation } from "convex/react";
import { api } from "@/convex/_generated/api";
import { useState } from "react";
export function useAgentStream(
groupId: string,
agentId: string,
threadId: string,
) {
const sendMessage = useMutation(api.mutations.agentChat.sendMessage);
const [messages, setMessages] = useState<
Array<{ role: string; content: string }>
>([]);
const [isStreaming, setIsStreaming] = useState(false);
const stream = async (userMessage: string) => {
setIsStreaming(true);
setMessages((prev) => [...prev, { role: "user", content: userMessage }]);
try {
const response = await sendMessage({
groupId,
agentId,
threadId,
message: userMessage,
});
// Handle streaming response
const reader = response.stream.getReader();
const decoder = new TextDecoder();
let fullResponse = "";
while (true) {
const { done, value } = await reader.read();
if (done) break;
const chunk = decoder.decode(value);
fullResponse += chunk;
// Update UI with streaming text
setMessages((prev) => {
const updated = [...prev];
const lastMsg = updated[updated.length - 1];
if (lastMsg?.role === "assistant") {
lastMsg.content = fullResponse;
} else {
updated.push({ role: "assistant", content: fullResponse });
}
return updated;
});
}
} catch (error) {
console.error("Stream error:", error);
} finally {
setIsStreaming(false);
}
};
return {
messages,
isStreaming,
stream,
};
}
```
## RAG Integration (Cycle 51-60)
### Connect Agent to Knowledge
```typescript
// backend/convex/agent/tools.ts (updated)
import { searchKnowledge } from "../services/rag";
export const agentTools = {
searchKnowledge: tool({
description: "Search the group's knowledge base",
parameters: z.object({
query: z.string(),
limit: z.number().default(5),
}),
execute: async (input, { ctx, groupId }) => {
const results = await searchKnowledge(ctx, {
groupId,
query: input.query,
limit: input.limit,
});
return {
results: results.map((r) => ({
title: r.title,
content: r.content,
relevance: r.relevance,
})),
};
},
}),
};
```
### RAG Service
```typescript
// backend/convex/services/rag.ts
import { QueryCtx } from "./_generated/server";
import { v } from "convex/values";
export async function searchKnowledge(
ctx: QueryCtx,
args: {
groupId: string;
query: string;
limit: number;
},
) {
// 1. Embed query using AI SDK
const queryEmbedding = await embedText(args.query);
// 2. Search knowledge table for similar embeddings
const results = await ctx.db
.query("knowledge")
.withIndex("by_groupId", (q) => q.eq("groupId", args.groupId))
.collect();
// 3. Calculate cosine similarity
const scored = results
.map((result) => ({
...result,
relevance: cosineSimilarity(queryEmbedding, result.embedding),
}))
.sort((a, b) => b.relevance - a.relevance)
.slice(0, args.limit);
return scored;
}
function cosineSimilarity(a: number[], b: number[]): number {
const dotProduct = a.reduce((sum, val, i) => sum + val * b[i], 0);
const magnitude =
Math.sqrt(a.reduce((sum, val) => sum + val * val, 0)) *
Math.sqrt(b.reduce((sum, val) => sum + val * val, 0));
return dotProduct / magnitude;
}
async function embedText(text: string): Promise<number[]> {
// Use OpenAI embeddings or similar
// Return 1536-dimensional vector
return [];
}
```
## Multi-Tenant Isolation (Cycle 71-80)
### Group Scoping
All agent operations are scoped to `groupId`:
```typescript
// ✅ Good: Scoped to group
export const listAgents = query({
args: { groupId: v.id("groups") },
handler: async (ctx, args) => {
return await ctx.db
.query("things")
.withIndex("by_type", (q) => q.eq("type", "agent"))
.filter((q) => q.eq(q.field("groupId"), args.groupId))
.collect();
},
});
// ❌ Bad: Not scoped
export const listAllAgents = query({
handler: async (ctx) => {
return await ctx.db.query("things").collect();
},
});
```
### Authorization
```typescript
// Verify user has access to group
async function verifyGroupAccess(ctx: any, groupId: string) {
const userId = ctx.auth?.getUserIdentity()?.tokenIdentifier;
if (!userId) throw new Error("Not authenticated");
// Check if user is member of group
const membership = await ctx.db
.query("connections")
.filter((q) =>
q.and(
q.eq(q.field("type"), "member_of"),
q.eq(q.field("groupId"), groupId),
q.eq(q.field("from"), userId),
),
)
.first();
if (!membership) throw new Error("Access denied");
return true;
}
```
## Usage Tracking & Billing (Cycle 81-90)
### Token Tracking
```typescript
// backend/convex/services/usage.ts
import { MutationCtx } from "./_generated/server";
export async function trackUsage(
ctx: MutationCtx,
args: {
groupId: string;
model: string;
provider: string;
inputTokens: number;
outputTokens: number;
},
) {
// Get group plan
const group = await ctx.db.get(args.groupId);
const plan = group?.properties?.plan || "starter";
// Calculate cost
const rates = {
openai: { input: 0.005, output: 0.015 },
anthropic: { input: 0.008, output: 0.024 },
};
const rate = rates[args.provider] || rates.openai;
const cost =
(args.inputTokens * rate.input + args.outputTokens * rate.output) / 1000;
// Log usage event
await ctx.db.insert("events", {
groupId: args.groupId,
type: "usage_tracked",
actorId: null,
targetId: null,
timestamp: Date.now(),
metadata: {
model: args.model,
provider: args.provider,
tokens: args.inputTokens + args.outputTokens,
cost,
plan,
},
});
return { cost, totalTokens: args.inputTokens + args.outputTokens };
}
```
### Usage Dashboard
```typescript
// backend/convex/queries/usage.ts
export const getGroupUsage = query({
args: { groupId: v.id("groups"), period: v.string() },
handler: async (ctx, args) => {
const now = Date.now();
const startTime = getPeriodStart(args.period, now);
const events = await ctx.db
.query("events")
.withIndex("by_groupId", (q) => q.eq("groupId", args.groupId))
.filter((q) => q.eq(q.field("type"), "usage_tracked"))
.filter((q) => q.gte(q.field("timestamp"), startTime))
.collect();
return {
totalTokens: events.reduce((sum, e) => sum + (e.metadata.tokens || 0), 0),
totalCost: events.reduce((sum, e) => sum + (e.metadata.cost || 0), 0),
byModel: groupBy(events, (e) => e.metadata.model),
byProvider: groupBy(events, (e) => e.metadata.provider),
};
},
});
```
## Error Handling & Recovery (Cycle 91-100)
### Graceful Error Handling
```typescript
// backend/convex/mutations/agentChat.ts (updated)
export const sendMessage = mutation({
args: {
groupId: v.id("groups"),
agentId: v.id("things"),
threadId: v.string(),
message: v.string(),
},
handler: async (ctx, args) => {
try {
// ... main flow ...
} catch (error) {
// Log error to events table
await ctx.db.insert("events", {
groupId: args.groupId,
type: "agent_error",
actorId: ctx.auth?.getUserIdentity()?.tokenIdentifier,
targetId: args.agentId,
timestamp: Date.now(),
metadata: {
error: error instanceof Error ? error.message : "Unknown error",
threadId: args.threadId,
},
});
// Return user-friendly error
return {
success: false,
error: "Failed to generate response. Please try again.",
};
}
},
});
```
### Retry Logic
```typescript
// web/src/lib/hooks/useAgentRetry.ts
export function useAgentRetry(maxRetries = 3) {
return async (fn: () => Promise<any>) => {
for (let i = 0; i < maxRetries; i++) {
try {
return await fn();
} catch (error) {
if (i === maxRetries - 1) throw error;
// Exponential backoff
await new Promise((resolve) =>
setTimeout(resolve, Math.pow(2, i) * 1000),
);
}
}
};
}
```
## Performance Optimization
| Optimization | Impact | Implementation |
| ------------------ | ------------------- | ------------------------------------ |
| Message pagination | Reduce context size | Load 10 messages initially, paginate |
| Embedding caching | Faster RAG | Cache query embeddings for 5 minutes |
| Tool call batching | Reduce API calls | Group multiple tool executions |
| Streaming chunking | Better UX | Stream 50-100 tokens at a time |
| Index optimization | Faster queries | Use proper indexes on groupId |
## Testing Strategy
### Unit Tests
```typescript
// test/agent/tools.test.ts
test("searchKnowledge returns relevant results", async () => {
const results = await searchKnowledge(ctx, {
groupId: "group_123",
query: "how to create a task?",
limit: 5,
});
expect(results).toHaveLength(5);
expect(results[0].relevance).toBeGreaterThan(0.5);
});
```
### Integration Tests
```typescript
// test/agent/integration.test.ts
test("full agent conversation flow", async () => {
// 1. Create group and agent
// 2. Create thread
// 3. Send message
// 4. Verify response streamed
// 5. Verify messages persisted
// 6. Verify usage tracked
});
```
## Common Patterns
### Pattern: Agent Context Injection
```typescript
// Inject group context into all agent operations
type AgentContext = {
groupId: string;
agentId: string;
userId: string;
thread: any;
};
async function withAgentContext<T>(
ctx: MutationCtx,
args: any,
handler: (context: AgentContext) => Promise<T>,
): Promise<T> {
const agent = await ctx.db.get(args.agentId);
return handler({
groupId: args.groupId,
agentId: args.agentId,
userId: ctx.auth?.getUserIdentity()?.tokenIdentifier || "",
thread: args.threadId,
});
}
```
### Pattern: Message Formatting
```typescript
// Format Convex messages for LLM
function formatMessagesForLLM(messages: any[]) {
return messages.map((msg) => ({
role: msg.role === "assistant" ? "assistant" : "user",
content: msg.content,
}));
}
```
## Deployment Checklist
- [ ] Install `@convex-dev/agent` package
- [ ] Add agent tables to schema
- [ ] Create agent thing with properties
- [ ] Implement streaming mutations
- [ ] Set up RAG integration
- [ ] Configure rate limiting
- [ ] Add usage tracking
- [ ] Test multi-tenant isolation
- [ ] Deploy to production
- [ ] Monitor thread creation rate
- [ ] Track token usage per group
## Next Steps
1. Install Convex Agents: `npm install @convex-dev/agent`
2. Update schema with agent tables
3. Create agent threads management mutations
4. Implement streaming response mutations
5. Connect to AI SDK generateText/streamText
6. Add RAG search tool
7. Test end-to-end flow
8. Deploy to production
## References
- [Convex Agents Documentation](https://www.convex.dev/components/agent)
- [Convex Database](https://docs.convex.dev/database)
- [Convex Mutations](https://docs.convex.dev/functions/mutations)
- [AI SDK Integration](./ai-sdk.md)
- [6-Dimension Ontology](../knowledge/ontology.md)