json-object-editor/docs/JOE_AI_Overview.md

JOE the Json Object Editor | Platform Edition

## JOE AI Overview

This document gives Custom GPTs, Assistants, and AI agents a single mental model for how AI works inside **Json Object Editor (JOE)**: which schemas are involved, how the `chatgpt` plugin behaves, how the `joe-ai` UI connects to OpenAI, and how **MCP** exposes JOE as tools.

---

## 1. High‑level AI Architecture

- **Core idea**: JOE is a schema‑driven data system. AI features are layered on top to:
  - Generate or transform field values (AI autofill)
  - Run persistent assistants and chats (in‑app + widgets)
  - Track AI job progress in real-time at the field level
  - Expose JOE data and writes via MCP tools for external agents
- **Main components**:
  - **AI schemas** (`ai_assistant`, `ai_prompt`, `ai_tool`, `ai_response`, `ai_widget_conversation`)
  - **Server plugins**:
    - `chatgpt.js` – **central / preferred** entry point for OpenAI (Responses API, tool calls, MCP bridge)
    - `chatgpt-assistants.js` – legacy Assistants‑API plugin (backward compatibility only)
    - `AiJobs.js` – job tracking and progress management
  - **Client / UI**:
    - `joe-ai.js` – in‑app AI chat UI (web components) and job polling
    - `field-jobs-container.js` – web component for displaying job progress
    - Standard JOE editor UI with **AI autofill** buttons on fields
  - **MCP**:
    - `/.well-known/mcp/manifest.json` – describes tools
    - `/mcp` – JSON‑RPC endpoint whose tools route into `JOE.Schemas`, `JOE.Storage`, `JOE.Cache`

When in doubt: **schemas define structure**, **plugins call OpenAI + MCP**, **MCP exposes tools**, and **UI (`joe-ai`) is just a thin chat/controls layer on top**.

### 1.1 End‑to‑end mental model

At a high level, almost every AI flow in JOE is one of these patterns:

- **Prompt / Autofill flow (schema‑driven)**  
  1. A human or field‑level `ai` config chooses an `ai_prompt` (and optionally an `ai_assistant`).  
  2. The UI calls `/API/plugin/chatgpt/<prompt_method>` (usually `executeJOEAiPrompt`) with `ai_prompt` id, parameters, and any referenced objects/files.  
  3. `chatgpt.js` loads the prompt, runs any helper `functions`, builds `instructions` + `input`, and normalizes MCP config (`mcp_*`) from the prompt and/or assistant.  
  4. `runWithTools` calls the OpenAI **Responses API**, optionally with tools (including MCP tools), then persists the result as an `ai_response` (plus any `generated_thoughts`, `used_openai_file_ids`, etc.).  
  5. Optional: helpers like `compareResponseToObject` / `updateObjectFromResponse` merge results back into a target object.

- **Widget / AIHub / Object chat flow (conversation‑driven)**  
  1. The UI renders `<joe-ai-widget>` (optionally scoped to an object and seeded with an `ai_assistant_id`).  
  2. On the first message, the widget calls `/API/plugin/chatgpt/widgetStart`, which creates an `ai_widget_conversation` with model, assistant, system text, and optional `scope_itemtype`/`scope_id`.  
  3. Each user turn calls `/API/plugin/chatgpt/widgetMessage`, which:  
     - Resolves the active `ai_assistant` by `_id` and loads its `instructions` + MCP config.  
     - Builds `systemText` (assistant instructions + MCP tool list + optional scope hints / `understandObject` snapshot).  
     - Gathers any attached files (from scoped objects or assistant‑level files) and passes their OpenAI ids/roles into `runWithTools`.  
  4. `runWithTools` performs a Responses+tools call (with MCP tools where enabled) and returns the assistant's reply; `widgetMessage` appends it to `ai_widget_conversation.messages` and returns the updated history to the widget.

- **MCP‑only agent flow (Custom GPT / external agent)**  
  1. The agent discovers tools and named toolsets via `/.well-known/mcp/manifest.json`.  
  2. For reads/writes it calls `POST /mcp` with tool names like `hydrate`, `search`, `fuzzySearch`, `getObject`, `saveObject`, and `saveObjects`.  
  3. When the agent needs higher‑level workflows (thoughts, prompts, assistants), it can call into the same HTTP APIs that `joe-ai` uses (e.g., `/API/plugin/chatgpt/autofill`, `/API/plugin/chatgpt/executeJOEAiPrompt`, widget endpoints), treating JOE as the authoritative orchestrator of OpenAI + MCP rather than re‑implementing orchestration itself.

---

## 2. AI Schemas

JOE uses a small set of AI‑specific schemas. A good agent should know what each represents and how they relate.

### 2.1 `ai_assistant`

- **What it is**: Configuration record for a single AI assistant linked to OpenAI.
- **Key fields**:
  - Identity & model: `name`, `info`, `ai_model`, `assistant_id` (optional/legacy)
  - Prompting: `instructions`, `assistant_thinking_text`, `assistant_color`
  - Tools: `tools` (JSON OpenAI tools array – often imported from MCP)
  - MCP config: `mcp_enabled`, `mcp_toolset`, `mcp_selected_tools`, `mcp_instructions_mode`
- **How it's used**:
  - One `ai_assistant` usually maps 1:1 to an OpenAI Assistant.
  - A **DEFAULT** can be set via `setting.DEFAULT_AI_ASSISTANT` and is used by `joe-ai` as the default assistant.
  - The schema exposes helper methods such as:
    - `syncAssistantToOpenAI` – sync this record to OpenAI via `chatgpt-assistants.js`.
    - `loadMCPTolsIntoAssistant` – pull tools from the MCP manifest into `assistant.tools`.
    - `setAsDefaultAssistant` – update the `DEFAULT_AI_ASSISTANT` setting.

**Agent note**: When reasoning about which assistant is active in a chat or widget, look for:
- `ai_widget_conversation.assistant` (JOE cuid)
- Or the instance‑level default via the `setting` schema.

### 2.2 `ai_prompt`

- **What it is**: Reusable AI **prompt configuration** – how to call the `chatgpt` plugin or Responses API.
- **Key fields**:
  - Identity: `_id`, `name`, `info`, `itemtype:'ai_prompt'`
  - Integration: `prompt_method` (e.g. `executeJOEAiPrompt`), `content_items` (objectList of `{ itemtype, reference }`)
  - Instructions: `functions` (helper code), `instructions_format`, `instructions`, `user_prompt`
  - OpenAI tuning: `ai_model`, `temperature`
  - MCP config: `mcp_enabled`, `mcp_toolset`, `mcp_selected_tools`, `mcp_instructions_mode`
- **Typical flows**:
  - **AI Autofill (field‑level)**: Fields include an `ai` config. The UI calls `/API/plugin/chatgpt/autofill`, which uses `ai_prompt` definitions and returns structured patch JSON.
  - **Explicit prompts**: UI actions call `/API/plugin/chatgpt/<prompt_method>?ai_prompt=<id>&...`. The plugin locates the prompt, merges helper `functions`, and runs OpenAI.

**Agent note**: Treat `ai_prompt` as the **source of truth** for how to talk to OpenAI for a particular workflow. Never invent `prompt_method` names; reuse existing ones or ask a human to add a new prompt record.

### 2.3 `ai_response`

- **What it is**: Persistent record of a single AI response, for **auditing**, **reuse**, and **merge‑back** into JOE.
- **Key fields**:
  - Links: `ai_prompt` (reference), `referenced_objects` (array of JOE `_id`s), `tags`
  - Request context: `user_prompt`, `prompt_method`, `model_used`, `response_type`
  - Response data: `response` (raw string), `response_keys`, `response_id`, `usage` (token usage)
  - MCP audit: `mcp_tools_used[]`, `used_openai_file_ids[]`
- **Methods**:
  - `compareResponseToObject(response_id, object_id, do_alert)` – validate response JSON vs object fields
  - `updateObjectFromResponse(response_id, object_id, fields)` – apply response values into a JOE object
  - `listResponses(obj)` – list AI responses referencing a given object

**Agent note**: When updating objects via tools, favor using `ai_response` records to compare/merge changes, especially if the response contains multiple fields.

### 2.4 `ai_widget_conversation`

- **What it is**: Lightweight conversation record for **embeddable AI widgets** (external sites, test pages, object chat).
- **Key fields**:
  - Participant & assistant: `user`, `assistant` (JOE cuid), `user_name`, `user_color`, `assistant_color`
  - OpenAI info: `model`, `assistant_id`, `system` (effective instructions)
  - Conversation: `messages` (JSON array of `{ role, content, created_at }`), `last_message_at`
  - Source: `source` (widget origin), `scope_itemtype`, `scope_id`, `tags`
- **Behavior**:
  - Accessed/updated via `chatgpt.js` widget endpoints: `widgetStart`, `widgetMessage`, `widgetHistory`
  - Schema exposes **subsets** and **filters** for grouping by `source`, `user`, and `assistant`

**Agent note**: `ai_widget_conversation` is the **primary** conversation schema for modern chats (`<joe-ai-widget>`, AIHub cards, object chat). The legacy `ai_conversation` schema is only used by older `<joe-ai-chatbox>` flows.

### 2.5 `ai_tool` (optional)

- **What it is**: Definition of a reusable **function‑calling tool** that OpenAI can call and JOE can execute server‑side.
- **Key fields**: `name`, `info`, `description`, `tool_id`, `tool_properties` (JSON), `server_code` (Node.js async code)
- **Agent note**: For tools exposed via MCP, the canonical shape is in the **MCP manifest**; `ai_tool` is more for high‑level catalog and advanced OpenAI tools, not the core MCP schema/search/save tools.

---

## 3. Server Plugins

### 3.1 `server/plugins/chatgpt.js` (Core Plugin)

This is the **core server plugin** that wires JOE to OpenAI and MCP.

- **Responsibilities**:
  - Read OpenAI API key from `setting.OPENAI_API_KEY`
  - Provide shared helpers for building OpenAI clients (`OpenAI` SDK)
  - Implement orchestration for **Responses API** calls (models like `gpt-4.1`, `gpt-4.1-mini`, `gpt-5.1`)
  - **Tool calling** and follow‑up calls via MCP tools
  - Bridge between JOE and MCP with helpers like:
    - `callMCPTool(toolName, params, ctx)` – call MCP tools **in‑process** (without HTTP)
    - Response parsing helpers such as `extractToolCalls(response)`
    - Payload shrinking helpers like `shrinkUnderstandObjectMessagesForTokens(...)`
- **Expose routes** under `/API/plugin/chatgpt/...`:
  - Field autofill (`autofill`) driven by schema‑level `ai` configs
  - Prompt execution handlers for `ai_prompt.prompt_method` values
  - Widget endpoints: `widgetStart`, `widgetMessage`, `widgetHistory`
- **Key patterns**:
  - **Slimming payloads**: When embedding object graphs into messages (e.g. `understandObject` results), `chatgpt.js` converts them into a slim representation to avoid token limits
  - **Error handling**: `isTokenLimitError(err)` identifies token/TPM error shapes for better logs and fallbacks
  - **Job tracking**: Registers and updates AI jobs via `AiJobs` module for progress visibility

**Agent note**: Any time you see endpoints in docs like `/API/plugin/chatgpt/<method>`, they resolve into functions in `chatgpt.js`. Follow those conventions instead of inventing new paths.

### 3.2 `server/modules/AiJobs.js` (Job Tracking)

Manages active AI job storage and provides API endpoints for real-time progress tracking.

- **Storage**: In-memory `active` object keyed by `{objectId}_{fieldName}`
- **Job Structure**:
  ```javascript
  {
    token: string,           // Full token: {objectId}|{fieldName}|{timestamp}|{userid}
    startTime: ISO string,  // Job start timestamp
    status: string,          // 'starting', 'running', 'complete', 'error'
    promptId: string|null,  // Optional prompt ID
    promptName: string|null, // Human-readable job name
    fieldId: string|null,    // Field identifier
    progress: number,        // Current progress (0-100 or 0-1)
    total: number|null,      // Total steps (typically 100 or 1)
    message: string          // Status message
  }
  ```
- **Key Functions**:
  - `createJob(token, jobData)` - Register a new job
  - `updateJob(token, updates)` - Update job status/progress/message
  - `removeJob(token)` - Remove job immediately
  - `removeJobWithDelay(token, status, message, delaySeconds)` - Remove after delay (default 10s)
  - `extractKey(token)` - Parse token to get `{objectId}_{fieldName}` lookup key
- **API Endpoints**:
  - `GET /API/aijobs/:objectId/:fieldName` - Returns `{ jobs: [...], objectId, fieldName, count }` for a specific field (flat array)
  - `GET /API/aijobs/:objectId` - Returns `{ jobs: [{ lookupKey, jobs: [...] }], objectId, count }` for an object (grouped by field)
  - `GET /API/aijobs` - Returns `{ jobs: [{ lookupKey, jobs: [...] }], count, lookupKeys }` for all active jobs (grouped, optional/admin)

**Agent note**: Jobs are automatically registered by `chatgpt.js` and `MCP.js` when AI operations start. The token format uses `|` as delimiter to avoid conflicts with field names containing underscores.

### 3.3 Legacy Plugins (Backward Compatibility Only)

- **`chatgpt-assistants.js`**: Manages OpenAI Assistants and threads using the **older Assistants API**. Used by legacy `<joe-ai-chatbox>` flows. **New integrations should use the Responses‑based `chatgpt` plugin instead**.
- **`chatgpt-responses.js`**: Helpers for working with Responses API results (extracting text, tool calls, normalizing outputs).
- **`chatgpt-tools.js`**: Glue between AI tools and JOE (uses `ai_tool` definitions, executes server‑side code blocks).

**Agent note**: You typically don't call these plugins directly. Instead, use documented **APIs** (`/API/plugin/chatgpt/*`, widget endpoints) and/or **MCP tools** for reads/writes.

---

## 4. Client-Side UI (`joe-ai.js`)

The `js/joe-ai.js` file implements JOE's in‑app AI user interface using custom elements and an `Ai` namespace.

### 4.1 Chat Components

- **`<joe-ai-widget>`** – Modern embeddable chat component that talks to `chatgpt.js` (`widgetStart`, `widgetMessage`, `widgetHistory`). Bound to `ai_widget_conversation`.
- **`<joe-ai-assistant-picker>`** – Selector that applies an `ai_assistant._id` to a widget (`ai_assistant_id` attribute).
- **`<joe-ai-chatbox>`** – Legacy component that loads `ai_conversation` and uses `chatgpt-assistants` endpoints. Still present for existing flows but not used by new object chat or AIHub widget experiences.
- **`Ai.openObjectChatLauncher(object_id, itemtype, name)`** – Launches a floating shell containing a `<joe-ai-widget>` for **object‑scoped chat**:
  - Sets `source:"object_chat"`, `scope_itemtype`, `scope_id`, and passes the current user id
  - Uses `DEFAULT_AI_ASSISTANT` as the initial assistant, but allows per‑conversation overrides via the picker

### 4.2 Job Tracking & Polling

- **Global Poller**: `Ai.jobsPoller` object manages polling for all `field-jobs-container` components
- **Polling Intervals**:
  - **Active jobs present**: 3 seconds
  - **No active jobs**: 10 seconds (idle)
- **Immediate Polling**: When an AI button is clicked, `Ai.jobsPoller.poll()` is called immediately (with 200ms delay to let server register the job)
- **Poll Logic**:
  1. Finds all `field-jobs-container` elements on the page
  2. For each container, polls `/API/aijobs/{objectId}/{fieldName}`
  3. Updates container via `container.updateJobs(jobs)`
  4. Adjusts interval based on whether any active jobs were found
- **Token Generation**: `Ai.generateProgressToken(objectId, fieldName, userID)` creates structured tokens: `{objectId}|{fieldName}|{timestamp}|{userid}`
- **Debug Functions** (available in browser console):
  - `Ai.checkPoller()` - Shows poller status, interval, and container count
  - `Ai.restartPoller()` - Restarts the global poller
  - `Ai.debugJobs()` - Shows all active jobs in `Ai.activeJobs` Map

### 4.3 UI Component (`field-jobs-container`)

- **Web Component**: `<field-jobs-container>` custom element defined in `web-components/field-jobs-container.js`
- **Attributes**:
  - `data-object-id` - Object CUID
  - `data-field-name` - Field name
- **Rendering**:
  - Title shows active job count: "N active job(s)"
  - Token link on right side (small, bold) - links to `/API/aijobs/{objectId}/{fieldName}` endpoint page
  - Each job row shows:
    - Job name (from `promptName` or `promptId`)
    - Status (capitalized, inline): " - Running", " - Complete", etc.
    - Elapsed time: "(Xs)" in seconds
    - Progress percentage: "XX%" or "—" if not available
    - Message on separate line (smaller, lighter text)
  - Completed jobs remain visible until page/field refresh
- **Lifecycle**:
  - `connectedCallback()` triggers immediate poll on mount
  - `updateJobs(jobs)` updates the display
  - Elapsed time updates every second for active jobs

### 4.4 Field Integration

To enable job tracking on a field:

1. **Add `queryJobs: true` to field definition**:
   ```javascript
   {
     name: 'select_prompt',
     type: 'select_prompt',
     queryJobs: true  // Enables field-jobs-container rendering
   }
   ```

2. **Ensure button has required data attributes**:
   - `data-object-id` - Set automatically from current object
   - `data-field-name` - Set from field name
   - `data-progress-token` - Generated via `Ai.generateProgressToken()`

3. **Server-side job registration**:
   - Autofill: `chatgpt.js` → `registerAiJobIfToken()` → `updateAiJobIfToken()`
   - Prompts: `chatgpt.js` → `executeJOEAiPrompt()` → `updateAiJobIfToken()`
   - Thoughts: `MCP.js` → `runThoughtAgent()` → `MCP.updateJob()`

### 4.5 Job Lifecycle

1. **Start**: Button click → Generate token → Register job on server → Trigger immediate poll
2. **Progress**: Server calls `updateJob(token, { status, message, progress, total })` at key stages
3. **Complete**: Server sets `status: 'complete'`, `progress: 100` → Job remains visible in UI
4. **Cleanup**: Server calls `removeJobWithDelay(token, 'complete', 'Complete', 10)` → Job removed from server after 10 seconds, but remains in UI until refresh

**Agent note**: When adding new AI operations that should show progress:
1. Generate token using `Ai.generateProgressToken(objectId, fieldName, userID)`
2. Register job on server via `AiJobs.createJob(token, jobData)`
3. Update progress via `AiJobs.updateJob(token, { status, message, progress, total })`
4. Ensure the field has `queryJobs: true` if using `field-jobs-container`

---

## 5. MCP: How JOE Exposes Tools to Agents

MCP (Model Context Protocol) is how external agents (including Custom GPTs) call into JOE as a **tool provider**.

### 5.1 Endpoints and Manifest

- **Endpoints**:
  - `GET /.well-known/mcp/manifest.json` – public manifest with instance info, tool list, privacy/terms URLs
  - `POST /mcp` – auth‑protected JSON‑RPC 2.0 endpoint for **tool calls** (same auth rules as other APIs)
- **Implementation**: The concrete MCP tool implementations live in `server/modules/MCP.js`, which is wired up by `server/init.js` and exported via the manifest and `/mcp` endpoint.
- **Tools**:
  - `listSchemas(name?)`, `getSchema(name)`
  - `getObject(_id, itemtype?)` (supports optional `flatten` and `depth`)
  - `search` (unified search across cache/storage; supports `source`, `ids`, `flatten`, `depth`, `countOnly`)
  - `fuzzySearch` – typo‑tolerant search across weighted fields
  - `findObjectsByTag`, `findObjectsByStatus` – tag/status-based queries
  - `saveObject({ object })`
  - `saveObjects({ objects, stopOnError?, concurrency? })`
  - `hydrate` – returns core fields, schemas, statuses, tags (for agents to bootstrap context)

All tools map directly into JOE's APIs: `JOE.Schemas`, `JOE.Storage`, `JOE.Cache`, and related helpers. Sensitive fields are sanitized before returning data.

### 5.2 Agent Best Practices with MCP

- **Schema‑first**: Always discover schemas via `listSchemas`/`getSchema`, use `hydrate` on startup to cache schemas/statuses/tags, use `search` / `fuzzySearch` with explicit `itemtype` and filters
- **Reads vs writes**: Prefer **cache** for read operations (`search` default) unless you explicitly need DB‑level freshness (set `source:"storage"`). For writes, use `saveObject` for single‑item updates, `saveObjects` for batched updates
- **Performance**: Use `countOnly` before large reads to avoid pulling massive datasets. When embedding objects into prompts, prefer **slimmed** representations (id, itemtype, name, info)

**Agent note**: The MCP tools are your primary way to **inspect and modify JOE data** from a Custom GPT or other agent host. Direct HTTP calls to JOE's internal `/API/*` routes should be treated as implementation detail unless explicitly allowed in your instructions.

---

## 6. Recommended RAG / Context Files for JOE AI Agents

When configuring a JOE‑aware Custom GPT or dev agent, include the following docs in its RAG/context:

- **Core architecture & instance info**
  - `README.md` – especially "Architecture & Mental Model (Server)" and MCP overview + test instructions
  - `CHANGELOG.md` – AI/MCP‑related entries (0.10.43x+, 0.10.50x+, 0.10.62x+)
- **Agent / instruction docs (existing)**
  - `docs/JOE_Master_Knowledge_Export.md` – master context dump for schemas and concepts
  - `docs/joe_agent_spec_v_2.2.md` – agent behavior/specification
  - `docs/joe_agent_custom_gpt_instructions_v_3.md` – current Custom GPT / MCP prompt instructions
  - `docs/schema_summary_guidelines.md` – how schema summaries are constructed
- **This file**
  - `docs/JOE_AI_Overview.md` – **(you are here)** high‑level map of AI schemas, plugins, UI, and MCP

Optional / advanced (for deeper RAG or power users):
- Selected excerpts or exports of **schema summaries** for AI‑related schemas
- Any **project‑specific** AI workflow docs you maintain

---

## 7. What's Still Missing / Future Docs to Consider

For a fully self‑sufficient agent, the following additional docs can be helpful:

- **AI Workflows Cookbook**: Short task‑oriented examples (summarize a project, draft a plan, merge responses)
- **AI Safety & Guardrails**: Which schemas/fields are sensitive, allowed vs disallowed writes, expectations around review vs autonomous changes
- **Instance‑specific Prompts / Taxonomy**: Any custom `ai_prompt` records or `ai_tool` definitions that embody local conventions

If these don't exist yet, an agent should assume **conservative** behavior:
- Prefer read + propose changes over direct writes
- Tag all AI‑generated content clearly
- Use `ai_response` + compare/merge flows rather than overwriting objects blindly