json-object-editor/docs/JOE_AI_Overview.md

JOE the Json Object Editor | Platform Edition

## JOE AI Overview

This document gives Custom GPTs, Assistants, and AI agents a single mental model for how AI works inside **Json Object Editor (JOE)**: which schemas are involved, how the `chatgpt` plugin behaves, how the `joe-ai` UI connects to OpenAI, and how **MCP** exposes JOE as tools.

---

## 1. High‑level AI Architecture

- **Core idea**: JOE is a schema‑driven data system. AI features are layered on top to:
  - Generate or transform field values (AI autofill)
  - Run persistent assistants and chats (in‑app + widgets)
  - Expose JOE data and writes via MCP tools for external agents
- **Main components**:
  - **AI schemas** (`ai_assistant`, `ai_prompt`, `ai_tool`, `ai_response`, `ai_conversation`, `ai_widget_conversation`)
  - **Server plugins**:
    - `chatgpt.js` – **central / preferred** entry point for OpenAI (Responses API, tool calls, MCP bridge). New integrations should target this plugin.
    - `chatgpt-assistants.js` – legacy Assistants‑API plugin, still used by existing in‑app chat flows (`ai_conversation` / `<joe-ai-chatbox>`) but not recommended for new work.
    - `chatgpt-tools.js` / `chatgpt-responses.js` – helpers for tools + response shaping
  - **Client / UI**:
    - `joe-ai.js` – in‑app AI chat UI (web components) wired to `ai_conversation` + assistants
    - Standard JOE editor UI with **AI autofill** buttons on fields, driven by `ai_prompt`
  - **MCP**:
    - `/.well-known/mcp/manifest.json` – describes tools
    - `/mcp` – JSON‑RPC endpoint whose tools route into `JOE.Schemas`, `JOE.Storage`, `JOE.Cache`

When in doubt: **schemas define structure**, **plugins call OpenAI + MCP**, **MCP exposes tools**, and **UI (`joe-ai`) is just a thin chat/controls layer on top.

---

## 2. AI Schemas (Mental Model)

JOE uses a small set of AI‑specific schemas. A good agent should know what each represents and how they relate.

### 2.1 `ai_assistant`

- **What it is**: Configuration record for a single AI assistant linked to OpenAI.
- **Key fields** (from schema + summaries):
  - Identity & model: `name`, `info`, `ai_model`, `assistant_id`, `openai_assistant_version`
  - Capabilities: `file_search_enabled`, `code_interpreter_enabled`
  - Prompting: `instructions`, `assistant_thinking_text`, `assistant_color`
  - Tools: `tools` (JSON OpenAI tools array – often imported from MCP), `datasets`, `tags`, `status`
  - Sync meta: `last_synced`, timestamps (`created`, `joeUpdated`)
- **How it’s used**:
  - One `ai_assistant` usually maps 1:1 to an OpenAI Assistant.
  - A **DEFAULT** can be set via `setting.DEFAULT_AI_ASSISTANT` and is used by `joe-ai` as the default assistant.
  - The schema exposes helper methods such as:
    - `syncAssistantToOpenAI` – sync this record to OpenAI via `chatgpt-assistants.js`.
    - `loadMCPTolsIntoAssistant` – pull tools from the MCP manifest into `assistant.tools`.
    - `setAsDefaultAssistant` – update the `DEFAULT_AI_ASSISTANT` setting.

**Agent note**: When reasoning about which assistant is active in a chat or widget, look for:
- `ai_conversation.assistant` or `ai_widget_conversation.assistant`
- Or the instance‑level default via the `setting` schema.

### 2.2 `ai_prompt`

- **What it is**: Reusable AI **prompt configuration** – how to call the `chatgpt` plugin or Responses API.
- **Key fields**:
  - Identity: `_id`, `name`, `info`, `itemtype:'ai_prompt'`
  - Integration with plugin:
    - `prompt_method` – name of the server method on `chatgpt.js` to call (e.g. `executeJOEAiPrompt`).
    - `content_items` – `objectList` of `{ itemtype, reference }` describing which JOE objects to send and under what parameter name.
  - Instructions:
    - `functions` – helper code snippet (Node‑style `module.exports = async function(...) { ... }`) used to shape instructions/input.
    - `instructions_format` – format hint; `instructions` – main system‑level text.
    - `user_prompt` – optional user‑facing prompt template or per‑call input description.
  - OpenAI tuning: `ai_model`, `temperature`
  - Meta: `status`, `tags`, `datasets`, timestamps
- **Methods / behavior**:
  - `methods.buildURL(prompt, items)` – builds URLs like:
    - `/API/plugin/chatgpt/<prompt_method>?ai_prompt=<prompt._id>&<reference>=<item._id>...`
  - `methods.listExamples(prompt)` – picks sample content objects for exploration/testing.
- **Typical flows**:
  - **AI Autofill (field‑level)**:
    - Fields in other schemas include an `ai` config (documented in `CHANGELOG` + README).
    - The UI calls `/API/plugin/chatgpt/autofill`, which:
      - Uses `ai_prompt` definitions and schema metadata.
      - Sends the right JOE objects + instructions to OpenAI.
      - Returns structured patch JSON to update fields.
  - **Explicit prompts**:
    - UI actions or external tools call `/API/plugin/chatgpt/<prompt_method>?ai_prompt=<id>&...`.
    - The plugin locates the prompt, merges helper `functions`, and runs OpenAI.

**Agent note**: Treat `ai_prompt` as the **source of truth** for how to talk to OpenAI for a particular workflow (summaries, planning, refactors, etc). Never invent `prompt_method` names; reuse existing ones or ask a human to add a new prompt record.

### 2.3 `ai_response`

- **What it is**: Persistent record of a single AI response, for **auditing**, **reuse**, and **merge‑back** into JOE.
- **Key fields**:
  - Links: `ai_prompt` (reference), `referenced_objects` (array of JOE `_id`s used), `tags`
  - Request context: `user_prompt`, `prompt_method`, `model_used`, `response_type`
  - Response data:
    - `response` – raw string body (often JSON or text).
    - `response_keys` – array of keys returned (used for patch/merge).
    - `response_id` – OpenAI response ID.
    - `usage` – token usage object `{ input_tokens, output_tokens, total_tokens }`.
- **Methods**:
  - `compareResponseToObject(response_id, object_id, do_alert)` – validate response JSON vs object fields, optionally trigger a merge.
  - `updateObjectFromResponse(response_id, object_id, fields)` – apply response values into a JOE object and update tags.
  - `listResponses(obj)` – list AI responses referencing a given object.

**Agent note**: When you are updating objects via tools, favor:
- Using `ai_response` records to compare/merge changes, especially if the response contains multiple fields.
- Respect existing merge flows (`compareResponseToObject` / `updateObjectFromResponse`) instead of overwriting arbitrary fields.

### 2.4 `ai_tool`

- **What it is**: Definition of a reusable **function‑calling tool** that OpenAI can call and JOE can execute server‑side.
- **Key fields**:
  - Identity: `name`, `info`, `description`, `tool_id`
  - OpenAI schema: `tool_properties` (JSON, typically `{ type:'function', function:{ name, description, parameters } }`)
  - Execution: `server_code` – Node.js async code to run when the tool is invoked.
  - Meta: `datasets`, `tags`, timestamps.
- **Methods**:
  - `updateToolProperties(propObj)` – helper to keep `tool_properties.function.name` in sync with `tool_id`.

**Agent note**: For tools exposed via MCP, the canonical shape is actually in the **MCP manifest**; `ai_tool` is more for:
- High‑level catalog and advanced OpenAI tools, not the core MCP schema/search/save tools.

### 2.5 `ai_conversation`

- **What it is**: Metadata record for **in‑app AI conversations** (staff/admin chat in JOE).
- **Key fields**:
  - Participants: `user` (JOE user), `assistant` (`ai_assistant`), `members` (external participants)
  - OpenAI thread: `thread_id`
  - Summary: `summary` (WYSIWYG), `status`, `tags`
  - Meta: `_id`, `name`, `info`, timestamps
- **Behavior**:
  - Messages are **not stored** in JOE; they’re fetched from OpenAI at runtime (via `chatgpt-assistants.js`).
  - Schema methods + UI:
    - `methods.chatSpawner(object_id)` – calls `_joe.Ai.spawnContextualChat(object_id)`.
    - List view shows created time, user cubes, and context objects used for that chat.
  - `joe-ai.js` uses `ai_conversation` as the anchor object for chat history + context.

**Agent note**: If you need to understand a chat history, you:
- Read `ai_conversation` for metadata and context_objects.
- Use **MCP tools** or existing chat APIs to fetch live thread content; don’t invent local persistence.

### 2.6 `ai_widget_conversation`

- **What it is**: Lightweight conversation record for **embeddable AI widgets** (external sites, test pages).
- **Key fields**:
  - Participant & assistant: `user`, `assistant`, `user_name`, `user_color`, `assistant_color`
  - OpenAI info: `model`, `assistant_id`, `system` (effective instructions)
  - Conversation: `messages` (compact list, typically JSON array of `{ role, content, created_at }`), `last_message_at`
  - Source: `source` (widget origin), `tags`, timestamps
- **Behavior**:
  - Accessed/updated via `chatgpt-assistants` widget endpoints:
    - `widgetStart`, `widgetMessage`, `widgetHistory`, etc.
  - Schema exposes **subsets** and **filters** for grouping by `source`, `user`, and `assistant`.

**Agent note**: Treat `ai_widget_conversation` as the external/chat‑widget equivalent of `ai_conversation`. The fields are tuned for widgets and public UI, but the mental model is the same: small record pointing to assistant + messages.

---

## 3. `chatgpt` Plugin and Related Plugins

### 3.1 `server/plugins/chatgpt.js`

This is the **core server plugin** that wires JOE to OpenAI and MCP.

- **Responsibilities**:
  - Read OpenAI API key from `setting.OPENAI_API_KEY`.
  - Provide shared helpers for building OpenAI clients (`OpenAI` SDK).
  - Implement orchestration for:
    - **Responses API** calls (models like `gpt-4.1`, `gpt-4.1-mini`, `gpt-5.1`).
    - **Tool calling** and follow‑up calls via MCP tools.
  - Bridge between JOE and MCP with helpers like:
    - `callMCPTool(toolName, params, ctx)` – call MCP tools **in‑process** (without HTTP).
    - Response parsing helpers such as `extractToolCalls(response)`.
    - Payload shrinking helpers like `shrinkUnderstandObjectMessagesForTokens(...)`.
  - Expose specific routes under `/API/plugin/chatgpt/...`, including:
    - Field autofill (`autofill`) driven by schema‑level `ai` configs.
    - Prompt execution handlers for `ai_prompt.prompt_method` values.
- **Key patterns**:
  - **Slimming payloads**: When embedding object graphs into messages (e.g. `understandObject` results), `chatgpt.js` converts them into a slim representation to avoid token limits.
  - **Error handling**:
    - `isTokenLimitError(err)` identifies token/TPM error shapes for better logs and fallbacks.

**Agent note**: Any time you see endpoints in docs like `/API/plugin/chatgpt/<method>`, they resolve into functions in `chatgpt.js` or its siblings. Follow those conventions instead of inventing new paths.

### 3.2 `chatgpt-assistants.js`, `chatgpt-responses.js`, `chatgpt-tools.js`

- **`chatgpt-assistants.js`**:
  - Manages OpenAI Assistants and threads using the **older Assistants API**.
  - Endpoints such as:
    - `syncAssistantToOpenAI` (used from `ai_assistant.methods`).
    - `getThreadMessages`, `runAssistantChat`, widget endpoints like `widgetStart`, `widgetMessage`, `widgetHistory`.
  - Used by `joe-ai.js` and widget UIs to pull/push messages for a given conversation.
  - **Status**: legacy bridge for existing flows; **new integrations should use the Responses‑based `chatgpt` plugin instead**, matching the README.

- **`chatgpt-responses.js`**:
  - Helpers for working with Responses API results:
    - Extracting text and tool calls.
    - Normalizing outputs for storage in `ai_response`.
  - Often called from `chatgpt.js` orchestration functions.

- **`chatgpt-tools.js`**:
  - Glue between AI tools and JOE:
    - Uses `ai_tool` definitions.
    - Executes server‑side code blocks for tools when OpenAI calls them.
  - Can also support MCP‑style tools exported via the manifest.

**Agent note**: You typically don’t call these plugins directly from a GPT. Instead, you:
- Use documented **APIs** (`/API/plugin/chatgpt/*`, widget endpoints) and/or
- Use **MCP tools** for reads/writes.

---

## 4. `joe-ai.js` – In‑App AI UI

The `js/joe-ai.js` file implements JOE’s in‑app AI user interface using custom elements like `<joe-ai-chatbox>` and other helpers under a `Ai` namespace.

- **Key responsibilities**:
  - Manage open chat boxes and default assistant selection:
    - `Ai._openChats`, `Ai.default_ai`, `Ai.getDefaultAssistant()`.
  - Render **chat UI** bound to an `ai_conversation`:
    - Loads `ai_conversation` via `/API/object/ai_conversation/_id/<id>`.
    - Uses `chatgpt-assistants` endpoints (e.g. `getThreadMessages`) to fetch messages.
    - Shows assistant/user labels, context objects, and a message history.
  - Provide ergonomic chat behaviors:
    - Markdown rendering via `renderMarkdownSafe` (prefers `marked` + `DOMPurify`).
    - Greeting messages based on `user` and `context_objects`.
    - Keyboard handling (Enter to send).
- **High‑level flow** for in‑app chat:
  1. User opens an `ai_conversation` or clicks “Continue Conversation” from a context object.
  2. `joe-ai.js` spawns a `<joe-ai-chatbox>` bound to that `ai_conversation`.
  3. The chatbox:
     - Fetches conversation + thread from the server.
     - Uses `chatgpt-assistants` to send/receive messages to/from OpenAI.
     - Renders messages and any embedded JOE object markers into the UI.

**Agent note**: When describing UI behavior or helping with debugging, remember that `joe-ai.js` is **UI only**; persistent data lives in:
- `ai_conversation` (metadata) + OpenAI threads
- `ai_assistant` (config)
- `ai_widget_conversation` for external widgets

---

## 5. MCP: How JOE Exposes Tools to Agents

MCP (Model Context Protocol) is how external agents (including Custom GPTs) call into JOE as a **tool provider**.

### 5.1 Endpoints and Manifest

- **Endpoints**:
  - `GET /.well-known/mcp/manifest.json` – public manifest with:
    - Instance info `{ joe: { name, version, hostname } }`
    - Tool list (name, description, params schema)
    - Privacy/terms URLs (also wired into `/privacy` and `/terms` routes).
  - `POST /mcp` – auth‑protected JSON‑RPC 2.0 endpoint for **tool calls**.
    - Same auth rules as other APIs; if users exist, requires cookie or basic auth.
- **Implementation note**:
  - The concrete MCP tool implementations live in `server/modules/MCP.js`, which is wired up by `server/init.js` and exported via the manifest and `/mcp` endpoint.
- **Tools** (from README + CHANGELOG):
  - `listSchemas(name?)`, `getSchema(name)`
  - `getObject(_id, itemtype?)`
  - `search` (unified search across cache/storage; supports `source`, `ids`, `flatten`, `depth`, `countOnly`)
  - `fuzzySearch` – typo‑tolerant search across weighted fields
  - `saveObject({ object })`
  - `saveObjects({ objects, stopOnError?, concurrency? })`
  - `hydrate` – returns core fields, schemas, statuses, tags (for agents to bootstrap context)

All tools map directly into JOE’s APIs: `JOE.Schemas`, `JOE.Storage`, `JOE.Cache`, and related helpers. Sensitive fields are sanitized before returning data.

### 5.2 Agent Best Practices with MCP

- **Schema‑first**: Always:
  1. Discover schemas via `listSchemas`/`getSchema`.
  2. Use `hydrate` on startup to cache schemas/statuses/tags.
  3. Use `search` / `fuzzySearch` with explicit `itemtype` and filters rather than arbitrary text mining.
- **Reads vs writes**:
  - Prefer **cache** for read operations (`search` default) unless you explicitly need DB‑level freshness (set `source:"storage"`).
  - For writes, use:
    - `saveObject` for single‑item updates.
    - `saveObjects` for batched updates with clear error handling.
- **Performance**:
  - Use `countOnly` before large reads to avoid pulling massive datasets.
  - When embedding objects into prompts, prefer **slimmed** representations (id, itemtype, name, info).

**Agent note**: The MCP tools are your primary way to **inspect and modify JOE data** from a Custom GPT or other agent host. Direct HTTP calls to JOE’s internal `/API/*` routes should be treated as implementation detail unless explicitly allowed in your instructions.

---

## 6. Recommended RAG / Context Files for JOE AI Agents

When configuring a JOE‑aware Custom GPT or dev agent, include the following docs in its RAG/context:

- **Core architecture & instance info**
  - `README.md` – especially:
    - “Architecture & Mental Model (Server)”
    - MCP overview + test instructions
  - `CHANGELOG.md` – AI/MCP‑related entries (0.10.43x+, 0.10.50x+, 0.10.62x+), already summarize important AI behavior.
- **Agent / instruction docs (existing)**
  - `docs/JOE_Master_Knowledge_Export.md` – master context dump for schemas and concepts.
  - `docs/joe_agent_spec_v_2.2.md` – agent behavior/specification.
  - `docs/joe_agent_custom_gpt_instructions_v_3.md` – current Custom GPT / MCP prompt instructions.
  - `docs/schema_summary_guidelines.md` – how schema summaries are constructed and how agents should interpret/use them.
- **This file**
  - `docs/JOE_AI_Overview.md` – **(you are here)** high‑level map of AI schemas, plugins, UI, and MCP.

Optional / advanced (for deeper RAG or power users):
- Selected excerpts or exports of **schema summaries** for AI‑related schemas:
  - `ai_assistant`, `ai_prompt`, `ai_tool`, `ai_response`, `ai_conversation`, `ai_widget_conversation`
- Any **project‑specific** AI workflow docs you maintain (e.g. playbooks for content generation, planning workflows, merge policies).

---

## 7. What’s Still Missing / Future Docs to Consider

For a fully self‑sufficient agent, the following additional docs can be helpful:

- **AI Workflows Cookbook**:
  - Short task‑oriented examples:
    - “Summarize a project using schemas + ai_prompt + MCP search.”
    - “Draft a plan then write objects via `saveObjects`.”
    - “Use ai_response to compare vs existing data and merge.”
- **AI Safety & Guardrails**:
  - Which schemas/fields are sensitive.
  - Allowed vs disallowed writes.
  - Expectations around **review vs autonomous changes**.
- **Instance‑specific Prompts / Taxonomy**:
  - Any custom `ai_prompt` records or `ai_tool` definitions that embody local conventions (naming standards, planning templates, scoring rubrics, etc.).

If these don’t exist yet, an agent should assume **conservative** behavior:
- Prefer read + propose changes over direct writes.
- Tag all AI‑generated content clearly.
- Use `ai_response` + compare/merge flows rather than overwriting objects blindly.