json-object-editor/docs/JOE_Master_Knowledge_Export.md

JOE the Json Object Editor | Platform Edition

# JOE (Json Object Editor) — Master Knowledge
Last updated: Oct 2025

## What is JOE?
JOE is a schema‑first, lightweight, extensible admin platform and data workbench. It manages structured objects via dynamic, schema‑driven forms and list views.

- Server/Platform mode: Node.js app with storage backends, cache, plugins, apps, web UI, and MCP tools for agents.
- Client‑only mode: Embed `JsonObjectEditor` to render/edit objects in‑page using schemas.

## Purpose and who it's for
- JOE turns whatever your organization does into living JSON objects with relationships and history. If it can be described, it can be represented and managed in JOE.
- For any org size or type: businesses, non‑profits, churches, families, clubs, and projects.
- Use JOE to:
  - Model your domain (schemas) and manage data with zero/low code
  - Operate with lists, dashboards, statuses, tags, and reports
  - Coordinate across teams via curated apps per use‑case
  - Empower agents (via MCP) to search, reconcile, and update data safely
- Benefits: rapid time‑to‑value, consistent UI/flows, single source of truth, clean audit history, and agent‑friendly APIs.

## Where JOE fits (when to use vs not)
- Use JOE when you need:
  - Schema‑first CRUD/admin surfaces with evolving data models
  - Unified views over multiple datasets with fast cache reads
  - Lightweight workflows (statuses/tags), reports, and agent tooling
- Not a great fit when you need:
  - Heavy bespoke transactional systems requiring complex ACID workflows
  - Offline‑first mobile or hard real‑time (sub‑ms) constraints
  - Highly specialized UIs better served by purpose‑built apps (you can still use JOE as the data/admin layer)

## Core principles
- Schema‑first: behavior and UI flow derive from schemas.
- Separation of concerns: schemas define data and rules; UI renders dynamically.
- Minimal overhead: lean client rendering, unified server APIs, cache‑first reads.
- Extensibility: apps, plugins, and instance schemas compose capabilities.
- Single‑object overlay: focused edit/view with list/detail navigation.

## Mental model
- Objects conform to schemas; schemas define fields, validation, menus, filters, events.
- Reads prefer cache for speed; writes go through storage for durability.
- `saveObject` emits events (create/save/status/delete), writes history, and broadcasts via sockets.
- Universal shorthand `$J` exists on server and client for quick access.

## System overview
- Global `JOE` (from `server/init.js`) exposes:
  - `JOE.Utils`, `JOE.Schemas`, `JOE.Storage`, `JOE.Cache`, `JOE.Apps`, `JOE.Server` (Express), `JOE.Sites`, `JOE.io` (Socket), `JOE.Mongo`/`JOE.MySQL`, `JOE.auth`.
  - Hot‑reload module pattern: modules may export `init()` and are watched/reloaded.
- Instance directories:
  - `_schemas`: schema definitions loaded by `JOE.Schemas`
  - `_apps`: app definitions (collections, dashboards, UX)
  - `_plugins`: backend features (auth, forms, reports, calendar, etc.)
  - `_templates`: templates/layouts; `_datasets`: seed/aux data; `_files`: uploads; `_www`: static (includes MCP pages)
- Shorthand `$J`: `$J.get(_id[, schema])`, `$J.search(query)`, `$J.schema(name)`.

## Apps (the use‑case layer)
Apps are the layer between core JOE and an instance’s additional schemas. They curate the right data surfaces and UI for a use case by:

- Selecting relevant collections (schemas) for the app’s domain
- Defining dashboards, cards, reports, and default filters/subsets
- Wiring applicable plugins and lightweight workflows

Think of apps as scoped workspaces: they present a coherent subset of “everything JOE can do” plus instance‑specific schemas and UI that match the task at hand. Agents can use apps conceptually to understand a domain’s surface area without needing every schema in the instance.

## Schemas and fields (high‑level)
- Schema keys: `title`, `fields` (visible), `_listID`, `_listTitle`, menus, filters, events, optional `tableView`.
- Field properties: `label`, `type`, `value/default`, `hidden`, `locked`, `condition`, `width`, `tooltip`, `comment`, plus type‑specific props.
- Common field types: `text`, `number`, `int`, `select`, `boolean`, `date`, `geo`, `image`, `objectlist`, `objectReference`, `content`, `rendering`, `code`, `button`.
- Page helpers: `section_start/_label/_end`, `sidebar_start/_label/_end`.
- Events: schema‑level on `create`, `save`, `status`, `delete`.
- Statuses/tags: first‑class objects used across workflows and labeling.

## Server APIs and MCP (model context)

### REST highlights
- Exact search via `/API/search` (default). Fuzzy mode available with `mode=fuzzy` and params: `q`, `fields`, `threshold`, `limit`, `offset`, `highlight`. Default weighted fields: name (0.6), info (0.3), description (0.1).

### MCP bridge (for agents)
- Manifest: `GET /.well-known/mcp/manifest.json` (public)
- JSON‑RPC: `POST /mcp` (auth‑protected when users exist)
- Core tools (master doc surface):
  - `listSchemas` — names of available schemas
  - `getSchema` — schema definition by name
  - `search` — exact search (defaults to cache; set `source:"storage"` for DB)
  - `fuzzySearch` — typo‑tolerant search with optional weighted fields
  - `getObject` — fetch single object by `_id` (optional `schema`, `flatten`, `depth`)
  - `saveObject` — create/update an object (fires events/history)
  - `hydrate` — returns core fields, schema names, statuses, tags for context boot

Note: Additional MCP tools exist; see deeper‑dive docs for the complete surface.

## Agent integration and save policy
- Start of session: call `hydrate {}` to prime context, then `listSchemas`/`getSchema` as needed.
- Query strategy: prefer cache for speed; use storage for authoritative reads or post‑save validation; for large result sets, `search { countOnly: true }` first.
- Save policy (canonical): always save the full validated object (not a partial diff). Recommended flow: get from storage → merge changes → set `joeUpdated` → validate against schema → reflect final object to user → `saveObject`.
- Rationale: clean, consistent history and versioning; predictable integrity on write.

## Instance setup (concise)
- Config: see `server/webconfig.js` for defaults (e.g., `PORT`, `hostname`, `joedb`, `joepath` `/JsonObjectEditor/`).
- Structure: keep instance‑specific `_schemas`, `_apps`, `_plugins`, `_templates`, `_datasets`, `_files` at the project root alongside JOE; JOE auto‑watches and loads these.
- Static & tools: JOE serves `_www` both under `joepath` and as a root fallback, exposing MCP pages: `/mcp-test.html`, `/mcp-export.html`, `/mcp-prompt.html`.
- Identity for agents: the MCP manifest surfaces instance info (`name`, `version`, `hostname`). Agents should always hydrate and then treat apps as domain‑specific lenses over the instance’s schemas and data.

## Operations and behavior
- Storage backends per schema: `mongo`, `mysql`, `file`, `api` (fallbacks included). Missing IDs are assigned on save (prefers `cuid()`, fallback to GUID).
- Cache: `JOE.Cache.update` builds `JOE.Data` (by collection), a flattened list, and a `lookup` for fast ID resolution; `$J.search()` hits cache by default.
- History/events/sockets: saves create `_history` records, fire schema events, and emit `item_updated` to clients.
- Auth: Basic Auth recommended currently; OAuth2 plugin exists but is being hardened.
- Security: MCP tools sanitize sensitive fields; `/privacy` and `/terms` are public and use `Setting: PRIVACY_CONTACT` for contact email.

## Best practices
- Keep business rules and relationships in schemas; use dynamic `propAsFuncOrValue` where supported.
- Use delegated actions for interactive UI (e.g., `joe-button` with `action="..."`).
- Prefer cache for reads; use storage for accuracy‑critical operations.
- After updates, refresh overlays (`_joe.show()`), or fetch fresh via `_joe.Object.getFromServer(id, { goto: true })` when needed.
- Never suppress errors; surface and handle validation explicitly.
- Agents: hydrate first, scope results (limit 10–25), and always confirm before `saveObject`.

## Known limitations and roadmap
- OAuth2/SSO via plugin (currently buggy; Basic Auth recommended short‑term)
- Pagination and streaming for extremely large datasets
- Not offline‑first
- Optional audit trail is basic (history collection)
- Packaging: future Docker bundles for admin + server

## Quick references
- MCP manifest: `GET /.well-known/mcp/manifest.json`
- MCP call template (JSON‑RPC):
```json
{ "jsonrpc": "2.0", "id": "1", "method": "<tool>", "params": { } }
```
- Fuzzy search (JSON‑RPC example):
```json
{ "jsonrpc":"2.0", "id":"6", "method":"fuzzySearch", "params": { "schema":"<schema>", "q":"term", "threshold":0.5, "limit":25 } }
```