UNPKG

comfyui-mcp

Version:

Local-first, agent-native control plane for ComfyUI — MCP server + autonomous sidebar agent that drives your live graph in natural language on ANY LLM: Claude/ChatGPT/Gemini on your subscription (no API key), free local models via Ollama (fully offline),

175 lines (137 loc) • 6.99 kB
--- name: ltx-director description: Drive the LTX Director (Timeline) node — its Add Image/Text/Audio buttons are DOM-only and cannot be clicked by an agent; edit the hidden timeline_data JSON widget instead. Load when a workflow contains LTXDirector / LTXDirectorGuide / PromptRelayEncodeTimeline, or when asked to add, move, retime, or remove timeline segments (image / text / audio / motion). --- # LTX Director (Timeline) `LTXDirector` (pack: **WhatDreamsCost-ComfyUI**, category `WhatDreamsCost`) is a video *timeline editor* node. Its on-canvas UI has **Add Image**, **Add Text** and **Add Audio** buttons. ## You cannot click those buttons. Don't try. They are not node inputs. The pack builds them as raw DOM elements with JS handlers: ```js addTextBtn.addEventListener("click", () => this.addTextSegmentFreeSpace()); ``` Panel/MCP tools drive the LiteGraph **node model** (widgets + inputs). They cannot invoke arbitrary DOM handlers, so "click Add Text" is impossible — this is a hard limitation, not flakiness. Say so plainly rather than retrying. ## The real control surface: `timeline_data` Everything those buttons do is serialized into ONE string widget, `timeline_data`. The pack's own code treats it as *"the absolute source of truth"*. Set it and the editor renders it. It is listed in the pack's `HIDDEN_WIDGET_NAMES`, so it does **not** appear in the node's visible widget list — but it is an ordinary input and is settable: ``` modify_workflow → { op: "set_input", node_id: "42", input_name: "timeline_data", value: "<json string>" } ``` The value is a JSON **string**, not an object. Empty state is `"{}"`. Sibling hidden widgets: `local_prompts`, `segment_lengths`, `guide_strength`, `audio_data`, `use_custom_audio`, `inpaint_audio`, `use_custom_motion`, `override_audio`. ## Schema (verified against a real production workflow) ```jsonc { // TRACK GATES — a track's segments are IGNORED unless its gate is true. "mainTrackEnabled": true, "audioTrackEnabled": false, "motionTrackEnabled": false, // must mirror the node's start_frame / duration_frames widgets (see Edges) "normalStartFrame": 0, "normalDurationFrames": 4393, "global_prompt": "", "overrideAudio": false, "inpaint_audio": false, // retake mode "retakeMode": false, "retakeStart": 24, "retakeLength": 48, "retakePrompt": "", "retakeStrength": 1, "retakeVideo": null, "retake_global_prompt": "", // display-only "propHeight": 90, "globalPropHeight": 60, "showFilenames": true, "segments": [], // main track: image AND text segments "motionSegments": [], "audioSegments": [] } ``` ### Text segment (`segments`) ```json { "id": "seg1", "start": 0, "length": 48, "prompt": "wide shot, neon city", "type": "text" } ``` ### Image segment (`segments`) ```json { "id": "1784657553220x4fxu", "start": 0, "length": 241.68, "prompt": "", "type": "image", "imageFile": "whatdreamscost/Nintendo_ZeldaUrbosaHotspringKF_00002_.png", "imageB64": "/api/view?filename=Nintendo_ZeldaUrbosaHotspringKF_00002_.png&type=input&subfolder=whatdreamscost" } ``` **`imageB64` is a misnomer — it holds a `/api/view` URL, not base64.** So an image segment just points at a file already in ComfyUI's **input** dir. Full agent-drivable recipe: 1. `upload_image` → puts the file in the input dir (note its `subfolder`/name) 2. `imageFile` = `"<subfolder>/<name>.png"` 3. `imageB64` = `"/api/view?filename=<name>.png&type=input&subfolder=<subfolder>"` ### Audio segment (`audioSegments`) ```json { "id": "17846562234967gjvk", "type": "audio", "start": 0, "length": 4393, "trimStart": 0, "audioDurationFrames": 4393, "audioFile": "whatdreamscost/Intergalactic Hip Hop.mp3", "fileName": "Intergalactic Hip Hop.mp3", "waveformPeaks": [0.047, 0.274, 0.537, "…~200 floats…"] } ``` `waveformPeaks` is the rendered waveform. It is cosmetic — omit or pass `[]` if you're writing a segment programmatically; the audio still plays. Don't fabricate plausible-looking peaks and imply they were measured. ## Edges (get these wrong and it silently does nothing) 1. **Track gates.** Pushing into `audioSegments` does nothing while `audioTrackEnabled` is `false`. Set the gate in the same edit. (The reference workflow ships an audio segment with the track OFF — easy to misread as broken.) 2. **Frame bookkeeping must agree.** `normalDurationFrames` == `duration_frames` widget == `segment_lengths` widget (observed: all `4393`), and `normalStartFrame` == `start_frame`. Change the timeline length and you must update all of them. 3. **Frames, not seconds.** `start` / `length` are pixel-space frames and may be fractional (`241.68`). `duration_seconds` / `frame_rate` / `time_units` are display concerns. 4. **`global_prompt` exists twice** — inside `timeline_data` and as a `forceInput` socket on the node. In the reference workflow the socket is wired from a `PrimitiveStringMultiline`; prefer the wired source and keep them consistent. 5. **`guide_data` must go somewhere.** `LTXDirector.guide_data` → `LTXDirectorGuide`. That partner node is what injects the keyframes. 6. **Unsupported by the pack.** The widget tooltip says *"auto-managed; do not edit by hand"*. Well-formed blobs load fine; malformed JSON will break the editor. Round-trip and re-read after writing, and tell the user this is unofficial. ## Reference wiring (from a working LTX 2.3 Director graph) ``` Power Lora Loader (rgthree) ─ model ─┐ DualCLIPLoader ──────────── clip ────┤ VAELoaderKJ ───────────── audio_vae ─┤ LTXDirector #42 PrimitiveStringMultiline ─ global_prompt ─┘ │ model → LTX2_NAG │ positive → ConditioningZeroOut, LTXVConditioning │ audio_latent → LTXVConcatAVLatent │ guide_data → LTXDirectorGuide ← required partner │ frame_rate → AudioToFrameCount, VHS_VideoCombine └ combined_audio→ AudioToFrameCount, PreviewAudio, … ``` ## Procedure 1. Find the node: `query_workflow` with `types: ["LTXDirector"]`, `fields: "detail"` (or `panel_query_graph` on the live canvas). 2. Read the current `timeline_data`, `JSON.parse` it — **never** hand-splice the string. 3. Mutate the parsed object (append a segment, flip a track gate, retime). 4. Keep `normalStartFrame` / `normalDurationFrames` in sync with the widgets (edge 2). 5. Write it back as a string via `set_input`. 6. Re-read to confirm it round-trips, then `validate_workflow`. ## Related `PromptRelayEncodeTimeline` (pack: ComfyUI-PromptRelay) carries the **same** `timeline_data` hidden-widget pattern — this skill's approach applies there too. More broadly: any node whose controls are DOM widgets over hidden state is invisible to clicking and must be driven through its state widget. LTX Director is just the clearest example.