UNPKG

@sogni-ai/sogni-creative-agent-skill

Version:

Sogni Creative Agent Skill: agent skill and CLI for Sogni AI image, 3D, speech, video, and music generation.

401 lines (348 loc) • 26.5 kB
# Hosted Sogni Intelligence API Modes Read this when a request should be planned, multi-step, resumable, or server-orchestrated: `--api-chat`, `--durable-chat`, `--api-workflow`, workflow templates, replay records, and the hosted tool surfaces. All hosted modes require `SOGNI_API_KEY`. ## When to prefer the hosted path The thing calling this API is usually a frontier LLM that is **more capable than Sogni's hosted planning model**. So the default split is: *you* do the planning and tool selection, and the hosted endpoints do what only the server can — run on the GPU network, persist assets/manifests, orchestrate durable multi-step runs with replay, and apply Structured Contracts v1 (gating policies, repair recipes, prompt contracts). Routing a request through `--api-chat` so a weaker model re-plans it is usually a downgrade; reach for the hosted *planner* deliberately, not by default. - **You already know the single tool + args** → direct-to-SDK flags. Lowest latency/cost, no LLM round-trip. - **Multi-step, durable, resumable** → `--api-workflow` with an explicit `--workflow-input` step graph that *you* author (`steps[]` with `toolName`, `arguments`, and `dependsOn` bindings). The server executes and repairs it deterministically with replay/resumability and **no hosted-LLM re-planning**. This is the best fit when a frontier client drives the work. - **You want the hosted model to own a long loop** → `--api-chat` / `--durable-chat`. Worth it when offloading a long async tool loop server-side saves client round-trips, when structured-contract repair should govern, or when several local files must be uploaded for one turn (only supported here). ```bash # You author the exact durable plan; the server executes it (no hosted re-planning) sogni-agent --api-workflow --workflow-input @plan.json # Storyboard → GPT Image 2 sheet → Seedance, all server-side (preset plan) sogni-agent --api-workflow storyboard-video --storyboard-frames 6 -Q hq \ "Create a 9:16 bakery launch video with a neon street-window reveal" # Deliberately hand planning to the hosted model (long async job / multi local-file upload) sogni-agent --api-chat "Turn the attached product photo into a launch poster" --ref product.jpg # Durable hosted chat run (persisted event log + SSE stream) SOGNI_SKILL_USE_SDK_TRANSPORT=1 sogni-agent --durable-chat \ "Create a four-shot launch campaign, generate the key art, and animate the hero clip" ``` The direct-to-SDK flags remain the right call for explicit one-shot generation when you already know the exact model, dimensions, and prompt — use them whenever latency or cost rules out an LLM round-trip. ## Preserve the requested work - A model/capability question is an answer, not permission to render. A script or storyboard-review request ends at the draft; a storyboard-image request ends at the still. The `storyboard-video` preset creates a full video, so use it only when that is the requested deliverable. - When the user requests images, animation, and stitching together, finish those stages in order unless they ask to review or select an intermediate result. Treat returned artifacts as evidence of completion; do not resubmit a successful stage because the assistant's summary omitted its tool call. - Preserve exact image prompts, visible lettering, chosen models, source selection, aspect ratio, and the latest requested duration. New tasks do not inherit settings from unrelated previous renders. A pure size increase uses `upscale_image` for images and `upscale_video` for videos; editing requires an actual source reference. - Separate image variations need `numberOfVariations` and a complete Dynamic Prompt branch per output. Repeat shared text, branding, dialogue, and loop motion in every affected branch. A single storyboard sheet instead needs all requested scenes in order, with complete visual/action, camera, transition, dialogue, and sound instructions. Preserve supplied titles and layout. In “six 9:16 frames,” six is the count and 9:16 is the frame shape. - Preserve reference order and bind the correct uploaded or generated asset. An original image can be a prior generated result; it is not necessarily an upload. For a loop ending on the original still, bind that original as the final reference. Distinguish loose audio guidance from synchronized audio. - Check the chosen model's duration and reference limits. A long video may fit one native clip or require segments and stitching; do not silently shorten it or replace an explicit model. For segmented storyboards, assign each clip its own time window and scene beats while preserving locked music and timing. - Let the hosted runtime repair malformed calls before dispatch. A correction round does not authorize a second successful render. If a result says to wait for the user or all outputs failed, surface that result and stop. - Optional content-filter changes require the user's explicit choice. In hosted chat, a supported settings tool must report success before retrying; declined or unavailable confirmation leaves the setting unchanged. In direct CLI mode, do not add `--no-filter` on your own. The setting cannot change GPT Image's mandatory policy or resolve an unknown rejection. Attribute a rejection to a provider only when the returned error supports that attribution. When explicitly requested, `--no-filter` is forwarded on new durable workflow starts as well as hosted chat. Resuming a workflow retains its original preference; passing the flag on a resume does not change it. These are also instructions for the calling agent when it authors direct CLI commands or workflow JSON. The public runtime exposes local helpers and shared contracts; the full hosted planner, call repair, and confirmation behavior run on the server. Updating this skill does not deploy those server components. ## --api-chat (`POST /v1/chat/completions`) Text-first natural-language workflows through Sogni API's OpenAI-compatible loop. The public REST body uses snake_case controls such as `tool_choice`, `response_format`, `task_profile`, `token_type`, `app_source`, `media_references`, `chat_template_kwargs`, `sogni_tools`, and `sogni_tool_execution`. The endpoint normalizes OpenAI `developer` messages to `system`; when a developer message is present and no explicit `task_profile` is supplied, the server treats the task as `coding`. The CLI sanitizes prompt-injection markers before forwarding messages and sends API-key auth so hosted Sogni tools can execute server-side. Tune with `--api-tools creative-agent|creative-tools|none`, `--no-api-tool-execution`, `--llm-model <id>`, `--system <text>`, `--task-profile general|coding|reasoning`, `--max-tokens <n>`, and `--thinking` / `--no-thinking` (forwarded as `chat_template_kwargs.enable_thinking`; hosted Qwen may normalize thinking server-side, so do not rely on `--no-thinking` as a hard suppression switch). The default output budget is 4,096 tokens for hosted chat (including durable chat) and storyboard planning. Keep enough room for every requested scene and complete tool arguments; `--max-tokens` overrides that budget. HTTP chat and storyboard-planning requests ask the API to keep long calls connected: a call still working after 90 seconds answers `200` and sends a space every 15 seconds until it finishes, for up to 35 minutes, and prints nothing meanwhile. They allow two minutes for the response to start by default; `SOGNI_HTTP_TIMEOUT_MS` overrides that wait, and it can no longer end a call once the response has started. Stopping the CLI cancels the call and refunds its renders, except a vendor-model render that has started, which keeps running and finishes into the account's history. Other HTTP requests keep their existing timeouts. ### Hosted tool surfaces (`sogni_tools`) - `creative-tools` — the public API default when `sogni_tools` is omitted or true. Generation/editing tools (`generate_image`, `generate_video`, `generate_music`, `generate_speech`, `edit_image`, `upscale_image`, `upscale_video`, `apply_style`, `restore_photo`, `refine_result`, `animate_photo`, `change_angle`, `video_to_video`, `stitch_video`, `orbit_video`, `dance_montage`, `sound_to_video`, `extend_video`, `replace_video_segment`, `overlay_video`, `add_subtitles`), media-analysis tools (`analyze_image`, `analyze_video`, `extract_metadata`), and lightweight composition tools (`enhance_prompt`, `compose_lyrics`, `compose_instrumental`, `compose_script`). - `creative-agent` — this CLI's default for `--api-chat`. Includes `creative-tools` plus session-control tools (`ask_clarifying_question`, `finalize_response`), asset-manifest tools (`create_asset_manifest`, `inspect_asset`, `label_asset`, `map_assets_for_model`, `validate_asset_references`), and durable planning tools (`compose_workflow`, `compose_workflow_template`). Use this surface when the model should design one-shot workflow plans, draft savable workflow templates, or maintain stable asset references across a multi-step turn. - `none` — disables Sogni tool injection, leaving only caller-supplied OpenAI tools on raw API/SDK requests. In the CLI, combine with `--no-api-tool-execution` for text-only planning. For a pure resolution increase, call `upscale_image` with one source image and either `scale: 2|3|4` or `targetLongestEdge`. Use `7680` for 8K or `15360` for 16K. The promptless NVIDIA RTX VSR path preserves the source aspect ratio, aligns both output edges to 8px, and returns JPG when the target exceeds 8K. For a pure resolution increase of a video, call `upscale_video` with `sourceVideoIndex` (generated results are 0-based; -1 is the first upload) and an optional `targetResolution` of 1080 or 1440. The promptless FlashVSR path keeps every frame, the frame rate, the aspect ratio, and the audio; the server reads the source metadata itself, so the model never supplies frames or fps. It cannot produce 4K, and it is not a substitute for `video_to_video` when the user asks a generative model such as Seedance to re-render the clip. ## --durable-chat (`POST /v1/chat/runs`) Long-running, LLM-in-the-loop turns persisted as chat-run records instead of a single completion request. Chat runs keep an event log, stream via `/v1/chat/runs/:id/events/stream`, support cancellation, and can pause for persisted cost approval (`/v1/chat/runs/:id/confirm-cost`) in first-party clients. Requires `SOGNI_SKILL_USE_SDK_TRANSPORT=1`. The CLI streams assistant deltas plus de-duplicated per-job progress / ETA / result lines from hosted run events. The SDK exposes `sogni.chat.runs.{create, get, cancel, streamEvents}`. Durable chat runs the full creative agent with server-side tool execution. Use `--api-chat` for `--api-tools creative-tools|none` or `--no-api-tool-execution`; durable mode rejects those options before starting a run rather than silently ignoring them. Completed assistant snapshots reconcile with streamed text so each message appears once, including multi-round turns. Review returned storyboard titles, visible text, and timings against the request before rendering. Hosted draft generation can paraphrase supplied scene titles; correct the draft first when exact wording matters. ## --api-workflow (`POST /v1/creative-agent/workflows`) Durable, async workflow records with event streaming and cancellation. The API accepts either an inline durable plan (`input.steps`) or a saved workflow template invocation (`workflow_id` plus `inputs`) and rejects requests that provide both. The CLI's generated-keyframe and `storyboard-video` presets submit inline `input.steps`; `--workflow-input <json|@path>` supplies the `input` object directly (use `@path` to load from a file). - Saved template CRUD lives at `/v1/creative-agent/workflows/templates`; run a saved template later with `workflow_id + inputs`. Draft savable templates with `compose_workflow_template` through `--api-chat` — the caller persists the returned `template_draft`. - Exact multi-step plans should use explicit step dependencies, including `replace_video_segment` steps with bounded `replacementStartSeconds` / `replacementEndSeconds` when interleaving existing video slices. Workflow JSON can bind request media into step arguments with `sourceStepId: "$input_media"`. - LoRA steps: `generate_image`, `edit_image`, `generate_video`, and `animate_photo` take an ordered `loras` array plus a positionally aligned `loraStrengths` (order matters, adapters do not commute). Step arguments are schema-validated before dispatch, so 1-8 entries with matching lengths or the step fails with a `400` — omit `loraStrengths` entirely to apply every adapter at `1.0`. LoRAs are also model-gated, and there the mismatch is silent: ids the chosen model cannot load are dropped without an error, so set the model in the same step — `model: krea-2-turbo` or `dark-beast-krea2` for `generate_image`, `model: krea-identity-edit` or `dark-beast-krea2-identity-edit` for `edit_image`, and a `minimax-h3-*` `videoModel` for `generate_video` / `animate_photo`. Published ids and ranges come from `GET /v1/loras/comfy?modelId=<canonical-model-id>`. - MiniMax H3 keyframes in a step: add `keyframes: [{ "imageIndex": -2, "atSeconds": 3.5 }]` to an `animate_photo`, `sound_to_video`, or Ref2VA `generate_video` step's `arguments`, with the images attached through `-c/--context` (negative indices count image uploads, -1 first). See [MiniMax H3 keyframes in hosted tools](#minimax-h3-keyframes-in-hosted-tools). - `--api-workflow storyboard-video` generates a storyline, creates one GPT Image 2 storyboard sheet, then feeds that artifact into Seedance as the video reference. `-Q fast|hq|pro` maps to GPT Image 2 low|medium|high quality for the storyboard sheet. - Cost controls: `--workflow-max-cost <n>` rejects workflow starts above a capacity-unit ceiling; `--confirm-cost` / `--no-confirm-cost` forward explicit billing confirmation. Use `--workflow-idempotency-key <key>` when retrying a start request. - Manage runs with `--watch-workflow`, `--list-workflows`, `--get-workflow <id>`, `--workflow-events <id>`, `--stream-workflow <id>`, `--cancel-workflow <id>`, `--resume-workflow <id>`. In `--json` mode, SSE progress frames stream to stderr so stdout stays a single JSON object. ```bash # Durable workflow with a media reference and a cost ceiling sogni-agent --api-workflow --ref https://cdn.example.com/sketch.png \ --workflow-max-cost 25 --confirm-cost \ --video-prompt "The camera slowly pushes in as the sketch comes alive" \ "Animate the referenced sketch" # Exact durable workflow input sogni-agent --api-workflow --workflow-input @workflow.json \ --workflow-idempotency-key product-teaser-v1 ``` ## Media references in hosted modes Hosted API requests forward media references from `-c`, `--ref`, `--ref-end`, `--keyframe`, `--ref-audio`, `--reference-audio-identity`, and `--ref-video` as `media_references` metadata. Image references keep that order, so negative image indices count `-c` images first, then `--ref`, `--ref-end`, and the `--keyframe` images in time order (see [MiniMax H3 keyframes in hosted tools](#minimax-h3-keyframes-in-hosted-tools)). `--ref-audio` and `--ref-video` are repeatable in api-chat / durable-chat mode — each entry uploads independently and is exposed to the hosted LLM as `@Audio1` / `@Audio2` / `@Video1` etc. API chat also attaches image refs as vision inputs. Local file references are uploaded to Sogni media storage first, then forwarded as retrievable URLs so durable executors do not depend on `data:` URI support. **Use direct CLI mode for private media that must not leave the local machine.** ## Seedance reference modes (mutually exclusive) When `--video -m seedance2`, `-m seedance2-mini`, `-m seedance2-fast`, or `-m seedance2-5` is selected, pick one mode per video request: - **Dedicated frame mode — `--ref` and/or `--ref-end`.** First-class first-frame / last-frame anchoring; the Seedance worker pins them as parameter-mode firstFrame / lastFrame. Max 2 images. - **Loose reference mode — `-c/--context` plus optional `--ref-audio` and `--ref-video` extras.** Anchor frame intent in the prompt with `@Image1` / `@Video1` / `@Audio1` etc. (e.g. *"Use @Image1 as the opening shot reference"*). Each `-c/--context` image may be a **local file or an HTTPS URL** (PNG, JPEG, WebP, or GIF) — local files are uploaded to Sogni media storage automatically, so you do **not** need `--api-chat` / `--durable-chat` just to attach a local loose-reference image. The 2.0 family supports up to 9 image refs, 3 video refs, 3 audio refs, and 12 total reference assets per request; `seedance2-5` raises the caps to 30 image / 10 video / 10 audio / 50 total (canonical per-model caps come from `SEEDANCE_REFERENCE_LIMITS` / `validateSeedanceReferenceCounts()` in `@sogni-ai/sogni-intelligence-client/tools`). Combining native `--ref` / `--ref-end` frame anchors with any loose image, video, or audio reference on Seedance is rejected client-side with an error pointing at the correct mode. The typed `--workflow ia2v` path is the explicit exception: there, `--ref` is sent as a loose `@Image` reference beside `--ref-audio`, not as `first_frame`; `--ref-end` remains invalid. In CLI direct-gen mode, local `-c/--context` images and the primary `--ref-audio` / `--ref-video` are uploaded to Sogni media storage automatically and forwarded as HTTPS URLs; only *additional* `--ref-audio` / `--ref-video` entries beyond the first must already be HTTPS URLs (use `--api-chat` / `--durable-chat` when you need to attach several local audio or video files in one request). Seedance accepts public HTTPS image, video, and audio references that pass the CLI URL safety checks; localhost and private-network URLs are rejected before forwarding. Seedance 2.0 audio references must be paired with an image or video reference; Seedance 2.5 `reference` accepts audio by itself. For direct Seedance 2.5 loose-reference operations, use `--seedance-task-type reference|edit|extend`. Edit and extend require `--ref-video`, reject `--ref`/`--ref-end` frame anchors, and inherit the source video's aspect ratio. Edit requires `--duration` equal to the source duration; extend uses the selected 4-30 second continuation duration. On hosted semantic tool surfaces, express the same intent with `generate_video`, `video_to_video`, or `extend_video`; do not pass the direct SDK's `seedanceTaskType` routing field as a hosted tool argument. ## Models, replays, and contract debugging - `--list-api-models` / `--get-api-model <id>` inspect `/v1/models`. - `--list-replays [n]`, `--get-replay <id>`, `--ingest-replay <json|@path>` manage `/v1/replay/records` RunRecords for replay/debug viewers. List/get output is run through `redactRunRecord` before printing, so signed URLs, bearer tokens, JWTs, and PEM blocks cannot leak via the CLI. `--skip-redact` / `--no-redact` bypass redaction (debug-only). - `--turn-classify`, `--compile-tools`, `--dispatch-tool <name>` (+ `--tool-args <json>`) print the public-skill Structured Contracts v1 verdicts (turn policy, compiled tool surface, dispatch verdict) the default contract runtime would produce. - `--storyboard-plan` builds a storyboard project locally (`buildStoryboardProject` + `compileForModel`) and prints the plan as JSON without network calls. It expects scene-structured prompt input (`SCENE NN - Title` / `VISUAL:` / `ACTION:` / `CAMERA:` / `AUDIO/SFX:` blocks) — for casual prompts use `--api-workflow storyboard-video`, which runs an LLM storyline expansion first. Pair with `--storyboard-plan-frames`, `--storyboard-plan-model` (seedance, seedance2, gpt-image-2, ltx23, wan), `--storyboard-plan-stage` (storyboard_image, scene_clip). ## Endpoint safety Override the API origin with `--api-base-url`, `SOGNI_API_BASE_URL`, or `SOGNI_REST_ENDPOINT`. Hosted API credentials are only sent to `https://api.sogni.ai` by default. Add trusted custom hosts with `SOGNI_API_ALLOWED_HOSTS`; loopback or non-HTTPS local testing requires `SOGNI_ALLOW_UNSAFE_API_BASE_URL=1`. With `SOGNI_SKILL_USE_SDK_TRANSPORT=1`, hosted workflow + chat operations route through the SDK transport; the skill's `sogni-hosted-client.mjs` factory still validates `restEndpoint` / `socketEndpoint` against the SSRF guard before constructing the SDK client. ## Original-image utilities and speech Submit these media tools as durable workflow steps (`toolName` and `arguments`) to `POST /v1/creative-agent/workflows`, or use hosted chat. The synchronous `/tools/execute` endpoint supports text composition only: | Tool | Inputs and controls | Output | | --- | --- | --- | | `remove_background` | `sourceImageIndex` or `source_image_url`, `applyMask` (default true) | Transparent PNG; false returns mask | | `image_to_3d` | Front `sourceImageIndex`/`source_image_url`; optional `leftViewImageIndex`, `backViewImageIndex`, `rightViewImageIndex` or named `*_view_image_url` URLs; `meshTargetFaces`, `textureSize`, `normalMapSize`, `ambientOcclusionSize`, `shapeResolution` | Binary GLB with media type `model` | | `segment_image` | Original source; `text`, `points`, or `boxes`; `threshold`, `multimask`, `maxInstances`, `applyMask` | PNG mask or original-pixel cutout | | `generate_speech` | Literal `prompt`; `model=voice/clone/design`; `voice`, `voiceDescription`, `voiceSourceIndex`, `voiceTranscript`, `language`, `creativity`, `outputFormat`, `seed` | Spoken audio | Negative indices address uploads (-1 first); non-negative indices address generated results. 3D views are from the subject's perspective: left means the subject's own left side faces camera (subject faces screen-left), right means screen-right. Supply original images of the same subject at consistent height and framing; any subset of the three orbit views enables multi-view reconstruction. Keep the GLB in the model artifact lane, never feed it into an image tool. For MiniMax reference-to-video, default to **Balanced R2V** (`minimax-h3-r2v-balanced`), the tier that passes the quality bar for mature content and holds Mystic X at full strength (as of 2026-10). Use `minimax-h3-r2v-balanced-2stage` for two-stage, 1080p or 2K output. Keep Standard, LightX2V Turbo, or FastH3 Omni R2V only when explicitly requested. For two-stage Turbo H3, use explicit `minimax-h3-fasth3-*-turbo-2stage` selectors on the relevant video tool with `targetResolution`. For 1080p or 2K reference-to-video, prefer `minimax-h3-r2v-balanced-2stage` (Balanced R2V Two-Stage) on `generate_video` with the same references as `minimax-h3-r2v` and `targetResolution` naming the delivered size; `minimax-h3-r2v-balanced` is Balanced Reference at 768p, and `minimax-h3-fasth3-r2v-turbo` (FastH3 Reference) stays an explicit choice. Audio-guided IA2V/FLFA2V/A2V selectors belong on `sound_to_video` and require the uploaded audio plus their named endpoint images. They preserve the supplied audio and reject LoRAs. Other H3 modes, FastH3 Reference included, discover personal LoRAs through the [authenticated catalog](personal-loras.md). ## MiniMax H3 keyframes in hosted tools `animate_photo`, `sound_to_video`, and `generate_video` take an optional `keyframes` argument that pins up to 8 images at exact moments inside one MiniMax H3 clip. Its schema description, identical on all three tools: > MiniMax H3 only. Pin up to 8 images at exact moments inside the video, in addition to the first/last frame. Each item is {imageIndex, atSeconds}: imageIndex uses the endImageIndex convention (negative = uploads, 0+ = generated results); atSeconds is when the video should land on that image. Keyframes must fall strictly inside the clip (not on the first or last frame) and at distinct times. Describe what each keyframe shows in the prompt at its time; a keyframe with a new angle, place or light starts a new shot. Two keyframes are included in the price; each additional keyframe adds a little. | Tool | `videoModel` selectors that take `keyframes` | | --- | --- | | `animate_photo` | Every MiniMax H3 image-to-video and first/last-frame selector, with any `frameRole` | | `sound_to_video` | Every MiniMax H3 audio selector (image + audio, first and last frame + audio, audio only; one- and two-stage) | | `generate_video` | Only the Reference to Video selectors (`minimax-h3-r2v*`); never text-to-video | - `imageIndex` follows `endImageIndex` on every tool: -1 is the first image upload, -2 the second, and 0 and up are generated results. On `sound_to_video` this differs from the 0-based upload numbering of `sourceImageIndex`. - Each `atSeconds` lands on frame `round(atSeconds × 24)`, strictly inside the job's real frame count (a Sound to Video clip without `duration` runs for the audio window). Every other model, a time outside the clip, two keyframes on one frame, or more than 8 is refused with a `PARAMETER_INVALID` error that says what to change; nothing is clamped or dropped. Set `duration` so every keyframe falls inside the clip. - Keyframes add to the first and last frame and are not references (on `generate_video` they do not go in `referenceImageIndices`), but the H3 prompt names each one `<Picture N>`, numbered in time order after the first/last-frame or reference pictures. - Price: two keyframes are included; each additional keyframe adds 0.75 s of output time on FastH3 (0.3 s on other tiers) at the job's rate. - Error 4100 means no online worker serving that model can pin keyframes right now. It is a temporary availability refusal: retry shortly, or render without keyframes. From the CLI, `--keyframe <image>@<seconds>` in `--api-chat` / `--durable-chat` uploads each image as an image media reference after `--ref` / `--ref-end` and tells the hosted agent the exact argument to use, for example `keyframes: [{"imageIndex": -2, "atSeconds": 3.5}]` after one `--ref`. In an `--api-workflow --workflow-input` plan, put `keyframes` in the step's `arguments` and attach the images with `-c/--context` (`--api-workflow` refuses `--keyframe`). See [models.md § MiniMax H3 keyframes](models.md#minimax-h3-keyframes) for supported models and prices, and [video-prompting.md § Intermediate keyframes](video-prompting.md#intermediate-keyframes) for how to describe keyframes in the prompt.