openclaw
Version:
Multi-channel AI gateway with extensible messaging integrations
214 lines (186 loc) • 12.6 kB
Markdown
---
summary: "The exported span catalog with its attributes, and the diagnostic event catalog behind the metrics and spans"
title: "Exported spans and diagnostic events"
sidebarTitle: "Spans and events"
read_when:
- You need the exact span names or attribute shapes to build dashboards or alerts
- You are subscribing a plugin to public diagnostic events
- You need session-correlated usage that the exported metrics intentionally omit
---
## Exported spans
- `openclaw.gateway.rpc.response`, `openclaw.gateway.rpc.handler`, `openclaw.gateway.rpc.dispatch`
- Completed phase observations with `openclaw.gateway.rpc.method`, `openclaw.gateway.rpc.phase`, and `openclaw.gateway.rpc.outcome`
- Handler spans include `openclaw.gateway.rpc.admission_ms`; dispatch spans include `openclaw.gateway.rpc.response`, the response state at dispatch settlement
- Preserve a supplied upstream request parent; they do not introduce a long-lived RPC parent span or change downstream trace propagation
- `openclaw.model.usage`
- `openclaw.channel`, `openclaw.provider`, `openclaw.model`, optional `openclaw.agent` (the agent that owns the run when known)
- Optional host-derived `openclaw.plugin` only for trusted plugin runtime completions
- `openclaw.tokens.*` (input/output/cache_read/cache_write/total)
- `gen_ai.system` by default, or `gen_ai.provider.name` when the latest GenAI semantic conventions are opted in
- `gen_ai.request.model`, `gen_ai.operation.name`, `gen_ai.usage.*`
Plugin attribution is span-only. It does not add a plugin dimension to shared
OpenTelemetry metrics or change Prometheus metric labels.
- `openclaw.run`
- `openclaw.outcome`, `openclaw.channel`, `openclaw.provider`, `openclaw.model`, `openclaw.errorCategory`, optional `openclaw.agent`
- `openclaw.model.call`
- `gen_ai.system` by default, or `gen_ai.provider.name` when the latest GenAI semantic conventions are opted in
- `gen_ai.request.model`, `gen_ai.operation.name`, `openclaw.provider`, `openclaw.model`, `openclaw.api`, `openclaw.transport`, `openclaw.model_call.observation_unit` (`request` or `turn`), optional `openclaw.agent`
- `openclaw.errorCategory`, `error.type`, and optional `openclaw.failureKind` on errors
- `openclaw.model_call.request_bytes`, `openclaw.model_call.response_bytes`, `openclaw.model_call.time_to_first_byte_ms`
- `openclaw.model_call.prompt.input_messages_count`, `openclaw.model_call.prompt.input_messages_chars`, `openclaw.model_call.prompt.system_prompt_chars`, `openclaw.model_call.prompt.tool_definitions_count`, `openclaw.model_call.prompt.tool_definitions_chars`, `openclaw.model_call.prompt.total_chars` (safe component sizes only, no prompt text)
- `openclaw.model_call.usage.*` and `gen_ai.usage.*` when the result carries usage for that request or aggregate turn
- Span event `openclaw.provider.request` with attribute `openclaw.upstreamRequestIdHash` (bounded, hash-based) when the upstream provider result exposes a request id; raw ids are never exported
- With `OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental`, request spans use the latest GenAI inference span name `{gen_ai.operation.name} {gen_ai.request.model}`. Turn spans use `invoke_agent` because OpenClaw does not claim a native agent name from the opaque CLI boundary. Both use `CLIENT` span kind instead of `openclaw.model.call`.
- `openclaw.harness.run`
- `openclaw.harness.id`, `openclaw.harness.plugin`, `openclaw.outcome`, `openclaw.provider`, `openclaw.model`, `openclaw.channel`, optional `openclaw.agent`
- On completion: `openclaw.harness.result_classification`, `openclaw.harness.yield_detected`, `openclaw.harness.items.started`, `openclaw.harness.items.completed`, `openclaw.harness.items.active`
- On error: `openclaw.harness.phase`, `openclaw.errorCategory`, optional `openclaw.harness.cleanup_failed`
- Span event `openclaw.agent.commentary` for completed preambles from supported harnesses, including the built-in runtime, Codex, and Claude CLI. Attributes include `openclaw.commentary.sequence`, `openclaw.commentary.text_length`, and `openclaw.commentary.content_truncated`. The existing `diagnostics.otel.captureContent` setting controls bounded, redacted output-message content.
- `openclaw.tool.execution`
- `gen_ai.tool.name`, `gen_ai.operation.name` (`execute_tool`), `openclaw.toolName`, `openclaw.tool.source`, optional `gen_ai.tool.call.id`, `openclaw.tool.owner`, `openclaw.tool.params.*`, optional `openclaw.agent`
- Optional `openclaw.errorCategory`/`openclaw.errorCode` on errors, `openclaw.deniedReason` and `openclaw.outcome=blocked` when denied by policy or sandbox
- `openclaw.exec`
- `openclaw.exec.target`, `openclaw.exec.mode`, `openclaw.outcome`, `openclaw.failureKind`, `openclaw.exec.command_length`, `openclaw.exec.exit_code`, `openclaw.exec.exit_signal`, `openclaw.exec.timed_out`
- `openclaw.webhook.processed`
- `openclaw.channel`, `openclaw.webhook`
- `openclaw.webhook.error`
- `openclaw.channel`, `openclaw.webhook`, `openclaw.error`
- `openclaw.message.processed`
- `openclaw.channel`, `openclaw.outcome`, `openclaw.reason`, optional `openclaw.agent` (the agent that initially ingested the prompt)
- `openclaw.message.delivery`
- `openclaw.channel`, `openclaw.delivery.kind`, `openclaw.outcome`, `openclaw.errorCategory`, `openclaw.delivery.result_count`
- `openclaw.session.stuck`
- `openclaw.state`, `openclaw.ageMs`, `openclaw.queueDepth`
- `openclaw.context.assembled`
- `openclaw.prompt.size`, `openclaw.history.size`, `openclaw.context.tokens`, `openclaw.errorCategory` (no prompt, history, response, or session-key content)
- `openclaw.tool.loop`
- `openclaw.toolName`, `openclaw.loop.level`, `openclaw.loop.action`, `openclaw.loop.detector`, `openclaw.loop.count`, optional `openclaw.loop.paired_tool`, optional `openclaw.agent` (no loop messages, params, or tool output)
- `openclaw.memory.pressure`
- `openclaw.memory.level`, `openclaw.memory.reason`, `openclaw.memory.rss_bytes`, `openclaw.memory.heap_used_bytes`, `openclaw.memory.heap_total_bytes`, `openclaw.memory.external_bytes`, `openclaw.memory.array_buffers_bytes`, optional `openclaw.memory.threshold_bytes`/`openclaw.memory.rss_growth_bytes`/`openclaw.memory.window_ms`
When content capture is explicitly enabled, model and tool spans can also
include bounded, redacted `openclaw.content.*` attributes for the specific
content classes you opted into.
## Diagnostic event catalog
The events below back the
[metrics](/gateway/opentelemetry/model-calls-and-metrics#exported-metrics) and
spans above. Public events are also
available for direct plugin subscription; trusted core events such as
`model.usage` are restricted to authorized internal consumers.
`run.progress` and `run.execution_phase` are direct-only lifecycle signals;
the diagnostics-otel plugin does not export them as standalone OTLP signals.
Event kinds and `run.execution_phase.phase` values are additive. TypeScript
consumers should keep default branches instead of assuming either union is
permanently exhaustive.
`agent.commentary` records completed preambles, not text deltas. It carries the
original agent-event sequence and timestamp and attaches to the active harness
span. Recent duplicate completions are suppressed per attempt. Like other
queued diagnostics, commentary may be dropped under queue pressure; the session
transcript remains the durable conversation record. At debug level, the
`diagnostic` logger also records completion metadata without commentary text.
**Model usage**
`model.usage` is a trusted, in-process diagnostic event, not a JSONL log
record. A representative event has this shape:
```json
{
"type": "model.usage",
"ts": 1735689600000,
"seq": 42,
"provider": "openai",
"model": "gpt-5.4",
"channel": "webchat",
"agentId": "main",
"sessionId": "session-123",
"sessionKey": "agent:main:main",
"usage": {
"input": 120,
"output": 40,
"cacheRead": 30,
"cacheWrite": 10,
"promptTokens": 160,
"total": 200
},
"lastCallUsage": {
"input": 120,
"output": 40,
"cacheRead": 30,
"cacheWrite": 10,
"total": 200
},
"context": { "limit": 128000, "used": 160 },
"costUsd": 0.0012,
"durationMs": 850,
"trace": {
"traceId": "4bf92f3577b34da6a3ce929d0e0e4736",
"spanId": "00f067aa0ba902b7",
"traceFlags": "01"
}
}
```
- `ts` is a Unix timestamp in milliseconds; `seq` is process-local.
- `usage` holds turn-level token counts. `promptTokens` includes `input`,
`cacheRead`, and `cacheWrite`; `lastCallUsage`, when available, describes the
final model call.
- `context.used` is the current prompt/context snapshot and can be lower than
`usage.total` when cached input or tool-loop calls are involved.
- Provider/model/session identifiers, token buckets, `lastCallUsage`,
`context`, `costUsd`, `durationMs`, and `trace` fields are optional.
`costUsd` is an estimate and can be absent when model pricing is unavailable;
it is not provider-reported billing. Trace context can also include
`parentSpanId`.
The Gateway's `/tmp/openclaw/openclaw-YYYY-MM-DD.log` JSONL file and
`diagnostics.otel.logsExporter: "stdout"` contain ordinary log records, not raw
`model.usage` events. Public diagnostic subscriptions and
`diagnostics.stability` do not expose trusted core usage events. The
diagnostics-otel plugin converts them to metrics such as `openclaw.tokens` and
`openclaw.cost.usd` and to `openclaw.model.usage` spans; those usage metrics
and spans intentionally omit session identifiers.
For an external integration that needs session-correlated usage, query the
authenticated Gateway instead:
```bash
openclaw gateway call sessions.usage --params '{"range":"30d","agentScope":"all"}' --json
openclaw gateway usage-cost --days 30 --all-agents --json
```
Both commands require `operator.read`. `sessions.usage` can include per-session
`sessionId`, provider/model details, and token/cost summaries; per-session usage
can be temporarily `null` while its cache refreshes. `usage-cost` provides
aggregate estimates. Omit `agentScope` or `--all-agents` to scope the report
to the default agent. For continuously updated clients,
[subscribe to session changes instead of polling usage reports](/gateway/clients#subscribe-instead-of-polling-usage).
See the [Gateway RPC method reference](/gateway/protocol/rpc-methods#rpc-method-families)
for usage methods and request options.
**Message flow**
- `webhook.received` / `webhook.processed` / `webhook.error`
- `message.queued` / `message.processed`
- `message.delivery.started` / `message.delivery.completed` / `message.delivery.error`
**Gateway RPC**
- `gateway.rpc` - trusted request observations with phases `received`, `response`,
`handler`, and `dispatch`. Response outcomes are `ok`, `error`, `unavailable`,
or `suppressed`; handler outcomes are `returned` or `threw`; dispatch outcomes
are `returned`, `threw`, `rejected`, or `cancelled`. Dispatch records its response
state (`none`, `sent`, `unavailable`, or `suppressed`) at settlement; a later
response can still arrive. Durations and queue/admission semantics are described
in [Gateway RPC metrics](/gateway/opentelemetry#gateway-rpc).
**Queue and session**
- `queue.lane.enqueue` / `queue.lane.dequeue`
- `session.state` / `session.long_running` / `session.stalled` / `session.stuck`
- `run.attempt` / `run.progress`
- `run.execution_phase` (public, session-correlated embedded-runner startup milestones)
- `diagnostic.heartbeat` (aggregate counters: webhooks/queue/session)
- `gateway.event_loop.sample` (internal metrics-only completed window: `intervalMs`, `delayMaxMs`; no reader identity)
**Harness lifecycle**
- `harness.run.started` / `harness.run.completed` / `harness.run.error` -
per-run lifecycle for the agent harness. Includes `harnessId`, optional
`pluginId`, provider/model/channel, and run id. Completion adds
`durationMs`, `outcome`, optional `resultClassification`, `yieldDetected`,
and `itemLifecycle` counts. Errors add `phase`
(`prepare`/`start`/`send`/`resolve`/`cleanup`), `errorCategory`, and
optional `cleanupFailed`.
**Exec**
- `exec.process.completed` - terminal outcome, duration, target, mode, exit
code, and failure kind. Command text and working directories are not
included.
- `exec.approval.followup_suppressed` - stale approval follow-up dropped
after a session rebound. Includes `approvalId`, `reason`
(`session_rebound`), `phase` (`direct_delivery` or `gateway_preflight`),
and the dispatcher timestamp. Session keys, routes, and command text are
not included.