UNPKG

workflow

Version:

Workflow SDK - Build durable, resilient, and observable workflows

72 lines (49 loc) • 5.82 kB
--- title: deployment-mismatch description: A workflow run was delivered to a deployment other than the one it is pinned to. type: troubleshooting summary: Understand how Workflow recovers from a misrouted delivery, and why a run eventually fails with DEPLOYMENT_MISMATCH. prerequisites: - /docs/foundations/workflows-and-steps related: - /docs/foundations/versioning - /docs/errors/runtime-decryption-failed - /docs/foundations/errors-and-retries --- Every run is pinned to a single deployment when it starts. When a queued workflow or step callback is delivered to a **different** deployment, Workflow does not execute it there. Instead it re-routes the message to the deployment the run is pinned to, and only if the run keeps arriving elsewhere does it fail with the `DEPLOYMENT_MISMATCH` classification. This is an SDK/runtime signal, not an error thrown by your workflow code, and it is not catchable inside a workflow function. ## Error message ```text Workflow run "wrun_..." is pinned to deployment "dpl_A", but was received by deployment "dpl_B". The runtime re-routed the message to "dpl_A" 3 times and it kept arriving elsewhere, so the run was stopped to protect against code-skew errors. Verify that the run's deployment is still available and that queue callbacks are routed to it. ``` When the queue definitively reports that the run's deployment cannot be reached (it was deleted, or aged out of its retention window), no re-route is possible and the message omits the re-routing clause. Transient or unknown publishing failures leave the current delivery unacknowledged so the queue can redeliver it; they do not fail the run or consume this recovery budget. ## Why a run is pinned A run's deployment is chosen once, at [`start()`](/docs/api-reference/workflow-api/start): - By default it is the deployment that called `start()`. See [Versioning](/docs/foundations/versioning) for why runs are pinned this way. - With `start(workflow, args, { deploymentId })` it is the id you pass, so a run can deliberately target a deployment other than the one that created it. - With `deploymentId: "latest"` it is the most recent deployment for the current environment, resolved at start time. Whichever it is, that `deploymentId` is recorded on the run, and every subsequent workflow replay and step execution must happen on that deployment. Continuing on a different one is unsafe: 1. **Code skew.** The workflow and step bundles on the receiving deployment may not match the code that produced the run's recorded history, so replay could diverge or produce incorrect results. 2. **Encryption.** Step inputs and other event-log payloads are encrypted with a per-run key derived from the pinned deployment's key material. A different deployment derives the wrong key and cannot decrypt them, previously the source of a confusing [runtime-decryption-failed](/docs/errors/runtime-decryption-failed) that exhausted retries with no clear cause. So the runtime checks the pinned deployment before it executes anything, and `DEPLOYMENT_MISMATCH` names the result, instead of the mismatch surfacing later as an unrelated decryption failure. ## Automatic recovery A deployment that receives a run it does not own first tries to fix the delivery rather than fail the run: 1. It re-enqueues the message **explicitly addressed** to the run's own deployment. This is strictly better-addressed than the send that misrouted, which inherited the producing deployment's ambient id. 2. Delivery is delayed with a short exponential backoff (1s, 2s, 4s). 3. If the run keeps arriving at the wrong deployment, the run is failed with `DEPLOYMENT_MISMATCH` after `WORKFLOW_DEPLOYMENT_MISMATCH_MAX_RETRIES` attempts (default `3`). Set it to `0` to fail on the first misrouted delivery instead. Nothing is executed on the wrong deployment during recovery: no workflow code, no step body, no `step_started`, and no hook resume. Whatever the delivery was carrying travels with it, so a pending step keeps its identity and a hook resume keeps its payload: they run on the deployment that can actually decrypt them. Recovery attempts do not create events on the run, so a run that self-heals looks completely normal. They are reported on the invocation's trace span (`workflow.deployment.pinned_id`, `workflow.deployment_mismatch.retry_count`, `workflow.deployment_mismatch.recovered`) and as a runtime warning in your function logs. ## What to do - **Re-run from the current deployment.** Trigger the workflow again from your latest deployment (or use the **Re-run** button in the Workflow Dashboard). The new run is pinned to the current deployment. - **Keep a run's deployment available** for the lifetime of that run. A run whose deployment has been deleted or has aged out cannot be resumed and must be re-run: recovery cannot help, so these fail on the first misrouted delivery. This applies to runs started with an explicit `deploymentId` too: pinning a run to an older deployment keeps it dependent on that deployment for its whole lifetime. - **Report it** if the pinned deployment was still available. Include both deployment IDs and the run ID from the error message, plus the trace span attributes above. A run that failed this way despite a reachable target is a routing fault worth investigating rather than something to work around. ## This error cannot be caught Like other runtime signals, `DEPLOYMENT_MISMATCH` is **not catchable** inside your workflow function: the run is failed before any workflow or step code executes on the receiving deployment. Check the run status from outside instead: ```typescript lineNumbers import { getRun } from "workflow/api"; const run = getRun("wrun_abc123"); const status = await run.status; if (status === "failed") { console.error("Run failed"); } ```