> ## Documentation Index > Fetch the complete documentation index at: https://docs.open.cx/llms.txt > Use this file to discover all available pages before exploring further. # 2026 08 25 voice lifecycle readiness design # Voice Lifecycle and Deployment Readiness Design ## Context PR #2260 introduced the network-isolated LiveKit voice worker and its multi-region rollout. The first production rollout exposed five follow-up areas: the deployment gate waited on eventually-consistent replica counts, room-join latency was not measured, `lk agent logs` hid cross-replica failures, regional fallback behavior was easy to misread, and a MoneyGram survey call recovered from an SDK teardown race only because `ShutdownCrashGuard` ran its fallback. The production memory alert also mixes CI and production events. During the seven days ending 2026-08-25, CI emitted 76 `supervised process run failed` events. Production US emitted two distinct supervisor failures and two distinct jobs that crossed the 2400 MB memory limit. The most recent Slack page was a CI runner-initialization timeout, not production memory pressure. ## Goals * Give SIP caller disconnects one shutdown owner so the SDK does not close the same session concurrently. * Preserve post-call side effects and make future teardown errors queryable. * Treat dispatch-to-room-join latency as the deployment readiness signal, with a 5000 ms hard limit. * Tolerate slow replacement-process initialization without treating replica statistics as call counts. * Separate production memory, production supervisor, and CI failure signals. * Document multi-region rollout, fallback, rollback, replica interpretation, and cross-replica diagnosis. ## Non-goals * Upgrade `@livekit/agents` from 1.5.1. The 1.6 line includes a broader telemetry/runtime migration, while upstream direct-close deduplication remains unresolved. * Patch LiveKit SDK internals. * Raise the two-process production warm pool or the 2000/2400 MB warning/kill thresholds. Two real production jobs reached the current kill limit; capacity must not be increased until the memory recovery workload proves it safe. * Claim that a GitHub-hosted smoke test exercises every production region. LiveKit routes the shared agent name to an available regional deployment and exposes no per-dispatch region pin for deployments sharing that name. ## SIP shutdown ownership For SIP sessions, start `AgentSession` with `RoomInputOptions.closeOnDisconnect=false`. The existing caller-disconnect listener remains responsible for recognizing every SIP disconnect reason, including reasons omitted by the SDK's default allowlist. On the first eligible caller disconnect: 1. Mark call teardown as started. 2. Arm `ShutdownCrashGuard`. 3. Mark context shutdown as initiated. 4. Call `ctx.shutdown('participant_disconnected')` directly. The LiveKit job runner then performs the only `session.close()`. When that close emits the session Close event, the listener sees that context shutdown is already initiated and does not issue another shutdown. WebRTC sessions keep the SDK default `closeOnDisconnect=true` behavior. `ShutdownCrashGuard` remains installed for unrelated SDK teardown faults and continues to single-flight all post-call effects. ## Error telemetry `logArgs` currently drops an `Error` nested in an object such as `{ _e: error }`, which is the repository's normal structured logging form. Flatten nested errors into deterministic attributes: * `.name` * `.message` * `.stack` when present Top-level `Error` arguments continue to emit the existing `error.message` and `error.stack` attributes. This makes the exact teardown exception available in ClickHouse without changing logging call sites. ## Deployment and join readiness * Increase `WORKER_SERVER_OPTIONS.initializeProcessTimeout` from 60 seconds to 90 seconds. The 60-second value still produced a CI replacement-runner timeout during the intentional 15-room burst. * Increase LiveKit `status-retry` from 10 to 15 minutes. * Change the regional readiness script to require two consecutive `Running` observations but only report replica counts. A `0 / 1 / 16` reading is eventually-consistent control-plane state, not evidence that no worker can accept a call. * Retain the 45-second regional soak so the two idle children prewarm before the next region drains. * Measure from successful dispatch submission to the remote agent participant appearing in the smoke room. * Record `smoke_test.agent_join_latency_ms` and `smoke_test.agent_join_slo_ms` on the attempt span and print the measured value. * Reject an attempt above 5000 ms with the low-cardinality reason `agent_join_slo_exceeded`. Existing retries create a fresh room, so transient routing can recover while the slow attempt remains visible in ClickHouse. ## Deployment identity Every deployment must emit explicit resource attributes rather than requiring hostname parsing: * `deployment.environment=ci|production` * `deployment.region=ci-us-east|us-east|eu-central|ap-south` * `livekit.agent.id=` The workflow pins the full `OTEL_RESOURCE_ATTRIBUTES` value through the existing `lk agent update-secrets` merge for the CI agent and each production regional agent. The Dockerfile retains safe production defaults for manual deployments. ## Memory and supervisor alerts The existing combined alert is not an actionable production alert. Replace it in HyperDX with separate queries: 1. Production memory-limit kills: `deployment.environment=production` and exact body `job process exceeded memory limit, killing it`; page immediately. 2. Sustained production memory warnings: production-only warning events, grouped by agent/host/process and requiring consecutive windows. 3. Production supervisor failures: production-only exact body `supervised process run failed`; page separately with the nested error attributes. 4. CI failures: exclude from production paging. GitHub Actions remains the primary CI failure surface. Checked-in queries must filter the `otel_logs` ORDER BY prefix (`ServiceName`, then `TimestampTime`), remain time-bounded, and use explicit limits in investigative variants. This follows `schema-pk-filter-on-orderby`, `agent-query-safety`, and `query-index-skipping-indices` from the ClickHouse best-practices skill. The repository does not contain HyperDX alert infrastructure-as-code. The PR therefore checks in the exact replacement queries and operator procedure; applying them to the hosted alert is an external configuration action. ## Regional operations and observability The voice debugging runbook will document: * `current / minimum / maximum` replicas are scheduler state, not calls, jobs, or proof that a test call created an instance. * A saturated or rolling regional deployment can cause LiveKit to route a shared agent name to another healthy region. * Serial rollout keeps two regions available, but a failed rollout can leave a mixed-version fleet; stop and roll back only the affected production deployment before continuing. * `lk agent logs` is a live tail of one attached replica and cannot prove fleet-wide absence of an error. * ClickHouse queries join calls, jobs, rooms, session IDs, agent IDs, regions, and versions across replicas. * Join latency is queried from `lk-smoke-verifier` attempt spans and compared with the 5000 ms SLO. ## Verification Behavior tests cover SIP input ownership, single context shutdown, nested error serialization, the 5000 ms join boundary, worker initialization timeout, and workflow readiness semantics. Per the requested workflow, tests execute on the PR branch rather than locally. Required local type checking, linting, shell syntax checks, `actionlint`, and `git diff --check` still run before the implementation is presented as complete. The PR is not ready until its required CI checks pass.