Skip to main content

​
Voice Lifecycle and Deployment Readiness Design

​
Context

PR #2260 introduced the network-isolated LiveKit voice worker and its multi-region rollout. The first production rollout exposed five follow-up areas: the deployment gate waited on eventually-consistent replica counts, room-join latency was not measured, lk agent logs hid cross-replica failures, regional fallback behavior was easy to misread, and a MoneyGram survey call recovered from an SDK teardown race only because ShutdownCrashGuard ran its fallback. The production memory alert also mixes CI and production events. During the seven days ending 2026-08-25, CI emitted 76 supervised process run failed events. Production US emitted two distinct supervisor failures and two distinct jobs that crossed the 2400 MB memory limit. The most recent Slack page was a CI runner-initialization timeout, not production memory pressure.

​
Goals

  • Give SIP caller disconnects one shutdown owner so the SDK does not close the same session concurrently.
  • Preserve post-call side effects and make future teardown errors queryable.
  • Treat dispatch-to-room-join latency as the deployment readiness signal, with a 5000 ms hard limit.
  • Tolerate slow replacement-process initialization without treating replica statistics as call counts.
  • Separate production memory, production supervisor, and CI failure signals.
  • Document multi-region rollout, fallback, rollback, replica interpretation, and cross-replica diagnosis.

​
Non-goals

  • Upgrade @livekit/agents from 1.5.1. The 1.6 line includes a broader telemetry/runtime migration, while upstream direct-close deduplication remains unresolved.
  • Patch LiveKit SDK internals.
  • Raise the two-process production warm pool or the 2000/2400 MB warning/kill thresholds. Two real production jobs reached the current kill limit; capacity must not be increased until the memory recovery workload proves it safe.
  • Claim that a GitHub-hosted smoke test exercises every production region. LiveKit routes the shared agent name to an available regional deployment and exposes no per-dispatch region pin for deployments sharing that name.

​
SIP shutdown ownership

For SIP sessions, start AgentSession with RoomInputOptions.closeOnDisconnect=false. The existing caller-disconnect listener remains responsible for recognizing every SIP disconnect reason, including reasons omitted by the SDK’s default allowlist. On the first eligible caller disconnect:
  1. Mark call teardown as started.
  2. Arm ShutdownCrashGuard.
  3. Mark context shutdown as initiated.
  4. Call ctx.shutdown('participant_disconnected') directly.
The LiveKit job runner then performs the only session.close(). When that close emits the session Close event, the listener sees that context shutdown is already initiated and does not issue another shutdown. WebRTC sessions keep the SDK default closeOnDisconnect=true behavior. ShutdownCrashGuard remains installed for unrelated SDK teardown faults and continues to single-flight all post-call effects.

​
Error telemetry

logArgs currently drops an Error nested in an object such as { _e: error }, which is the repository’s normal structured logging form. Flatten nested errors into deterministic attributes:
  • <key>.name
  • <key>.message
  • <key>.stack when present
Top-level Error arguments continue to emit the existing error.message and error.stack attributes. This makes the exact teardown exception available in ClickHouse without changing logging call sites.

​
Deployment and join readiness

  • Increase WORKER_SERVER_OPTIONS.initializeProcessTimeout from 60 seconds to 90 seconds. The 60-second value still produced a CI replacement-runner timeout during the intentional 15-room burst.
  • Increase LiveKit status-retry from 10 to 15 minutes.
  • Change the regional readiness script to require two consecutive Running observations but only report replica counts. A 0 / 1 / 16 reading is eventually-consistent control-plane state, not evidence that no worker can accept a call.
  • Retain the 45-second regional soak so the two idle children prewarm before the next region drains.
  • Measure from successful dispatch submission to the remote agent participant appearing in the smoke room.
  • Record smoke_test.agent_join_latency_ms and smoke_test.agent_join_slo_ms on the attempt span and print the measured value.
  • Reject an attempt above 5000 ms with the low-cardinality reason agent_join_slo_exceeded. Existing retries create a fresh room, so transient routing can recover while the slow attempt remains visible in ClickHouse.

​
Deployment identity

Every deployment must emit explicit resource attributes rather than requiring hostname parsing:
  • deployment.environment=ci|production
  • deployment.region=ci-us-east|us-east|eu-central|ap-south
  • livekit.agent.id=<CA_...>
The workflow pins the full OTEL_RESOURCE_ATTRIBUTES value through the existing lk agent update-secrets merge for the CI agent and each production regional agent. The Dockerfile retains safe production defaults for manual deployments.

​
Memory and supervisor alerts

The existing combined alert is not an actionable production alert. Replace it in HyperDX with separate queries:
  1. Production memory-limit kills: deployment.environment=production and exact body job process exceeded memory limit, killing it; page immediately.
  2. Sustained production memory warnings: production-only warning events, grouped by agent/host/process and requiring consecutive windows.
  3. Production supervisor failures: production-only exact body supervised process run failed; page separately with the nested error attributes.
  4. CI failures: exclude from production paging. GitHub Actions remains the primary CI failure surface.
Checked-in queries must filter the otel_logs ORDER BY prefix (ServiceName, then TimestampTime), remain time-bounded, and use explicit limits in investigative variants. This follows schema-pk-filter-on-orderby, agent-query-safety, and query-index-skipping-indices from the ClickHouse best-practices skill. The repository does not contain HyperDX alert infrastructure-as-code. The PR therefore checks in the exact replacement queries and operator procedure; applying them to the hosted alert is an external configuration action.

​
Regional operations and observability

The voice debugging runbook will document:
  • current / minimum / maximum replicas are scheduler state, not calls, jobs, or proof that a test call created an instance.
  • A saturated or rolling regional deployment can cause LiveKit to route a shared agent name to another healthy region.
  • Serial rollout keeps two regions available, but a failed rollout can leave a mixed-version fleet; stop and roll back only the affected production deployment before continuing.
  • lk agent logs is a live tail of one attached replica and cannot prove fleet-wide absence of an error.
  • ClickHouse queries join calls, jobs, rooms, session IDs, agent IDs, regions, and versions across replicas.
  • Join latency is queried from lk-smoke-verifier attempt spans and compared with the 5000 ms SLO.

​
Verification

Behavior tests cover SIP input ownership, single context shutdown, nested error serialization, the 5000 ms join boundary, worker initialization timeout, and workflow readiness semantics. Per the requested workflow, tests execute on the PR branch rather than locally. Required local type checking, linting, shell syntax checks, actionlint, and git diff --check still run before the implementation is presented as complete. The PR is not ready until its required CI checks pass.