> ## Documentation Index
> Fetch the complete documentation index at: https://docs.open.cx/llms.txt
> Use this file to discover all available pages before exploring further.

# 2026 08 25 voice lifecycle readiness

# Voice Lifecycle and Deployment Readiness Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Eliminate the observed SIP teardown close race and make measured room-join latency, environment-scoped alerts, and regional operations part of the voice deployment contract.

**Architecture:** SIP sessions disable SDK disconnect auto-close and route the caller-disconnect event directly to a single context-shutdown request. The deployment workflow keeps control-plane status as a coarse gate while a production smoke verifier enforces the 5000 ms dispatch-to-join SLO and emits explicit deployment identity for cross-replica ClickHouse queries.

**Tech Stack:** TypeScript, Vitest, LiveKit Agents 1.5.1, Bash, GitHub Actions, OpenTelemetry, ClickHouse/HyperDX.

**Spec:** `docs/superpowers/specs/2026-08-25-voice-lifecycle-readiness-design.md`

## Global Constraints

* Keep `@livekit/agents` at 1.5.1; do not add a dependency patch.
* Keep `LIVEKIT_NUM_IDLE_PROCESSES=2`, `jobMemoryWarnMB=2000`, and `jobMemoryLimitMB=2400`.
* Enforce a 5000 ms dispatch-to-agent-join SLO.
* Keep the voice worker network-isolated from backend, database, and Redis code.
* Use strict TypeScript with no `any` and no type assertions except `as const`.
* Run behavior tests on the PR branch, not locally; run local type, lint, shell, actionlint, and diff checks.
* Keep every ClickHouse query bounded by `ServiceName` and `TimestampTime`.

***

### Task 1: Make SIP disconnect shutdown single-owner

**Files:**

* Create: `backend/voice-agent/src/voice-room-input-options.utils.ts`
* Create: `backend/voice-agent/src/voice-room-input-options.utils.spec.ts`
* Modify: `backend/voice-agent/src/index.ts`

**Interfaces:**

* Produces: `VoiceRoomInputOptionsUtils.forCall({ isSipCall }): { noiseCancellation: NoiseCancellationOptions; closeOnDisconnect?: false }` using the existing telephony/WebRTC noise-cancellation constructors.

* Consumes: existing `ShutdownCrashGuard`, `beginCallTeardown`, `ctxShutdownInitiated`, and `ctx.shutdown(reason)` lifecycle state in `index.ts`.

* [ ] **Step 1: Write the failing room-input test**

Add cases proving SIP options set `closeOnDisconnect: false` while WebRTC options omit the override and preserve their respective noise-cancellation objects. The production mutation caught is re-enabling SDK SIP auto-close and recreating the double-close race.

* [ ] **Step 2: Push the test-only commit and verify CI fails for the missing utility**

Expected failure: the spec cannot import `VoiceRoomInputOptionsUtils` or the SIP result lacks `closeOnDisconnect: false`.

* [ ] **Step 3: Implement the minimal input-options utility and wire it into `session.start`**

Use a static utility class, following the repository rule for util/config functions. Do not change WebRTC behavior.

* [ ] **Step 4: Replace SIP `session.shutdown` with guarded direct context shutdown**

On the first caller disconnect, call `beginCallTeardown()`, arm `shutdownCrashGuard`, set `ctxShutdownInitiated=true`, then call `ctx.shutdown('participant_disconnected')`. Keep the Close listener idempotent.

* [ ] **Step 5: Verify the PR test is green before continuing**

Expected: the new input-options spec and the existing caller-disconnect/shutdown guard coverage pass.

### Task 2: Preserve structured teardown errors

**Files:**

* Create: `backend/voice-agent/src/shared/telemetry.spec.ts`
* Modify: `backend/voice-agent/src/shared/telemetry.ts`

**Interfaces:**

* Consumes: `logArgs(...args: unknown[])`.

* Produces: nested error attributes `<key>.name`, `<key>.message`, and optional `<key>.stack`, while preserving top-level `error.message` and `error.stack`.

* [ ] **Step 1: Write failing serialization tests**

Test `logArgs('teardown failed', { _e: new Error('activity disappeared') })` against literal `_e.name` and `_e.message` values, plus a circular non-error object to preserve current best-effort behavior. The production mutation caught is silently dropping the nested `_e` object.

* [ ] **Step 2: Verify CI reports the missing nested attributes**

Expected failure: `_e.message` is undefined.

* [ ] **Step 3: Implement minimal nested Error flattening**

Handle `Error` before JSON serialization, add the three deterministic fields, and leave all other values unchanged.

* [ ] **Step 4: Verify the PR telemetry spec is green**

Expected: nested and top-level error cases pass without enabling OTLP in tests.

### Task 3: Enforce measured room-join readiness

**Files:**

* Create: `backend/scripts/voice-smoke/join-latency.utils.ts`
* Create: `backend/scripts/voice-smoke/join-latency.utils.spec.ts`
* Modify: `backend/scripts/voice-smoke-test-verify.ts`
* Modify: `.github/workflows/voice-agent-deploy.yml`

**Interfaces:**

* Produces: `VoiceSmokeJoinLatencyUtils.measure({ dispatchStartedAtMs, agentJoinedAtMs, maximumMs }): { latencyMs: number; withinSlo: boolean }` and `DEFAULT_AGENT_JOIN_SLO_MS=5000`.

* Consumes: existing `setAttemptAttr` and `failSmokeAttempt` in the verifier.

* [ ] **Step 1: Write failing boundary tests**

Use literal timestamps to prove 4999 ms passes, 5000 ms passes, 5001 ms fails, and negative elapsed input throws. The production mutation caught is weakening or inverting the join SLO boundary.

* [ ] **Step 2: Verify CI fails because the latency utility is absent**

Expected failure: module import or exported API missing.

* [ ] **Step 3: Implement the utility and instrument the verifier**

Connect the verifier participant first, capture time immediately before calling `createDispatch`, measure when the remote agent appears, emit `smoke_test.agent_join_latency_ms` and `smoke_test.agent_join_slo_ms`, log the duration, and fail slow attempts with `agent_join_slo_exceeded`.

* [ ] **Step 4: Pin the 5000 ms workflow value for CI and production smoke invocations**

Set `VOICE_SMOKE_MAX_AGENT_JOIN_LATENCY_MS: 5000` in both verifier steps so the reviewed workflow makes the SLO explicit.

* [ ] **Step 5: Verify the PR latency tests and deployment guards are green**

Expected: exact boundary behavior passes and both verifier invocations retain their environment.

### Task 4: Correct worker and rollout readiness configuration

**Files:**

* Modify: `backend/voice-agent/src/agent.config.spec.ts`
* Modify: `backend/voice-agent/src/agent.config.ts`
* Modify: `.github/scripts/wait-for-livekit-agent-readiness.sh`
* Modify: `backend/voice-agent/src/__tests__/deploy-workflow.guard.spec.ts`
* Modify: `.github/workflows/voice-agent-deploy.yml`

**Interfaces:**

* Produces: `WORKER_SERVER_OPTIONS.initializeProcessTimeout=90_000`.

* Produces: readiness script behavior that requires two consecutive Running samples, logs replica counts, and times out after 15 minutes without requiring `current>=1`.

* [ ] **Step 1: Change tests first**

Expect the 90-second worker timeout and add executable readiness-script coverage with a fake `lk` command whose status is `Running` with `0 / 1 / 16`; assert success after two polls. Add a timeout/failure case for never-Running status.

* [ ] **Step 2: Verify CI fails on the old 60-second and replica-count gates**

Expected failures: worker timeout remains 60000 and `0 / 1 / 16` never becomes ready.

* [ ] **Step 3: Implement the timeout and readiness changes**

Set initialization to 90000, status-retry to `15m`, readiness deadline/error text to 15 minutes, and make Running the evaluated state while counts remain diagnostics.

* [ ] **Step 4: Verify PR configuration and script tests are green**

Expected: the fake Running/zero-current status succeeds, never-Running fails, and workflow guard confirms `TIMEOUT: 15m`.

### Task 5: Scope deployment telemetry and alert guidance

**Files:**

* Modify: `.github/workflows/voice-agent-deploy.yml`
* Modify: `backend/voice-agent/src/__tests__/deploy-workflow.guard.spec.ts`
* Modify: `backend/voice-agent/docs/DEBUGGING.md`
* Modify: `backend/voice-agent/load-test/FINDINGS.md`

**Interfaces:**

* Produces deployment resource attributes `deployment.environment`, `deployment.region`, and `livekit.agent.id` for CI and production agents.

* Produces checked-in ClickHouse queries for production memory kills, sustained warnings, supervisor failures, join latency, and cross-replica call diagnosis.

* [ ] **Step 1: Add failing deployment-identity guards**

Require the CI update-secrets command to pin the CI environment/region/agent ID and the regional production command to pin the matrix region/agent ID. The production mutation caught is CI logs regaining the production default and waking the production alert.

* [ ] **Step 2: Verify CI fails on missing deployment identity**

Expected failure: no `OTEL_RESOURCE_ATTRIBUTES` update exists.

* [ ] **Step 3: Pin explicit attributes through existing update-secrets steps**

Pass the complete comma-separated resource attribute string for CI and production. Keep the Dockerfile default unchanged for manual production deployments.

* [ ] **Step 4: Add the operational runbook and rollout findings**

Document replica semantics, shared-name regional fallback, serial rollout/rollback, the single-replica limitation of `lk agent logs`, bounded ClickHouse queries, the corrected alert split, and the 5000 ms join SLO. Record the 2026-08-25 production evidence and the decision not to loosen memory limits or raise the warm pool.

* [ ] **Step 5: Run non-test verification**

Run `pnpm tsgo`, lint each edited TypeScript file, `bash -n` on edited shell scripts, `actionlint` on the workflow, and `git diff --check`. Do not run behavior tests locally.

* [ ] **Step 6: Push implementation and monitor required PR checks**

Inspect every failed check. Fix failures in the same branch until required CI is green or a real external blocker is identified.

### Task 6: Post-deploy evidence after merge

**Files:**

* No repository changes unless monitoring exposes a regression.

**Interfaces:**

* Consumes: production smoke span attributes and explicit deployment resource attributes.

* Produces: observed join latency, MoneyGram inbound-call health, teardown status, and regional deployment state for the handoff.

* [ ] **Step 1: Do not merge this follow-up PR without explicit user instruction**

The requested deliverable is an open PR. CI readiness does not authorize merge.

* [ ] **Step 2: After a future approved merge/deploy, query the new-version window**

Verify room-join measurements remain at or below 5000 ms, no MoneyGram call has a teardown fallback/error, no memory-limit kill or supervisor failure appears, and all three regional deployments report Running.