Skip to main content

​
Voice Lifecycle and Deployment Readiness Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.
Goal: Eliminate the observed SIP teardown close race and make measured room-join latency, environment-scoped alerts, and regional operations part of the voice deployment contract. Architecture: SIP sessions disable SDK disconnect auto-close and route the caller-disconnect event directly to a single context-shutdown request. The deployment workflow keeps control-plane status as a coarse gate while a production smoke verifier enforces the 5000 ms dispatch-to-join SLO and emits explicit deployment identity for cross-replica ClickHouse queries. Tech Stack: TypeScript, Vitest, LiveKit Agents 1.5.1, Bash, GitHub Actions, OpenTelemetry, ClickHouse/HyperDX. Spec: docs/superpowers/specs/2026-08-25-voice-lifecycle-readiness-design.md

​
Global Constraints

  • Keep @livekit/agents at 1.5.1; do not add a dependency patch.
  • Keep LIVEKIT_NUM_IDLE_PROCESSES=2, jobMemoryWarnMB=2000, and jobMemoryLimitMB=2400.
  • Enforce a 5000 ms dispatch-to-agent-join SLO.
  • Keep the voice worker network-isolated from backend, database, and Redis code.
  • Use strict TypeScript with no any and no type assertions except as const.
  • Run behavior tests on the PR branch, not locally; run local type, lint, shell, actionlint, and diff checks.
  • Keep every ClickHouse query bounded by ServiceName and TimestampTime.

​
Task 1: Make SIP disconnect shutdown single-owner

Files:
  • Create: backend/voice-agent/src/voice-room-input-options.utils.ts
  • Create: backend/voice-agent/src/voice-room-input-options.utils.spec.ts
  • Modify: backend/voice-agent/src/index.ts
Interfaces:
  • Produces: VoiceRoomInputOptionsUtils.forCall({ isSipCall }): { noiseCancellation: NoiseCancellationOptions; closeOnDisconnect?: false } using the existing telephony/WebRTC noise-cancellation constructors.
  • Consumes: existing ShutdownCrashGuard, beginCallTeardown, ctxShutdownInitiated, and ctx.shutdown(reason) lifecycle state in index.ts.
  • Step 1: Write the failing room-input test
Add cases proving SIP options set closeOnDisconnect: false while WebRTC options omit the override and preserve their respective noise-cancellation objects. The production mutation caught is re-enabling SDK SIP auto-close and recreating the double-close race.
  • Step 2: Push the test-only commit and verify CI fails for the missing utility
Expected failure: the spec cannot import VoiceRoomInputOptionsUtils or the SIP result lacks closeOnDisconnect: false.
  • Step 3: Implement the minimal input-options utility and wire it into session.start
Use a static utility class, following the repository rule for util/config functions. Do not change WebRTC behavior.
  • Step 4: Replace SIP session.shutdown with guarded direct context shutdown
On the first caller disconnect, call beginCallTeardown(), arm shutdownCrashGuard, set ctxShutdownInitiated=true, then call ctx.shutdown('participant_disconnected'). Keep the Close listener idempotent.
  • Step 5: Verify the PR test is green before continuing
Expected: the new input-options spec and the existing caller-disconnect/shutdown guard coverage pass.

​
Task 2: Preserve structured teardown errors

Files:
  • Create: backend/voice-agent/src/shared/telemetry.spec.ts
  • Modify: backend/voice-agent/src/shared/telemetry.ts
Interfaces:
  • Consumes: logArgs(...args: unknown[]).
  • Produces: nested error attributes <key>.name, <key>.message, and optional <key>.stack, while preserving top-level error.message and error.stack.
  • Step 1: Write failing serialization tests
Test logArgs('teardown failed', { _e: new Error('activity disappeared') }) against literal _e.name and _e.message values, plus a circular non-error object to preserve current best-effort behavior. The production mutation caught is silently dropping the nested _e object.
  • Step 2: Verify CI reports the missing nested attributes
Expected failure: _e.message is undefined.
  • Step 3: Implement minimal nested Error flattening
Handle Error before JSON serialization, add the three deterministic fields, and leave all other values unchanged.
  • Step 4: Verify the PR telemetry spec is green
Expected: nested and top-level error cases pass without enabling OTLP in tests.

​
Task 3: Enforce measured room-join readiness

Files:
  • Create: backend/scripts/voice-smoke/join-latency.utils.ts
  • Create: backend/scripts/voice-smoke/join-latency.utils.spec.ts
  • Modify: backend/scripts/voice-smoke-test-verify.ts
  • Modify: .github/workflows/voice-agent-deploy.yml
Interfaces:
  • Produces: VoiceSmokeJoinLatencyUtils.measure({ dispatchStartedAtMs, agentJoinedAtMs, maximumMs }): { latencyMs: number; withinSlo: boolean } and DEFAULT_AGENT_JOIN_SLO_MS=5000.
  • Consumes: existing setAttemptAttr and failSmokeAttempt in the verifier.
  • Step 1: Write failing boundary tests
Use literal timestamps to prove 4999 ms passes, 5000 ms passes, 5001 ms fails, and negative elapsed input throws. The production mutation caught is weakening or inverting the join SLO boundary.
  • Step 2: Verify CI fails because the latency utility is absent
Expected failure: module import or exported API missing.
  • Step 3: Implement the utility and instrument the verifier
Connect the verifier participant first, capture time immediately before calling createDispatch, measure when the remote agent appears, emit smoke_test.agent_join_latency_ms and smoke_test.agent_join_slo_ms, log the duration, and fail slow attempts with agent_join_slo_exceeded.
  • Step 4: Pin the 5000 ms workflow value for CI and production smoke invocations
Set VOICE_SMOKE_MAX_AGENT_JOIN_LATENCY_MS: 5000 in both verifier steps so the reviewed workflow makes the SLO explicit.
  • Step 5: Verify the PR latency tests and deployment guards are green
Expected: exact boundary behavior passes and both verifier invocations retain their environment.

​
Task 4: Correct worker and rollout readiness configuration

Files:
  • Modify: backend/voice-agent/src/agent.config.spec.ts
  • Modify: backend/voice-agent/src/agent.config.ts
  • Modify: .github/scripts/wait-for-livekit-agent-readiness.sh
  • Modify: backend/voice-agent/src/__tests__/deploy-workflow.guard.spec.ts
  • Modify: .github/workflows/voice-agent-deploy.yml
Interfaces:
  • Produces: WORKER_SERVER_OPTIONS.initializeProcessTimeout=90_000.
  • Produces: readiness script behavior that requires two consecutive Running samples, logs replica counts, and times out after 15 minutes without requiring current>=1.
  • Step 1: Change tests first
Expect the 90-second worker timeout and add executable readiness-script coverage with a fake lk command whose status is Running with 0 / 1 / 16; assert success after two polls. Add a timeout/failure case for never-Running status.
  • Step 2: Verify CI fails on the old 60-second and replica-count gates
Expected failures: worker timeout remains 60000 and 0 / 1 / 16 never becomes ready.
  • Step 3: Implement the timeout and readiness changes
Set initialization to 90000, status-retry to 15m, readiness deadline/error text to 15 minutes, and make Running the evaluated state while counts remain diagnostics.
  • Step 4: Verify PR configuration and script tests are green
Expected: the fake Running/zero-current status succeeds, never-Running fails, and workflow guard confirms TIMEOUT: 15m.

​
Task 5: Scope deployment telemetry and alert guidance

Files:
  • Modify: .github/workflows/voice-agent-deploy.yml
  • Modify: backend/voice-agent/src/__tests__/deploy-workflow.guard.spec.ts
  • Modify: backend/voice-agent/docs/DEBUGGING.md
  • Modify: backend/voice-agent/load-test/FINDINGS.md
Interfaces:
  • Produces deployment resource attributes deployment.environment, deployment.region, and livekit.agent.id for CI and production agents.
  • Produces checked-in ClickHouse queries for production memory kills, sustained warnings, supervisor failures, join latency, and cross-replica call diagnosis.
  • Step 1: Add failing deployment-identity guards
Require the CI update-secrets command to pin the CI environment/region/agent ID and the regional production command to pin the matrix region/agent ID. The production mutation caught is CI logs regaining the production default and waking the production alert.
  • Step 2: Verify CI fails on missing deployment identity
Expected failure: no OTEL_RESOURCE_ATTRIBUTES update exists.
  • Step 3: Pin explicit attributes through existing update-secrets steps
Pass the complete comma-separated resource attribute string for CI and production. Keep the Dockerfile default unchanged for manual production deployments.
  • Step 4: Add the operational runbook and rollout findings
Document replica semantics, shared-name regional fallback, serial rollout/rollback, the single-replica limitation of lk agent logs, bounded ClickHouse queries, the corrected alert split, and the 5000 ms join SLO. Record the 2026-08-25 production evidence and the decision not to loosen memory limits or raise the warm pool.
  • Step 5: Run non-test verification
Run pnpm tsgo, lint each edited TypeScript file, bash -n on edited shell scripts, actionlint on the workflow, and git diff --check. Do not run behavior tests locally.
  • Step 6: Push implementation and monitor required PR checks
Inspect every failed check. Fix failures in the same branch until required CI is green or a real external blocker is identified.

​
Task 6: Post-deploy evidence after merge

Files:
  • No repository changes unless monitoring exposes a regression.
Interfaces:
  • Consumes: production smoke span attributes and explicit deployment resource attributes.
  • Produces: observed join latency, MoneyGram inbound-call health, teardown status, and regional deployment state for the handoff.
  • Step 1: Do not merge this follow-up PR without explicit user instruction
The requested deliverable is an open PR. CI readiness does not authorize merge.
  • Step 2: After a future approved merge/deploy, query the new-version window
Verify room-join measurements remain at or below 5000 ms, no MoneyGram call has a teardown fallback/error, no memory-limit kill or supervisor failure appears, and all three regional deployments report Running.