Skip to main content

​
Luna Voice Search Hardening Design

​
Goal

Make Luna the only production voice-eval target and make voice knowledge/instruction answers search-first, multilingual, grounded, and low-latency without preloading an organization’s corpus into the worker.

​
LiveKit research decision

LiveKit Agents 1.5.1 enables preemptive generation by default. The SDK begins the LLM request when it receives a final transcript segment before end-of-turn confirmation, but it waits for the speech handle to be scheduled and authorized before executing function tools. LiveKit documents Agent.llmNode as the customization point for observing LLM chunks and tool calls. Primary documentation: The documented RAG hook, on_user_turn_completed, runs after the turn; it cannot hide retrieval under a long caller utterance. The exact installed SDK source also confirms that preemptive generation starts LLM inference early but defers normal function-tool execution until the speech handle is authorized. This is why the design observes completed search arguments in llmNode instead of speculatively executing arbitrary LiveKit tools. LiveKit documents interim transcript events, but does not document executing tools from unstable interim text or reconciling those effects when a transcript is revised. The design therefore does not invent an interim-transcript tool runner. It uses LiveKit’s supported preemptive generation window: a final STT segment can start the LLM while the caller is still speaking or while end-of-turn confirmation is pending, and only completed read-only search arguments are executed early. LiveKit also documents that preemptive generation is skipped once speech exceeds maxSpeechDuration, whose default is 10 seconds. Production raises that bound to 60 seconds so ordinary long support explanations can still overlap LLM and search work with the caller’s speech. Preemptive TTS remains disabled, retries remain capped at three, and turns longer than one minute fall back to normal post-turn generation to bound discarded model/search work. The worker will therefore observe completed search_knowledge tool-call arguments in llmNode. During a preemptive generation it may start that read-only search immediately and cache the in-flight promise. When LiveKit later authorizes the tool after end-of-turn, the normal tool execution reuses the exact promise. Speculative transfer, workflow, action, survey-submission, and hangup tools remain prohibited because they have side effects and a preemptive generation can be discarded.

​
Search contract

search_knowledge accepts one to four standalone queries. Luna supplies, in order when applicable:
  1. A concise query in the caller’s language.
  2. An English equivalent.
  3. English help-center title keywords.
  4. A positive-only query that omits explicitly excluded products or paths.
The model-generated query array remains the semantic retrieval input. It is not trusted as the sole workflow-state representation: Luna can occasionally write a query for the generic step it expects to take next instead of the branch the caller is actually in. Before preemptive generation begins, the worker therefore captures a separate bounded turnContext from the authorized chat context. It contains the latest caller utterance, the most recent structured-value length, and allowlisted call metadata such as caller country. Long numeric values are replaced with their digit count, and unrelated session metadata is excluded. turnContext stays outside the embedding batch and is used only to focus the canonical instruction excerpt on the active branch. The same redaction is enforced on the model-generated query array before the backend or embedding provider receives it. Prompting Luna to omit a raw reference is useful, but it is not treated as a privacy boundary: any run of four or more structured digits becomes a count marker such as [10-digit structured value] while product and branch terms remain intact. The backend embeds the array in one request, restores provider response order by embedding index, and searches instruction and knowledge candidates independently for every embedding. At most two query variants search concurrently to protect the database pool; all variants are still evaluated. The merge combines reciprocal rank with cross-query consensus, preserves represented source categories, deduplicates candidates, and applies a deterministic final rank. Instruction and knowledge allowlists remain independent and strict. No selection means all content in that category is searchable.

​
Grounding and context

The final rank reserves space for both source categories when both have relevant candidates. A calibrated relevance rule may return no_match even when vector search returned nearby documents. Results expose no internal score to the model; score and candidate diagnostics go to telemetry. The worker keeps only a small per-call cache of read-only searches keyed by normalized query arrays. It does not preload source content. A cached request is single-use and can be consumed only by a tool execution with exactly matching arguments. At most three speculative requests may remain in flight; pending requests are never evicted merely to start more backend work. Discarded preemptive generations cannot update call state. A follow-up query may include the IDs returned by the immediately preceding successfully executed search so the backend can pin those authorized sources into the candidate set. The backend revalidates every pinned ID against the current instruction/knowledge scope and contact segments. Instruction results are rendered from the current canonical instruction, never from a stale vector chunk. Secondary instructions contain only the section that matches the current branch. The primary instruction also contains the nearest numbered workflow prerequisite: Step 1 may include Step 2, while a later step includes its immediately preceding step. This preserves required lookup calls without exposing an unrelated later troubleshooting branch early. A streamed question is a hard runtime boundary: once the model asks the caller a question, dependent tool calls later in that generation are suppressed and must wait for the caller’s next turn. Structured-value ranking recognizes both numeric and spoken-word count phrases such as “exactly 8 digits” and “eight-digit confirmation.” An exact-count branch is promoted, while a title that explicitly describes fewer, more, invalid, or wrong values at that same count is penalized. Result order remains a relevance hint rather than an authorization boundary: an exact trigger in a later result overrides an unrelated earlier result, but never permits skipping a prerequisite. The backend gives embeddings a four-second tail budget inside a 5.5-second absolute search deadline; the worker aborts at six seconds, leaving transport headroom. These are ceilings, not delays. Successful K=10 TicketSwap tool calls remained 721 ms p50 and 1,567 ms p95. The OpenAI Responses adapter also converts LiveKit’s exact function-choice shape to the provider’s expected shape so the forced search contract works on both supported OpenAI API modes. The multi-query wire change is an intentional backend/worker contract break, so the voice contract version and deployment guard are bumped together. This keeps an old worker expecting singular query from deploying against a backend that requires queries, and vice versa.

​
Search-first enforcement

Before the initial LLM generation for a substantive caller turn, the worker forces search_knowledge when the agent has the reserved search tool and no search result follows the latest caller message. Greetings, short acknowledgements, single survey/keypad answers, and farewells are exempt. Long structured values such as reference numbers are substantive. After the search result, normal automatic tool selection resumes.

​
Workflow reliability

Search hardening ships first and the focused Luna evals are rerun before workflow changes. Remaining authentication, troubleshooting, and survey failures must be reproduced by focused tests. Reusable state belongs in worker/backend orchestration, not org-specific prompt branches. Tool results should expose a typed next action, required input, and attempts remaining where the underlying workflow contract can support them. A survey controller owns the active question, answer binding, handoff lock, and exactly-once submission.

​
Output and harness reliability

Spoken digit sequences, telephone numbers, and initialisms use deterministic formatting before TTS. Terminal tool-only turns are recorded as valid tool outcomes by the eval harness; latency evaluation does not require a streamed sentence when the turn is intentionally terminal. The production-derived MoneyGram scenarios exposed three harness defects that were not model failures: caller-role wording was too narrow, confirmation was checked after field-entry prompts, and the C V V evaluator rejected the natural compact spelling. Those parsers now accept production-equivalent wording. The remaining ten-digit failures were a real retrieval/context gap: the correct instruction reached the model, but a model-generated query and a broad secondary instruction could advance to sender/receiver troubleshooting before the exact money-order/money-transfer branch completed. Separate turn context plus section-scoped instruction output raised that focused Luna gate from 7/10 to 10/10 (voice-search-moneygram-ten-digit-luna-k10-v6-20260826).

​
Verification

Every production change follows red-green-refactor. Unit and focused integration tests run without starting, stopping, or killing external workers. Eval order is focused Luna K=1, full Luna K=1, focused Luna K=10, then the Luna-only K=10 gate. All gating scenarios must pass 100%.