Realtime Voice

Support reference for the current realtime voice server and client flows.

• HTTP/WebSocket runtime:
• `backend/src/realtimeServer.ts`
• WebSocket orchestration:
• `backend/src/components/realtime/RealtimeProxy.ts`
• Provider/runtime implementations:
• `backend/src/components/realtime/agents/realtime/OpenAIRealtimeAgent.ts`
• `backend/src/components/realtime/agents/realtime/GoogleAIRealtimeAgent.ts`
• `backend/src/components/realtime/agents/tts/TtsAgent.ts`
• WebSocket path:
• `/chat/realtime?session={sessionId}`
• Health endpoint:
• `GET /chat/realtime/health`

• Public webchat voice:
• `POST /chat/realtime/create-session/{employeeId}`
• The widget currently passes its configured `eventId` in that path slot.
• Internal workflow voice:
• `POST /chat/realtime/workflow/create-session/{organization_id}/{workflow_id}`
• Internal AI teammate and Ally voice:
• `POST /chat/realtime/employee/create-session/{organization_id}/{employee_id}`
• The frontend first resolves the current user's personal Ally employee, then passes that employee ID to this shared endpoint.

• Internal employee session creation requires exactly one of:
• `conversation_id`
• `create_new_conversation`
• Public webchat session creation can include:
• `contact`
• `multi_conversations`
• Successful create-session responses include:
• `sessionId`
• `isAudioAutoTurnOn`
• Internal employee responses also include `conversation_id`.
• Session IDs are one-time use and currently match the created workflow run ID.
• Pending session configs are stored in Redis with a 20-minute TTL and deleted on the first websocket connect.
• All realtime session routes reject `structured` workflows. The published starting step must resolve to an `agent` step with a configured TTS-capable employee.
• Supported `locale` values are currently:
• `en-US`
• `ru-RU`
• `es-ES`
• default fallback is `en-US`

• If the selected TTS model is realtime and the provider is Google:
• `GoogleAIRealtimeAgent`
• `GeminiLiveVoice`
• If the selected TTS model is realtime and not Google:
• `OpenAIRealtimeAgent`
• If the TTS model is non-realtime but still TTS-capable:
• `TtsAgent`
• OpenAI realtime STT is used when an STT model is configured
• The websocket runtime rebuilds a `RequestContext` from `workflowRun.state.runtimeContext` and adds `organizationId`, `employeeId`, and `runId` before the session starts.

• Gemini Live sends each registered tool through `parametersJsonSchema`; OpenAI Realtime sends it through `parameters`.
• Existing JSON Schema inputs pass through unchanged. Zod inputs are converted to JSON Schema without the top-level `$schema` marker.
• Provider declarations preserve nested object fields, array item schemas, required fields, optional fields, enums, nullable values, descriptions, and numeric constraints used by Alloy tools.
• OpenAI Realtime adapts the voice package execution envelope before invoking a current Mastra tool: the tool receives flat validated input, while runtime metadata including `RequestContext` is supplied separately.
• Gemini Live also invokes registered tools with the decoded argument object and the session request context.

• Clients send binary microphone audio over the websocket.
• The websocket wire format expects mono signed `16-bit PCM` audio at `24kHz`.
• The internal frontend explicitly resamples to `24kHz` before sending.
• Clients can also send JSON:
• `type: "user_message"`
• `data.text`
• optional `data.attachments`
• The server validates attachments for websocket `user_message` payloads before processing them.
• Attachment guardrails on websocket messages:
• maximum 5 files
• maximum 10 MB each
• allowed MIME families: images, PDF, plain text, XML, and SQLite
• For internal voice conversations, the client requests `session.capabilities` with `session.capabilities.request`; attachment sends require `data.attachments` and a `request_id` on `user_message`. The server advertises `data.attachment_messages` only when that session supports this path. The client does not submit attachments unless support is confirmed.
• The internal attachment path validates and saves the message before replying with `user_message.accepted` and the matching `request_id` (and a `message_id` when available). Before-save failures return `user_message.rejected`; if delivery to the AI fails after the save, the server sends `user_message.delivery_failed` instead. Repeating an accepted request ID returns the prior acknowledgement; simultaneous attachment requests are rejected. Text-only voice messages retain their existing path.
• In OpenAI realtime sessions, saved attachments are added to the provider conversation and a reply is requested when appropriate. Image content is limited to PNG, JPEG, WebP, and GIF in that path. The TTS path also saves attachments before invoking its text/voice response. If automation is off, internal voice attachment sends are unavailable. The ordinary webchat voice attachment path is separate.

• The realtime websocket currently sends JSON events such as:
• `session.created`
• `speech_started`
• `speech_stopped`
• `response.cancelled`
• `error`
• Audio responses are streamed back as binary websocket frames.
• `TtsAgent` additionally sends text-sidecar events such as:
• `text` with `role: user|assistant`
• `speaking.done`
• Internal chat also listens to org websocket events from the TTS path:
• `voice_thinking_started`
• `voice_thinking_ended`

• Realtime agents refresh automation state from the conversation while the session is running.
• If automation is disabled on the conversation, the voice session stays connected and still saves user messages, but skips AI response generation.
• If the organization is out of tokens or the resolved AI teammate has already reached its daily spending limit when voice starts, the server sets an eligible webchat conversation to `waiting_for_operator` and rejects the session.
• Native realtime and TTS paths record attributable usage and re-check the shared AI execution gate after each completed response.
• If a completed response crosses the organization token balance or employee daily spending limit, the response remains recorded, an eligible webchat conversation moves to `waiting_for_operator`, and the websocket closes with code `4008` and the canonical resource-limit message.
• A realtime close code at or above `4000` is mapped to a failed workflow run.