← @bgSponge

How we built a realtime system for responsive voice AI in six months

Summary

OpenAI engineers Justin Uberti and Zahan Malkani explain GPT-Live, a full-duplex voice system that drops the turn detector and streams audio in and out of a stateful voice model while frontier models and tools run asynchronously.

  1. Full-duplex removes the turn detector from the audio path.

    The voice model listens and speaks at once; deeper reasoning and tool use consult frontier models without stalling speech.

    GPT‑Live puts the voice model in control of the conversation

  2. Keep a dedicated live media path.

    Audio stays on a fast client-to-model loop written in Go over WebRTC; application logic and tools sit behind an async RPC boundary so slow backends cannot stall frames.

    Audio moves between the client and the voice model on a dedicated fast path.

  3. Statefulness needs seamless handoffs.

    Long sessions grow context past limits; compaction and instance swaps warm a replacement in parallel so the conversation never gaps.

  4. Derive turns after the fact.

    UI, analytics, and safety still need discrete messages, so the application server segments overlapping speech into provisional then finalized turns without imposing turn-taking on the live path.

  5. Startup and capacity are first-class latency.

    WARP plus Instant Connect start media with one UDP packet; silent production tests reframed capacity around concurrent sessions that keep every frame on schedule.

    the client can now start a session with a single UDP packet.

    Bringing GPT‑Live to ChatGPT scale required a whole new system built around one fundamental principle: the voice must flow.

Read the original ↗Save to your Sponge

Summarized with Sponge. Quotes are the author’s own words.

More from @bg

All readings →