How network routing, session setup, and interruption handling affect voice AI. Engineering lessons from OpenAI’s real-time media infrastructure.
This note draws on OpenAI’s account of low-latency voice infrastructure. The product checks below are our interpretation of its design lessons. Read the source article.
Voice interaction is unforgiving. A small delay in a text interface can look like a loading state. The same delay in conversation feels like hesitation. People notice a slow connection as an awkward pause, a missed interruption, clipped audio, or a reply that arrives after the moment has passed.
That makes latency a product concern. Model inference matters, but so do connection setup, network distance, packet loss, jitter, media routing, and recovery after a failure. A fast model cannot repair a media path that starts in the wrong place.
An original diagram of globally routed real-time media.
WebRTC solves a lot before the model runs
Standardize the client transport
WebRTC gives browsers and mobile clients a common way to establish real-time media sessions. It covers connectivity negotiation, NAT traversal, encrypted transport, codec agreement, and feedback about connection quality. That standardization is valuable because it keeps every client from inventing its own media stack.
For an AI product, the question becomes how to connect that well-understood client protocol to the systems that process audio and generate responses. At small scale, it is easy to give every session its own visible endpoint and keep all state close to that endpoint. At large scale, the public network footprint and the operational cost of that design become limiting.
The infrastructure has to preserve what the client expects while changing how the service handles packets behind the edge.
Separate packet forwarding from session ownership
Keep the edge narrow
A useful pattern is to split the public relay from the stateful session owner. The relay accepts packets at a small set of stable public addresses. It reads enough metadata to select the service that owns the session, then forwards the packet. The transceiver or session service remains responsible for the parts of WebRTC that require state: connectivity, encryption, codecs, and media handling.
This split keeps the public edge simple. The relay does not need to decrypt media, negotiate codecs, or reproduce the entire protocol state machine. It has one narrow job: get a packet to the right owner with as little added work as possible.
The session owner can then operate behind the public edge with normal internal scaling controls. It owns the context required to interpret later packets, while the relay remains easy to replicate across regions.
The first packet is the hard part
Route without a lookup pause
Routing later media packets is easier once a relay knows the mapping from a client address to a session owner. The challenge is the first packet. The relay must choose a destination before it has session state of its own.
OpenAI’s design uses routing information already present in the ICE connection flow. The server-controlled portion of the credential includes enough information for the relay to infer the intended cluster and session owner. The relay can inspect the first connectivity check, make the routing decision on the packet path, and establish a small forwarding record for subsequent traffic.
The broader principle is useful beyond WebRTC. If a protocol already carries a stable, authenticated hint that can guide routing, use it carefully instead of adding an external lookup to the latency-sensitive path. Every network round trip and shared dependency near the first packet becomes part of the user experience.
Global ingress is not only geography
Select an owner during setup
Putting relay points near users reduces the first hop across the public internet. That generally lowers latency, but physical distance is only part of the story. Internet routing, congestion, peering relationships, and the path between networks affect perceived quality too.
The signaling path can help choose the right regional session owner before media starts. The media relay can then provide a nearby ingress while forwarding to the owner selected during setup. The result is a deliberate distinction between where a packet enters the network and where the stateful work happens.
That distinction gives the system flexibility. A session can remain owned by the service that created it, while users begin from an edge location that is better for their connection.
Jitter and recovery are product features
Measure what the caller feels
Average latency can hide a poor voice experience. A call with good median latency but frequent bursts of delay still feels unreliable. Packet loss, jitter, route changes, and relay restarts all matter because people speak in real time.
The relay should therefore keep only the minimum state necessary for forwarding. If it restarts, the system should have a path to recover a route without asking the user to establish an entirely new conversation. Observability must also reflect what users feel: connection setup time, media round-trip time, packet loss, jitter, and interruption responsiveness.
Those measurements are more useful than a single generic latency number. They show whether a problem begins before the session, on the network path, inside media handling, or during model processing.
Optimize the common path first
Keep the relay’s work small
Real-time infrastructure invites ambitious optimization. Kernel bypass, specialized hardware, and deeply tuned packet pipelines can all be appropriate at a certain scale. They are not the first answer to every voice product.
The first goal is a narrow, observable path with clear ownership. Avoid unnecessary allocations, avoid expensive work at the relay, and keep the protocol state where it belongs. Measure the actual bottleneck before adding machinery that makes the system harder to operate.
This is a useful discipline for AI products in general. The most visible part of a system may be the model, but the first practical improvement often comes from making the data flow, routing, or state ownership clearer.
What I take from it
For interactive AI, the service is only as responsive as the slowest consequential step in the turn. The model may generate quickly, but users still wait for connection setup, media transport, and the system’s ability to handle an interruption.
The relay-and-owner pattern offers a clean way to reason about the system:
- Keep the public edge small and repeatable.
- Keep protocol state with the service that owns the session.
- Route the first packet using information already available in the connection flow.
- Measure stability, not only average speed.
Use these checks to separate network delays from model and tool delays. The user does not need to know where the relay or transceiver lives. They only need the interaction to feel immediate, interruptible, and dependable.