Build Voice Agents: OpenAI Realtime API Guide

Building low-latency audio pipelines with the OpenAI Realtime API lets developers launch human-like conversational voice agents in production. Traditionally, building a voice interface meant chaining three separate model layers: automatic speech recognition (ASR), a text-based LLM logic layer, and text-to-speech (TTS) synthesis. That multi-step pipeline introduced significant round-trip network delays, making natural conversation impossible. Native audio processing over a persistent WebSocket connection changes this, reducing network latency below 300 milliseconds. This guide explains how to establish connection states, stream raw audio buffers, and optimise session configurations.

[!IMPORTANT] API Security Warning: Never expose your OpenAI API key directly inside client-side browser scripts. Always proxy the WebSocket connection through a secure edge middleware (such as a Cloudflare Worker) that appends authorisation headers before forwarding packets to OpenAI.

Key Takeaways:

• WebSocket Connection: Connect directly to OpenAI’s realtime WebSocket gateway using edge proxies.
• Native Modalities: Specify both text and audio in your initial session update config payload.
• Audio Format: Stream user speech as base64-encoded mono PCM16 chunks at 24kHz.
• Speech Interruption: Monitor server speech-started signals to halt client playback instantly.

The Architecture of Native Audio LLMs

Traditional voice stacks treat both speech recognition and synthesis models as external wrappers around a central text model. Native audio models remove that overhead by handling speech directly.

With OpenAI’s realtime models, the network processes audio inputs and outputs natively. The model receives raw audio waveforms, parses tone, inflection, and content, and generates natural voice output directly. As a result, the pipeline eliminates ASR transcription errors and TTS synthesis bottlenecks.

Managing this persistent connection relies on WebSockets. The connection remains open throughout the call, allowing the agent to interrupt its output if it detects user speech, so the experience matches a real telephone call.

Prerequisites

Before writing a line of code, make sure your environment is ready. This build assumes you are comfortable with asynchronous JavaScript and have the following in place:

• An OpenAI account with Realtime access and a funded, active API key.
• Node.js 20 or later, or a Cloudflare Workers project scaffolded with npm create cloudflare@latest.
• A WebSocket client — the ws package for a Node.js gateway, or the native WebSocket global available inside Workers.
• A browser frontend that can capture microphone audio through the Web Audio API (getUserMedia plus an AudioWorklet for resampling).
• Working knowledge of base64 and PCM audio, since every frame you send or receive is a base64-encoded PCM16 payload.

Budget a little time for account setup too: Realtime access and billing must be enabled on your organisation before the gateway will accept a session.

Establishing WebSocket Connections

To start, you open a connection to the OpenAI Realtime gateway, specifying the realtime model in your headers.

The JavaScript code below demonstrates how to initialise the connection, configure session modalities, and handle streaming input and output buffers:

 1import WebSocket from "ws";
 2
 3export async function startVoiceAgent(env) {
 4  // Connect to the OpenAI Realtime WebSocket gateway
 5  const url = "wss://api.openai.com/v1/realtime?model=gpt-realtime";
 6  const ws = new WebSocket(url, {
 7    headers: {
 8      "Authorization": `Bearer ${env.OPENAI_API_KEY}`,
 9      "OpenAI-Beta": "realtime=v1"
10    }
11  });
12
13  ws.on("open", () => {
14    console.log("WebSocket connection established with OpenAI Realtime API");
15    
16    // Configure session modalities and voice parameters
17    const sessionConfig = {
18      type: "session.update",
19      session: {
20        modalities: ["text", "audio"],
21        instructions: "You are a helpful customer service assistant for Mecanik.",
22        voice: "alloy",
23        input_audio_format: "pcm16",
24        output_audio_format: "pcm16",
25        temperature: 0.7
26      }
27    };
28    ws.send(JSON.stringify(sessionConfig));
29  });
30
31  ws.on("message", (data) => {
32    const event = JSON.parse(data);
33    
34    // Handle incoming audio content from the server
35    if (event.type === "response.audio.delta") {
36      const audioBuffer = Buffer.from(event.delta, "base64");
37      // Output buffer to client audio player
38      playAudioChunk(audioBuffer);
39    }
40  });
41}

When building this handler, ensure your API key remains hidden from the client browser. Specifically, establish an edge middleware on your server to handle authorisation before proxying the WebSocket connection. To learn about edge API configurations, read our guide on building a serverless API with Cloudflare Workers .

Building the Secure Edge Proxy

The snippet above connects from a trusted server, but it never showed the piece that keeps your key safe: the proxy itself. On Cloudflare Workers you cannot attach custom headers to the new WebSocket() constructor, so you open the upstream connection with fetch and an Upgrade header instead. The Worker accepts the browser’s socket, dials OpenAI with your secret key attached, then pipes frames between the two.

 1export default {
 2  async fetch(request, env) {
 3    if (request.headers.get("Upgrade") !== "websocket") {
 4      return new Response("Expected a WebSocket upgrade", { status: 426 });
 5    }
 6
 7    // 1. Accept the browser <-> Worker socket
 8    const [client, server] = Object.values(new WebSocketPair());
 9    server.accept();
10
11    // 2. Open the Worker <-> OpenAI socket with the secret key attached
12    const upstreamResponse = await fetch(
13      "https://api.openai.com/v1/realtime?model=gpt-realtime",
14      {
15        headers: {
16          Upgrade: "websocket",
17          Authorization: `Bearer ${env.OPENAI_API_KEY}`,
18          "OpenAI-Beta": "realtime=v1"
19        }
20      }
21    );
22
23    const upstream = upstreamResponse.webSocket;
24    if (!upstream) {
25      return new Response("Upstream refused the upgrade", { status: 502 });
26    }
27    upstream.accept();
28
29    // 3. Pipe frames in both directions
30    server.addEventListener("message", (e) => upstream.send(e.data));
31    upstream.addEventListener("message", (e) => server.send(e.data));
32
33    const close = () => { try { server.close(); upstream.close(); } catch {} };
34    server.addEventListener("close", close);
35    upstream.addEventListener("close", close);
36
37    return new Response(null, { status: 101, webSocket: client });
38  }
39};

Store the key as an encrypted secret with wrangler secret put OPENAI_API_KEY rather than in wrangler.toml, so it never lands in your repository. The browser now connects to wss://your-worker.workers.dev and never sees the credential. Cloudflare documents this bidirectional pattern in its Workers WebSockets reference .

Handling User Audio Streaming

Once the session update is accepted, your client must capture microphone input, compress it to 24kHz mono PCM16 data, and stream it as base64 segments. Because the connection remains persistent, handling network dropouts is critical. A local chunk buffer ensures that brief cellular connection dropouts do not result in audio packet loss or jittery agent responses: the client retains the buffer and replays it immediately upon reconnect.

1// Example of streaming user microphone data
2function streamMicrophoneChunk(ws, base64AudioChunk) {
3  const audioEvent = {
4    type: "input_audio_buffer.append",
5    audio: base64AudioChunk
6  };
7  ws.send(JSON.stringify(audioEvent));
8}

Whenever the user stops speaking, the server processes the accumulated audio buffer automatically and triggers a model response. For a complete API event list, review the OpenAI Realtime Guide .

Additionally, implement echo cancellation in your frontend player. If the microphone picks up the speaker output, the agent will interpret its own voice as a user interruption, causing the session loop to fail. To learn more about frontend optimisation, check our guide on WordPress vs custom web development .

Managing Turn Detection and Interruptions

By default you have to tell the model when a turn ends. Enabling server-side voice activity detection (VAD) hands that job to OpenAI: the gateway watches the incoming buffer, decides when the user has stopped speaking, and triggers a response automatically. Add a turn_detection block to the session update you send during the handshake.

 1const sessionConfig = {
 2  type: "session.update",
 3  session: {
 4    modalities: ["text", "audio"],
 5    voice: "alloy",
 6    input_audio_format: "pcm16",
 7    output_audio_format: "pcm16",
 8    turn_detection: {
 9      type: "server_vad",
10      threshold: 0.5,
11      prefix_padding_ms: 300,
12      silence_duration_ms: 500
13    }
14  }
15};

With VAD active, the server emits input_audio_buffer.speech_started the instant it hears the user talk over the agent. Treat that event as a hard stop: flush every queued audio chunk in your player, otherwise the previous response keeps playing while the new one arrives.

 1let playbackQueue = [];
 2
 3ws.on("message", (raw) => {
 4  const event = JSON.parse(raw);
 5  if (event.type === "input_audio_buffer.speech_started") {
 6    // User is interrupting: drop everything still buffered
 7    playbackQueue = [];
 8    stopSpeaker();
 9  }
10});

Raising silence_duration_ms makes the agent wait longer before replying, which suits slow or hesitant speakers; lowering it makes the exchange feel snappier but risks cutting people off mid-sentence.

Step-by-Step Voice Agent Deployment

To deploy your voice agent middleware, start by configuring a secure WebSocket routing gateway on serverless edge nodes to protect your OpenAI credentials. This prevents scrapers from discovering your keys.

Next, draft your system prompts and rules carefully. Set the tone, vocabulary, and response instructions inside the initial payload update session event. Consequently, this establishes a clear scope for conversational flows.

Then implement robust speaker interruption logic. You should listen for the input_audio_buffer.speech_started event and halt audio playback in your frontend player immediately. Finally, optimise global latency by deploying your Workers in close regional proximity to your users to reduce routing hops. To explore edge hosting options, read our Cloudflare Workers AI tutorial .

Common Pitfalls and Troubleshooting

Most first-run failures come from a handful of predictable mistakes. The table below maps the symptom you will see to its usual cause and fix.

Two subtler issues deserve attention. A rate_limits.updated event arriving with a low remaining balance is your early warning that concurrent sessions are about to be throttled; log it and back off rather than hammering reconnects. And if the agent occasionally answers its own last sentence, the microphone is capturing speaker output, so tighten echo cancellation or move testers onto headsets.

Testing and Production Considerations

Local testing is awkward because you need real audio flowing in both directions. The fastest loop is to run the Worker with wrangler dev, point a small browser page at it, and watch the event stream in the console before you worry about audio quality. Log every inbound event type during development; once the sequence of session.updated, speech_started, response.audio.delta, and response.done looks correct, you know the plumbing is sound.

Before you ship, weigh a few production realities:

• Cost. Realtime audio is billed per minute of input and output audio, which is far pricier than plain text tokens. Cap session length, and consider a text fallback for long informational answers.
• Latency. Deploy the proxy close to your users and keep it thin. Every extra hop between browser, edge, and OpenAI adds to the round trip you worked so hard to shrink.
• Reconnection. Sockets drop on mobile networks. Persist a short rolling buffer on the client and replay it on reconnect so a lost frame does not derail the conversation.
• Observability. Emit structured logs for session start, interruption count, and error events so you can spot degraded behaviour before users complain.
• Safety and privacy. Voice is personal data. Make retention explicit, and add server-side moderation on transcripts if your agent handles sensitive workflows.

A short pilot with real callers will surface accent, noise, and interruption edge cases that no scripted test covers, so run one before any wide launch.

Key Takeaways

• The OpenAI Realtime API processes audio natively via persistent WebSockets, keeping latency under 300ms.
• Always configure modalities, formats, and prompts during the initial WebSocket handshake.
• Stream user audio as mono PCM16 base64 segments in real-time.
• Handle speaker interruption events by monitoring speech-started event outputs.
• Maintain API credential security by proxying connection details through edge worker routers.

Frequently Asked Questions

What is the OpenAI Realtime API? The OpenAI Realtime API is a WebSocket interface that allows developers to stream raw audio in and out of the model, bypassing separate ASR/TTS modules. By processing audio natively, the model preserves emotional tone, accent inflections, and speech nuances, achieving latency times below 300 milliseconds.

How do I handle voice interruptions? Listen for the input_audio_buffer.speech_started server event. When received, clear your frontend audio buffers and halt speaker playback immediately. Consequently, this creates a natural conversational flow, allowing the AI agent to stop talking instantly when the user begins speaking.

What audio formats does the API support? The API natively supports 24kHz mono PCM16 (raw 16-bit signed integer) and G.711 (u-law and a-law) audio formats. Developers must capture microphone data, convert it to these specific formats on the client-side, and stream it as base64-encoded strings inside JSON WebSocket frames.

Do I need a separate server to coordinate WebSocket traffic? Yes. Running a serverless edge coordinator (such as Cloudflare Workers or a lightweight Node.js gateway) is highly recommended. The proxy receives microphone inputs from client browsers, attaches secure authorization headers, and redirects the binary stream to OpenAI’s gateway.

How do I prevent echo loops in real-time voice agents? Echo loops occur when the speaker audio leaks back into the client microphone, triggering false speaker interruption events. Developers must implement echo cancellation algorithms on the client-side or use headsets during testing to prevent feedback loops from breaking sessions.