Audio Streaming Protocols and Formats
You'll learn how to choose and implement the right protocols for transmitting live audio between clients and servers in real-time speech applications.
Why Protocols Matter for Live Audio
When building real-time ASR or TTS systems, you can't simply send entire audio files back and forth—latency would be unacceptable. Instead, you stream audio in small chunks as it's captured or generated. The protocol you choose determines latency, reliability, and compatibility.
Core Streaming Approaches
WebRTC
WebRTC (Web Real-Time Communication) is designed for peer-to-peer audio/video streaming with ultra-low latency (~50-200ms). It handles network adaptation, echo cancellation, and codec negotiation automatically. Perfect for conversational AI where you need bidirectional audio streaming with minimal delay. However, it's complex to set up and requires signaling servers.
WebSocket Audio Streaming
WebSockets provide a persistent, bidirectional TCP connection between client and server. You send audio chunks (like PCM frames) as binary messages in real-time. This gives you full control over encoding, buffering, and processing. It's simpler than WebRTC but lacks built-in audio optimizations. Most ASR APIs use this approach for streaming transcription.
Chunked Transfer Encoding (HTTP)
HTTP chunked transfer allows you to stream audio over a regular HTTP connection without knowing the total length upfront. The server sends Transfer-Encoding: chunked and transmits audio pieces sequentially. While not truly real-time, it's useful for progressive TTS delivery where you generate and stream speech as text is processed.
Format Considerations
Common formats for streaming include raw PCM (uncompressed, high bandwidth), Opus (low-latency compression), and μ-law/A-law (telephony). Your choice balances latency, quality, and bandwidth.
Key Takeaway: Choose WebRTC for ultra-low-latency bidirectional streaming, WebSockets for flexible real-time audio APIs, and HTTP chunked transfer for progressive audio delivery—each protocol fits different real-time audio deployment scenarios.