Course contentsShow
← AI Engineering
Lesson 1683 of 1,88641. Speech and Audio DeploymentFree lesson

Whisper Model Basics

Introduction to OpenAI's Whisper model architecture and its variants for speech recognition tasks.

Whisper Model Basics

What you'll learn: You'll understand OpenAI's Whisper model architecture and how its variants serve different speech recognition needs.

What is Whisper?

Whisper is OpenAI's open-source automatic speech recognition (ASR) model trained on 680,000 hours of multilingual and multitask supervised data from the web. Think of it as a universal translator for audio—it can transcribe speech in multiple languages, translate to English, and even identify what language is being spoken, all in one unified model.

Architecture Overview

Whisper uses an encoder-decoder Transformer architecture:

  • Encoder: Processes the audio input (converted to mel-spectrogram features) and creates a rich representation of what it "hears"
  • Decoder: Generates text tokens autoregressively, predicting one word at a time based on the encoded audio and previous words

This is similar to machine translation models, but adapted for the audio domain.

Model Variants

Whisper comes in five sizes, trading accuracy for speed and memory:

  • tiny (~39M parameters): Fastest, lowest memory, good for edge devices
  • base (~74M parameters): Balanced for quick processing
  • small (~244M parameters): Better accuracy, still reasonable latency
  • medium (~769M parameters): High accuracy for production systems
  • large (~1550M parameters): Best accuracy, requires GPU acceleration

Choosing the right variant depends on your latency requirements, hardware constraints, and accuracy needs. For real-time transcription on mobile devices, tiny or base work well. For high-quality offline transcription with GPU availability, large delivers superior results.

Key Takeaway: Whisper is a versatile encoder-decoder Transformer ASR model available in five sizes, allowing you to balance accuracy against computational resources for your specific deployment scenario.