What is the OpenAI Whisper Speech Model?

Introduced by Alec Radford and OpenAI researchers in September 2022, Whisper represents a watershed breakthrough in automatic speech recognition. By releasing open model weights trained on an unprecedented scale, Whisper democratized state-of-the-art transcription, enabling developers to run private, high-accuracy speech-to-text directly on local consumer hardware.

Prior to Whisper, achieving human-parity transcription required subscribing to expensive, proprietary cloud APIs (such as Google Cloud Speech-to-Text or Amazon Transcribe). Whisper proved that a standardized Sequence-to-Sequence Transformer, when trained on massive and diverse real-world audio, could match or exceed commercial cloud benchmarks.

The Whisper architectural design

Whisper is structured as a classic Encoder-Decoder Transformer:

[Raw Audio @ 16kHz] 
        ↓
[Log-Mel Spectrogram (80 or 128 channels)]
        ↓
[2x 1D Convolutional Layers (stride 2)]
        ↓
[Transformer Encoder Blocks (Self-Attention + MLP)]
        ↓
[Transformer Decoder Blocks (Cross-Attention + Language Modeling)]
        ↓
[Text Tokens + Punctuation + Timestamps]

Key architectural features:

1. Audio windowing: Audio is processed in deterministic 30-second chunks, resampled to 16 kHz with 80-channel log-Mel filterbanks.
2. Multitask token format: Whisper unifies language identification, voice activity detection, speech transcription, translation to English, and timestamp alignment into a single sequential token vocabulary.
3. Weakly supervised dataset: Rather than relying on expensive human-annotated speech corpora (like LibriSpeech), Whisper learned from 680,000 hours of paired web audio and subtitles, imparting robust handling of background chatter, accents, and technical jargon.

Whisper model parameter tiers

Model Size	Parameters	VRAM Required	Relative Speed	Typical Application
Tiny	39 Million	~1 GB	~32x real-time	Low-power mobile & background scans
Base	74 Million	~1 GB	~16x real-time	Fast meeting notes on baseline M1
Small	244 Million	~2 GB	~6x real-time	High-accuracy balance for laptops
Medium	769 Million	~5 GB	~2x real-time	Technical interviews & accented speech
Large-v3	1,550 Million	~10 GB	~1x real-time	Multi-language translation & legal archive

The timestamp tokenization mechanism

A key innovation of Whisper is how it represents audio alignment without using an external phoneme aligner or CTC layer:

• Whisper uses special timestamp tokens representing 20-millisecond increments (e.g., <|0.00|>, <|0.02|>, …, <|30.00|>).
• The autoregressive decoder emits pairs of timestamp tokens surrounding speech phrases: <|0.40|> Meeting starts right now. <|2.80|>.
• This allows transcription software to extract precise word-level and sentence-level timecodes directly from the raw token probability distribution, enabling automatic clickable audio playback sync.

Decoding heuristics and hallucination mitigation

In noisy acoustic conditions or silent intervals, standard autoregressive decoding can enter repetitive loops or hallucinate text. Production Whisper runtimes apply several defensive heuristics:

1. Temperature fallback: The decoder begins with greedy decoding (temperature = 0.0). If the repetition penalty or average log probability fails a quality threshold, temperature increases in increments of 0.2 up to 1.0 to escape local probability valleys.
2. Compression ratio threshold: If the zlib compression ratio of the decoded text exceeds 2.4, the model has likely entered a degenerative repetition loop; the segment is discarded or re-decoded with fallback parameters.
3. No-speech probability gate (no_speech_prob): If the decoder predicts that an audio frame contains no human speech with greater than 60% confidence, the entire chunk is emitted as silence, preventing hallucinations over HVAC hums or keyboard clicks.

Whisper execution in Oyma

In Oyma on macOS, Whisper operates completely offline. When you record a video call or import an audio file, Oyma loads the model directly into Apple Silicon unified memory.

By taking advantage of Metal Performance Shaders and quantized 4-bit and 8-bit model weights, Oyma delivers sub-second transcription latency without running hot fans or burning battery life. Your conversations remain confidential on your Mac.