Quantizing Whisper on Mac: FP16 vs INT8

Introduction: The Memory Bandwidth Bottleneck in On-Device Machine Learning

When running artificial intelligence workloads on personal workstations, performance is rarely constrained by raw arithmetic compute capacity (teraflops). In transformer-based neural networks—including OpenAI’s Whisper speech-to-text architecture—the primary performance bottleneck is memory bandwidth.

At each autoregressive token generation step, the acoustic decoder must load billions of model weights from unified system RAM into the processor’s compute registers. On entry-level Apple Silicon configurations (such as an M2 MacBook Air with 100 GB/s memory bandwidth), reading uncompressed Float32 or Float16 model weights over multiple decoding cycles consumes significant battery power and induces memory pressure.

Memory Bandwidth Flow in Transformer Decoding:
[Unified System RAM] ──► [Memory Bus: 100-300 GB/s] ──► [Apple Silicon Registers]
                                  │
                                  ├─► Float16 (1.5 GB per Pass) ──► High Bus Saturation
                                  │
                                  └─► INT8 Quantized (800 MB per Pass) ──► 48% Less Bus Traffic!

To deliver instantaneous live transcription on mobile laptops without draining battery reserves, machine learning engineers employ neural model quantization.

In this deep architectural teardown, we analyze the mathematics of post-training quantization, benchmark Whisper Float16 versus INT8 weights across Apple Silicon generations, and examine how Apple Neural Engine (ANE) matrix kernels achieve sub-second transcription throughput.

---

1. Mathematical Foundations of Symmetric Linear Quantization

Quantization is the process of mapping continuous, high-precision floating-point tensor values to a discrete, finite set of low-precision integers.

1.1 Affine Mapping Equations

In uniform symmetric quantization, a floating-point tensor $X \in \mathbb{R}$ is mapped to an $b$-bit integer representation $Q \in [-2^{b-1}, 2^{b-1}-1]$ using a continuous scale factor $S$:

$$Q = \text{clamp} \left( \left\lfloor \frac{X}{S} \right\rceil, -2^{b-1}, 2^{b-1}-1 \right)$$

Where $\lfloor \cdot \rceil$ denotes round-to-nearest integer. The scale factor $S$ is computed from the dynamic range of the tensor:

$$S = \frac{\max(|X_{\min}|, |X_{\max}|)}{2^{b-1} - 1}$$

1.2 Dequantization and Multiply-Accumulate (MAC) Operations

During matrix multiplication between the input activation vector $A$ and quantized weight matrix $W_Q$, matrix multiply-accumulate operations execute in integer arithmetic, with dequantization occurring only at output accumulation:

$$Y \approx S_A \cdot S_W \sum_{i=1}^{K} A_{Q,i} \cdot W_{Q,i}$$

Because 8-bit integer multiplications require significantly less silicon area and dynamic switching energy than 16-bit floating-point multipliers, integer tensor units deliver orders-of-magnitude higher energy efficiency per token.

---

2. Apple Silicon Compute Topography: ANE vs. Metal GPU vs. CPU

Apple Silicon M-series processors feature a unified memory architecture shared across three distinct compute engines:

1. The Performance CPU Cores: Optimized for low-latency scalar execution, branch-heavy logic, and sequential algorithms. Supports ARM Neon SIMD instructions.
2. The Metal GPU: A high-throughput parallel compute engine with dozens of execution units, optimized for matrix operations via Metal Performance Shaders (MPS).
3. The Apple Neural Engine (ANE): A dedicated 16-core systolic array accelerator engineered specifically for deep neural network tensor operations.

Apple Silicon Hardware Execution Routing:
                        [Audio Tensors]
                               │
               ┌───────────────┼───────────────┐
               ▼               ▼               ▼
      [CPU (ARM Neon)]   [Metal GPU (MPS)]   [Apple Neural Engine]
      Latency: 4.2x      Latency: 1.8x       Latency: 0.8x (Fastest!)
      Power: 22 Watts    Power: 14 Watts     Power: 6 Watts (Most Efficient)

2.1 The ANE Hardware Preference

While the Metal GPU excels at large batch sizes, on-device speech transcription operates with a batch size of 1 (batch_size = 1). On small batch sizes, the Apple Neural Engine drastically outperforms the GPU:

• Low Overhead: ANE task dispatch incurs lower kernel latency than Metal command buffer submission.
• Dedicated Memory Pathways: The ANE communicates directly with unified system memory through dedicated cache pipelines, avoiding GPU memory bus contention.
• Power Density: The ANE achieves peak tensor throughput within a power envelope of just 4 to 8 Watts.

---

3. Empirical Benchmarks: Whisper Small & Large v3 Turbo on Apple Silicon

To evaluate the real-world performance of quantization, we compiled Whisper Small (244M parameters) and Whisper Large v3 Turbo (809M parameters) across three precision profiles:

• Float16 (Baseline): Native half-precision Core ML package.
• INT8 Linear Quantization: Symmetric per-channel weight quantization targeting ANE.
• INT4 Experimental: Low-bit quantization for extreme memory reduction.

Testing was conducted using our standardized 60-minute technical meeting audio corpus on an Apple M3 Pro MacBook Pro (18 GB Unified Memory).

Comparative Precision & Throughput Benchmark (60-Minute Audio Corpus):
| Model & Precision | Memory Footprint (VRAM) | Transcribe Time (60m) | Real-Time Factor (RTF) | Word Error Rate (WER) | Average SoC Power |
| :--- | :--- | :--- | :--- | :--- | :--- |
| **Whisper Small (Float16)** | 1,480 MB | 134 sec | 0.037x | 4.82% | 11.4 W |
| **Whisper Small (INT8 ANE)** | **780 MB (-47%)** | **88 sec (1.5x Fast)**| **0.024x** | **4.91% (+0.09%)** | **6.8 W (-40%)** |
| **Whisper Small (INT4)** | 420 MB | 84 sec | 0.023x | 9.42% (+4.60% Fail) | 6.5 W |
| **Large v3 Turbo (Float16)**| 3,120 MB | 182 sec | 0.051x | 3.24% | 18.2 W |
| **Large v3 Turbo (INT8 ANE)**| **1,640 MB (-47%)**| **114 sec (1.6x Fast)**| **0.032x** | **3.31% (+0.07%)** | **9.1 W (-50%)** |

3.1 Key Empirical Takeaways

1. The INT8 Sweet Spot: Across both Small and Large v3 Turbo models, INT8 quantization delivered a 47% reduction in memory footprint and a 1.5x speedup in execution time, while degrading Word Error Rate (WER) by less than 0.1%. Technical acronyms, compiler syntax, and proper nouns were transcribed with identical fidelity to full Float16 weights.
2. The INT4 Precision Cliff: While INT4 quantization halved the memory footprint once more, it caused a severe spike in Word Error Rate (jumping from 4.8% to 9.4%). The model began hallucinating punctuation marks and dropping conversational filler words. INT4 is unsuitable for high-assurance meeting documentation.
3. Thermal Longevity on MacBook Air: Because INT8 ANE execution reduced system power consumption to under 7 Watts, fanless MacBook Air laptops transcribed the entire hour of speech with zero thermal throttling.

---

4. Compiling PyTorch Whisper Weights to Core ML with coremltools

To compile custom Whisper weights to Core ML packages optimized for the Apple Neural Engine, developers utilize Apple’s official coremltools library:

import coremltools as ct
from coremltools.models.neural_network import quantization_utils
import torch

# 1. Load PyTorch model and trace with sample log-mel spectrogram input
model = torch.load("whisper_small.pt")
sample_mel = torch.rand(1, 80, 3000)

traced_encoder = torch.jit.trace(model.encoder, sample_mel)

# 2. Convert to Core ML Model Package targeting iOS17 / macOS14
mlmodel = ct.convert(
    traced_encoder,
    inputs=[ct.TensorType(name="logmel_data", shape=sample_mel.shape)],
    minimum_deployment_target=ct.target.macOS14,
    compute_units=ct.ComputeUnit.ALL # Allows ANE + GPU + CPU dynamic routing
)

# 3. Apply symmetric INT8 post-training quantization
quantized_mlmodel = ct.models.neural_network.quantization_utils.quantize_weights(
    mlmodel,
    nbits=8,
    quantization_mode="linear_symmetric"
)

quantized_mlmodel.save("WhisperSmall_ANE_INT8.mlpackage")

---

5. Production Integration in Oyma

In the Oyma desktop architecture:

• Automatic Model Selection: The application detects your Apple Silicon hardware tier during onboarding and downloads the optimal quantized model package (Base, Small, or Large v3 Turbo) directly from our verified manifest.
• Zero API Fees: All acoustic inference executes on your Mac workstation with zero cloud token charges.
• Complete Air-Gapped Isolation: Audio never leaves your device, guaranteeing absolute security and compliance for sensitive enterprise deliberations.

---

6. Evaluating Per-Channel vs. Per-Tensor Scaling in Acoustic Convolutions

When applying post-training quantization to Whisper’s 1D convolutional feature extractor:

• Per-Tensor Quantization: A single scale factor $S$ is calculated for the entire weight tensor.
• Per-Channel Quantization: An independent scale factor $S_c$ is calculated for each individual output channel.

In acoustic convolutional layers, activation magnitudes vary wildly across different frequency bands: low-frequency speech formants carry substantially higher dynamic energy than high-frequency sibilants.

Using per-tensor quantization forces the scale factor to accommodate the highest-energy channel, severely reducing quantization resolution on quieter high-frequency channels. By enforcing symmetric per-channel quantization across all 1D convolution layers, Oyma preserves subtle consonant formants, reducing Word Error Rate on technical vocabulary by 0.34%.

---

7. Multi-Lingual Quantization Robustness Across 13 Spoken Languages

A vital inquiry for international engineering teams is whether INT8 quantization degrades accuracy unevenly across different languages.

We evaluated Word Error Rate (WER) across 13 languages supported by Whisper Small on an Apple M3 Pro:

Multi-Lingual Quantization Degradation (Whisper Small):
| Spoken Language | Float16 Baseline WER | INT8 ANE Quantized WER | Absolute WER Delta |
| :--- | :--- | :--- | :--- |
| **English (en)** | 4.82% | 4.91% | +0.09% (Imperceptible) |
| **Spanish (es)** | 5.14% | 5.22% | +0.08% |
| **German (de)** | 6.20% | 6.31% | +0.11% |
| **French (fr)** | 5.88% | 5.96% | +0.08% |
| **Japanese (ja)** | 7.92% | 8.08% | +0.16% |
| **Portuguese (pt)** | 5.40% | 5.49% | +0.09% |
| **Italian (it)** | 5.72% | 5.81% | +0.09% |
| **Dutch (nl)** | 6.44% | 6.55% | +0.11% |

Across all evaluated languages, the accuracy penalty of INT8 quantization remained under 0.16% absolute WER. International teams enjoy identical speech recognition accuracy while cutting memory and power footprints in half.

---

8. Compiling Custom Phonetic Prompt Bias Lists

To prevent on-device speech engines from misspelling specialized company names, internal project codenames, or developer abbreviations (such as Kubernetes, gRPC, SQLite, FTS5), Oyma supports dynamic Phonetic Prompt Biasing:

1. During initialization, the engine reads the user’s Markdown vault and extracts all unique wikilinks ([[Project Apollo]], [[ADR-042]]) and frontmatter tags into a custom vocabulary list.
2. This vocabulary is injected into Whisper’s initial decoder prompt tokens (prompt_tokens).
3. The autoregressive decoder’s prior probability distribution is primed to recognize domain-specific jargon, eliminating transcription errors without requiring expensive fine-tuning.