The maximum quantity of tokenized information (prompt plus response) that a large language model can hold in active memory and process during a single inference cycle.
The context window (or context length) of a large language model is the maximum volume of sequential text tokens the model can analyze simultaneously. It defines the temporal and informational horizon of the model’s attention mechanism: anything outside the context window is completely invisible to the neural network.
When generative AI first entered mainstream awareness in 2020 with GPT-3, context windows were restricted to 2,048 tokens (roughly 1,500 words). Today, modern models feature windows ranging from 8,192 tokens up to hundreds of thousands or millions of tokens.
The quadratic attention bottleneck
The physical limitation on context windows stems from the foundational mathematical formulation of Transformer self-attention:
$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$
For an input sequence of length $N$ tokens:
- The query-key matrix multiplication $QK^T$ produces an $N \times N$ attention matrix.
- Computational time and RAM memory footprint scale as $\mathcal{O}(N^2)$.
- Processing a 2,000-token prompt requires computing 4,000,000 attention values; processing a 32,000-token prompt requires computing 1,024,000,000 attention values.
While modern algorithmic breakthroughs (such as FlashAttention, Rotary Position Embeddings (RoPE), and sliding window attention) have flattened this memory curve, long context windows still demand substantial RAM bandwidth.
Context lengths across modern local models
| Model | Context Window | Equivalent Words | Typical Local RAM Required |
|---|---|---|---|
| Llama 3 8B | 8,192 tokens | ~6,000 words | ~6 GB (4-bit quant) |
| Mistral 7B v0.3 | 32,768 tokens | ~24,000 words | ~6.5 GB (4-bit quant) |
| Llama 3.1 8B | 128,000 tokens | ~96,000 words | ~9 GB (with large KV cache) |
| Gemma 2 9B | 8,192 tokens | ~6,000 words | ~7.5 GB (4-bit quant) |
Managing context windows in Oyma
In meeting recording, a 60-minute discussion typically generates between 7,000 and 10,000 words of spoken dialogue (~13,000 tokens).
Oyma handles long meeting recordings on macOS with elegance:
- Short calls are fed directly into local model context windows.
- Extended multi-hour workshops are processed through hierarchical chunking: individual agenda topics are transcribed and distilled into structured sections, which are then assembled into an executive summary.
- The entire process executes locally in unified memory without overflowing RAM or dropping crucial conversation details.
Frequently asked questions about Context Window in Language Models
What is a token in the context of LLM windows?
A token is a subword fragment of text. In English, one token corresponds to approximately 4 characters or 0.75 words. A 1,000-word transcript represents approximately 1,330 tokens.
Why can't context windows be infinitely large?
Standard Transformer self-attention scales quadratically with sequence length: $O(N^2)$ in compute and memory. Doubling context length quadruples attention memory requirements.
How does Oyma process multi-hour meetings within local context limits?
Oyma segments extended meeting transcripts into semantic sections (e.g., using timestamps or agenda items), summarizing each section before synthesizing an overall executive record.
Master context window in language models with private Mac notes.
Store your thoughts in standard Markdown, transcribe calls on-device, and retain 100% data sovereignty.