Unified memory, explained: why a 128 GB Mac Studio runs models a 24 GB GPU can't

Ask anyone who runs language models locally what holds them back and the answer is rarely "compute". It's memory. A 70-billion-parameter model is too big for the graphics card in most workstations, however fast that card is. A Mac Studio with 128 GB of unified memory runs it comfortably.

This post explains why: what unified memory is, why capacity decides what fits, why bandwidth then decides how fast it runs, and where the context window's memory goes.

The memory wall#

To generate text, a model's weights have to sit in memory the GPU can read quickly. On a PC, that means the graphics card's own memory (VRAM): 24 GB on an RTX 4090, 32 GB on an RTX 5090.

Model weights take a predictable amount of space: the number of parameters times the bytes per parameter.

The 8-bit and 4-bit figures include the small overhead of quantisation scales, about half a bit per weight with MLX's default settings.

A 24 GB card holds an 8B model at any precision and a 32B model at 4 bits, just. A 70B model at 4 bits — about 40 GB — doesn't fit. You can split it between the card and system RAM, but then every generated token waits for data crossing the PCIe bus at roughly 32 GB/s (PCIe 4.0 x16), and speed collapses to a token or two per second.

What unified memory changes#

Apple silicon doesn't have separate CPU and GPU memory. The CPU, the GPU and the Neural Engine share one pool of memory in the same package as the chip. There's nothing to copy between "system" and "video" memory, because they're the same memory.

On a Mac Studio with an M5 Max and 128 GB, the GPU can use most of those 128 GB. By default macOS lets the GPU use roughly three quarters of the memory on large-memory Macs — about 96 GB here — and keeps the rest for the operating system and other programs. That's four times a 24 GB card, enough for the 70B model above with room to spare.

That's the headline: unified memory turns "does it fit on the GPU?" into "does it fit in the computer?"

Capacity decides what fits, bandwidth decides how fast#

Once a model fits, a second number takes over: memory bandwidth, how many bytes per second the GPU can read.

Generating text happens one token at a time, and for a dense model each new token needs every weight: the GPU reads the whole model from memory for every token. So the speed limit for generation is simple:

tokens per second ≤ memory bandwidth ÷ bytes read per token

For a 70B model at 4 bits (40 GB) on the M5 Max's 614 GB/s, that ceiling is about 15 tokens per second. Real-world speed lands below the ceiling — expect something like two-thirds of it — but the rule explains most of what you'll see.

Notice that the graphics card has the most bandwidth. For a small model that fits in its memory, it's faster than the Mac: an 8B model at 4 bits has a ceiling of about 220 tokens per second on the 4090 against about 130 on the M5 Max. Unified memory isn't magic bandwidth; it's capacity at good bandwidth. It wins when the model is too big for the card.

Prompt processing is different#

Before generating, the model reads your whole prompt. That step, called prefill, processes all the prompt's tokens at once, so it's limited by compute rather than bandwidth. Discrete NVIDIA GPUs have far more raw compute than a Mac's GPU, so long prompts — a whole document, a big code file — start answering faster on them. The M5 generation narrows the gap: its GPU cores include Neural Accelerators for the matrix maths that prefill is made of. Still, if your workload is mostly very long prompts with short answers, weigh compute as well as memory.

The context window needs memory too#

Weights aren't the only thing in memory. As the model reads and writes text, it keeps a KV cache: for every token in the context, a key and a value vector for each layer. Its size per token is:

2 × layers × KV heads × head size × bytes per value

Llama 3.3 70B has 80 layers, 8 KV heads of size 128, and stores the cache in 16-bit values:

2 × 80 × 8 × 128 × 2 bytes = 320 KB per token

So a 32,000-token context takes about 10 GB, and the full 128,000-token context about 40 GB — as much as the weights themselves. Smaller models need less: Qwen3-8B's cache is about 144 KB per token, or 4.5 GB for 32,000 tokens.

This is where the extra headroom of a big unified memory pays off twice: the model fits, and it fits with a long context.

A 128 GB budget, worked through#

Here's how the M5 Max's memory might be spent running Llama 3.3 70B with a long context:

That leftover could hold a second, smaller model — say an 8B model for quick drafting or embeddings — or a longer context.

Mixture-of-experts models love unified memory#

Many recent models are mixture-of-experts (MoE): they have lots of parameters, but each token only uses a few of them. Qwen3-30B-A3B has 30 billion parameters but activates about 3 billion per token.

MoE models need the capacity for all their parameters — 17 GB for Qwen3-30B-A3B at 4 bits — but generate as fast as a much smaller model, because each token reads only the active experts. That combination, big but cheap to run, is exactly what a large unified memory is good at.

When unified memory isn't the answer#

• Training large models from scratch needs far more compute than any desktop machine has.
• CUDA-only code won't run on a Mac. If your stack depends on CUDA kernels, look at the DGX Spark, which pairs a Blackwell GPU with 128 GB of unified memory.
• High-throughput serving of many users at once is compute-bound, and data-centre GPUs do it better.

Try it without buying one#

A Mac Studio with 128 GB is a serious purchase to make on a hunch. You can rent a dedicated M5 Max by the day from the catalog and see how your model behaves before deciding — rent or buy? runs the numbers. When you've got one, Running Qwen3 and Llama with MLX gets you from a fresh machine to a working model server.