Running Qwen3 and Llama with MLX on a Mac Studio M5 Max

MLX is Apple's machine-learning framework for Apple silicon. It's built around unified memory: arrays live in memory the CPU and GPU share, so there are no copies between them. On top of it, mlx-lm loads, runs, quantises and serves language models with a handful of commands.

This guide goes from a fresh Mac Studio M5 Max with 128 GB — for example one reserved by the day — to a model answering questions over an OpenAI-compatible API. Everything runs in the terminal over SSH.

1. Install mlx-lm#

Use a virtual environment, so everything lives in your home directory:

python3 -m venv ~/.venvs/mlx
source ~/.venvs/mlx/bin/activate
pip install --upgrade pip mlx-lm

Check that MLX sees the GPU:

python -c "import mlx.core as mx; print(mx.default_device())"

It should print Device(gpu, 0).

2. Choose a model#

The mlx-community organisation on Hugging Face publishes ready-converted MLX versions of popular models. The suffix tells you the precision: -4bit, -8bit or -bf16.

On a 128 GB machine, the GPU can use roughly 96 GB by default, so every model in this table fits, most of them several times over. M5 Max vs M5 Ultra explains how to size bigger ones.

Llama models are released under Meta's licence. Some Llama repositories on Hugging Face are gated: you accept the licence on the model's page and download with an access token (hf auth login). Qwen3 is Apache 2.0 and ungated.

3. Download#

mlx-lm downloads a model the first time you use it, into ~/.cache/huggingface. To download ahead of time — in a startup script, for example — use the Hugging Face CLI:

pip install "huggingface_hub[cli]"
hf download mlx-community/Qwen3-32B-4bit

Download on the machine itself rather than through your laptop: the weights make one trip instead of two, and your own connection stays free.

4. Generate#

mlx_lm.generate \
  --model mlx-community/Qwen3-8B-4bit \
  --prompt "Explain in three sentences why memory bandwidth limits LLM speed." \
  --max-tokens 400

After the answer, mlx_lm.generate prints the prompt and generation speeds in tokens per second and the peak memory used. Those are the numbers to compare when trying models.

Qwen3 models think out loud before answering, inside <think> tags. That helps with hard questions but costs time; add /no_think to the end of your prompt to skip it for simple ones.

5. Chat#

For an interactive session in the terminal:

mlx_lm.chat --model mlx-community/Qwen3-32B-4bit

The model stays loaded between turns, and the conversation's KV cache is reused, so follow-up questions start answering quickly.

6. Serve an OpenAI-compatible API#

mlx_lm.server exposes the model over HTTP with the OpenAI chat completions API, so existing clients and tools work with it:

mlx_lm.server --model mlx-community/Qwen3-32B-4bit --port 8080

It listens on the machine only. From your own computer, forward the port over SSH:

ssh -N -L 8080:localhost:8080 user@my-llm-box.res-4ad0a522.ssh.gputolease.ai

Then call it as if it were local:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [{"role": "user", "content": "Write a haiku about unified memory."}],
    "max_tokens": 200
  }'

Any OpenAI SDK works too: point its base URL at http://localhost:8080/v1. Run the server inside tmux or screen so it keeps going when you disconnect.

7. Use it from Python#

For your own scripts and evaluations, the Python API is two calls:

from mlx_lm import load, generate

model, tokenizer = load("mlx-community/Qwen3-8B-4bit")

messages = [{"role": "user", "content": "List three uses of a 128 GB Mac Studio."}]
prompt = tokenizer.apply_chat_template(messages, add_generation_prompt=True)

text = generate(model, tokenizer, prompt=prompt, max_tokens=300, verbose=True)

verbose=True streams the text and prints the same speed statistics as the command line.

Quantisation: which precision?#

Quantisation stores weights in fewer bits. It saves memory and, because generation reads every weight for every token, it also makes generation faster.

• bf16 is the original precision: best quality, twice the size of 8-bit and four times 4-bit.
• 8-bit is very close to the original for most tasks. A good default when the model fits comfortably.
• 4-bit roughly halves the memory again with a small loss in quality, often hard to notice in chat but sometimes visible in maths and code. It's what makes 70B models practical.

You can quantise a model yourself from its original Hugging Face repository:

mlx_lm.convert --hf-path Qwen/Qwen3-14B -q --q-bits 4 --mlx-path ~/models/qwen3-14b-4bit

--q-bits 8 gives 8-bit, and --q-group-size trades a little size for accuracy (smaller groups are more accurate).

A useful habit: evaluate your task at 8-bit first, then check whether 4-bit holds up. If it does, you get the speed for free.

What speed to expect#

Generation speed is capped by memory bandwidth divided by the bytes read per token. On the M5 Max's 614 GB/s, that gives these ceilings; real speeds typically land around 60–75% of them.

Estimates from the bandwidth arithmetic, not benchmarks. Measure your own with mlx_lm.generate.

Mixture-of-experts models break the pattern in your favour. Qwen3-30B-A3B reads only its roughly 3 billion active parameters per token, so despite needing 17 GB of memory it generates at speeds closer to the dense 8B model than to the 32B one.

Prompt processing (prefill) is limited by compute rather than bandwidth and runs much faster than generation, typically hundreds of tokens per second or more depending on the model's size. It matters when you paste in long documents.

Tips for long contexts#

• The KV cache grows with the context: about 320 KB per token for Llama 3.3 70B, so a 32,000-token conversation adds about 10 GB. Watch the peak memory figure.
• mlx_lm.generate --max-kv-size 8192 caps the cache for long generations, trading away the oldest context.
• Close models you're not using: each loaded server holds its weights in memory.

Make it repeatable#

Each reservation starts from a clean machine, so put the setup in a startup script: create the virtual environment, install mlx-lm and download your model. Our MLX download template does exactly that for Qwen3-8B; change one line to fetch your model instead. When you log in, the model is waiting.