Instance types

An instance type is a hardware configuration: a chip, an amount of unified memory, its bandwidth and the storage. A different memory or storage size is a different type. An instance is one physical machine of a type. When you reserve, you get one instance of the plan's type for your whole reservation.

The machines#

The catalog always has the current list, with full specs, prices and the next free day.

Unified memory#

On a typical PC, a graphics card has its own memory — often 16 or 24 GB — and everything the GPU works on must be copied into it. On these machines the CPU and GPU share one large pool. A model's weights load once and the GPU uses them directly, so the question "does it fit?" is answered by the whole memory size, not a graphics card's.

Two numbers matter most for running large language models:

• Capacity decides what fits: the model's weights plus the KV cache for your context length, plus room for the operating system.
• Bandwidth decides how fast it runs: generating each token reads the model's weights from memory, so tokens per second scale with memory bandwidth.

Our blog post Unified memory, explained goes into more depth, and M5 Max vs M5 Ultra shows which models fit where.

Mac Studio or DGX Spark?#

• Mac Studio runs macOS with Apple's MLX framework and Metal. It has the highest memory bandwidth of the machines here, which makes it fast at generating tokens. See Mac Studio environment.
• DGX Spark runs DGX OS (Ubuntu) with CUDA, Docker and NVIDIA's software stack, so CUDA code and containers run unchanged. See DGX Spark environment.

Mac Studio vs DGX Spark compares them in detail.