---
url: https://lettuceai.app/docs/multi-gpu
title: "Multi-GPU Local Models — LettuceAI"
description: "Split local llama.cpp models across multiple GPUs with automatic layer distribution, KV cache placement, and per-scenario tuning."
---

Menu 

# Multi-GPU Local Models

On desktop, the built-in llama.cpp engine can split a local model across two or more graphics cards. The app decides how many layers each GPU gets, keeps enough VRAM in reserve for the context, and lets you choose where the KV cache lives. Models that are too big for any single card can run fully on GPU instead of spilling to the CPU.

Requirements

Multi-GPU needs **two or more discrete GPUs** visible to a Vulkan or CUDA backend. Integrated GPUs are never used for multi-GPU, and the toggle stays disabled until at least two eligible cards are detected. For single-GPU offload basics, see the [Model Browser & Local Inference](/docs/model-browser) page.

## Where the settings live

There are two places, and they layer on top of each other:

-   **Settings → Models → your model → Performance**: the full per-model control set. Multi-GPU here can be set to **Inherit**, **Enabled**, or **Disabled**. Inherit means the global default applies.
-   **Local Runtime Defaults** (in Settings): the global fallback. It carries the multi-GPU switch, device selection, distribution mode, KV placement, main GPU, and the priority VRAM limit. Manual per-GPU layer counts and the single-GPU device override exist only in the model editor.

## Turning it on

1.  Open **Settings → Models**, pick a llama.cpp model, and go to the **Performance** section
2.  Set **Multi-GPU** to Enabled (or enable it once in Local Runtime Defaults and leave models on Inherit)
3.  In **GPU Devices**, tick at least two cards. Each entry shows its backend and free/total VRAM.
4.  Pick a **Distribution** mode and a **KV cache** placement (both explained below)

Below the device list the editor shows an estimated placement, e.g. "Estimated: 28 + 20 · 48 layers on GPU", so you can see the planned split before loading anything.

## How the split works

The engine always splits by **whole layers**: each GPU holds a contiguous slice of the model's transformer layers, and tokens flow through the cards in sequence. This keeps PCIe traffic low compared to splitting individual tensors across cards. Each layer's slice of the KV cache lives on the same GPU as that layer.

### Distribution modes

| Mode | What it does | Best for |
| --- | --- | --- |
| Balanced | Even split across the selected GPUs. The default. | Identical cards. |
| Proportional to VRAM | Each GPU's share is weighted by its free VRAM, and the total is capped by what the cards can actually hold. | Mismatched cards. |
| Priority fill | Fills the first selected GPU up to an optional VRAM limit, then overflows to the next. The first GPU becomes the main device. | Keeping most of the model on your fastest card, or capping how much VRAM the primary card gives up. |
| Manual per-GPU | You type the exact layer count for each GPU. The editor validates the sum against the model's total layer count. | Full control, unusual setups, working around a flaky card. |

Manual mode has no safety net

Automatic modes back off and retry with fewer layers if a load runs out of memory. Manual counts are used as-is: if you over-assign a card, the load fails and falls back to CPU. The editor warns before loading when your assignment exceeds a card's reported VRAM.

### KV cache placement

The KV cache is the memory that holds your conversation context, and it grows with context length. Where it lives is a speed/VRAM trade-off:

| Placement | What it does |
| --- | --- |
| Auto | Let the planner decide. KV stays on GPU when it fits, otherwise it moves to system RAM. |
| Split with layers | KV lives on the GPUs, divided the same way as the layers. Fastest option, uses VRAM. |
| System RAM | Keeps the KV cache in system RAM to free VRAM for more layers. Attention runs slower. |
| Prefer main GPU | Routes shared scratch buffers to one chosen main GPU. Each layer's KV still stays on that layer's card. Shows a **Main GPU** picker when selected. |

## The smart offloader

You normally never set a layer count yourself. When GPU layers are on Auto, the planner reads the model's real metadata (size, layer count, attention shape) and works out how many layers fit:

-   It budgets **90 percent** of usable VRAM and reserves extra for runtime scratch buffers, so loads do not sit at the edge of an out-of-memory failure
-   KV cache cost is priced **per layer** using your context length and KV quantization, so a huge context correctly shrinks the number of layers that fit
-   In multi-GPU mode the budget is the **combined capacity** of all selected cards, and the per-device shares respect each card's own limit
-   If a load still fails, automatic modes retry down a ladder of smaller layer counts before falling back to CPU. If the weights fit but the context does not, the engine retries once at a KV-aware layer count instead of pushing the whole cache to RAM.

Successful results are remembered per model and reused on the next load, but only while the configuration fingerprint matches: changing devices, distribution mode, KV placement, context bucket, or seeing the available VRAM shift by more than 5 percent forces a fresh estimate. Estimates are also recomputed once per app session.

## Single-GPU device override

Separate from multi-GPU, each model has a **GPU device** picker that pins it to one specific card instead of the default device. Useful for running a small model on your second GPU while the first one stays free for games or another model.

Override wins

If a single-GPU device override is set, multi-GPU is ignored for that model. Enabling multi-GPU in the editor clears the override automatically.

## Recommended configurations

| Scenario | Recommended setup |
| --- | --- |
| Two identical cards (e.g. 2 × RTX 3090) | Balanced distribution, KV cache Split with layers. Leave everything else on Auto. |
| Mismatched cards (e.g. 24 GB + 8 GB) | Proportional to VRAM so the bigger card carries most of the model. KV on Auto or Split. |
| One fast card plus one slow card | Priority fill with the fast card first and no VRAM limit. Only the layers that do not fit overflow to the slow card. |
| Primary card also drives your display | Priority fill with a VRAM limit on the first GPU, sized to leave a few GB for the desktop and browser. Or untick the display card entirely. |
| Long conversations, VRAM is tight | KV cache in System RAM to spend all VRAM on layers, and consider a quantized KV type (q8 or q4) on the model's Runtime tab. Expect slower prompt processing. |
| Small model alongside a game or a second model | Skip multi-GPU. Use the single-GPU device override to pin the model to the card you are not gaming on. |
| A card misbehaves or you know better than the planner | Manual per-GPU with explicit counts. Watch the placement summary and keep each card under its reported free VRAM. |

## Good to know

-   KV placement, main GPU, the priority limit, and manual layers only take effect while multi-GPU is actually active (enabled, two or more devices selected, no single-GPU override).
-   On AMD ROCm backends the KV cache is kept in system RAM and flash attention is disabled; the KV placement choice is not honored there.
-   If the pinned Main GPU is not among the selected devices, the pin is ignored and the distribution default applies.
-   KV quantization and context length both change how many layers fit. If the estimate looks low, check those two first.
-   After a load, the model's runtime report shows the requested vs actual layers per device and whether any fallback activated, which is the first place to look when performance seems off.

[

PreviousModel Browser & Local Inference

](/docs/model-browser)[

NextOllama

](/docs/ollama)