---
url: https://lettuceai.app/docs/model-browser
title: "Model Browser & Local Inference — LettuceAI"
description: "Browse, download, and run GGUF models locally with the built-in HuggingFace model browser and llama.cpp engine in LettuceAI."
---

Menu 

# Model Browser & Local Inference

LettuceAI can run AI models directly on your device using the built-in **llama.cpp** engine. Browse and download GGUF models from HuggingFace, load them locally, and chat completely offline. No cloud API key required.

What you need

Local inference requires a device with enough RAM (and ideally VRAM) to load the model. Smaller quantized models (Q4, Q5) can run on most modern desktops. Larger models or higher quantizations need more memory.

## Model Browser

The built-in model browser queries the HuggingFace public API with the `filter=gguf` tag, so every result is a GGUF repo you can download directly without leaving the app.

### Finding models

Open the browser from **Settings → Models → HuggingFace Browser**. You can search by name and sort results by:

-   **Trending** (default): HuggingFace trending score
-   **Downloads**: most downloaded overall
-   **Likes**: community favorites
-   **Last modified**: recently updated repos

Each result shows the author, download and like counts, tags, pipeline tag, last-modified date, parameter size (parsed from tags or model ID), and a thumbnail when available.

### Filters and views

The filter button (next to sort) lets you narrow results client-side by:

-   **Pipeline tag**: filter to a HuggingFace pipeline tag present in the current results (for example, `text-generation`, `image-text-to-text`).
-   **Parameter range**: minimum and maximum parameter count in billions, parsed from the repo tags or model ID.

You can also cycle the result layout between **list**, **grid**, and **gallery** views. The choice is stored in your browser.

### Choosing a file

Most HuggingFace GGUF repos contain multiple files at different quantization levels. The browser lists every `.gguf` file in the repo (excluding imatrix calibration data), parses the quantization tag from the filename, and detects `mmproj` (vision projector) files automatically.

Quantizations you may see, ordered roughly by quality:

| Quantization | Notes |
| --- | --- |
| F32 / BF16 / F16 | Full precision, largest files. |
| Q8\_0, Q8\_K (S/L/XL), Q6\_K | Near-original quality. |
| Q5\_K (S/M/L/XL), Q5\_0, Q5\_1 | Good quality at moderate size. |
| Q4\_K (S/M/L/XL), Q4\_0, Q4\_1, IQ4\_XS, IQ4\_NL | Common sweet spot for size vs quality. |
| Q3\_K (S/M/L), IQ3\_M, IQ3\_S, IQ3\_XS, IQ3\_XXS | Smaller, noticeable quality loss. |
| Q2\_K, IQ2\_\* | Aggressive compression. |
| IQ1\_S, IQ1\_M, MXFP4\_MOE | Specialty quants. |

### Runability scoring

For each GGUF file the browser computes a runability score from 0 to 100 using your device's available RAM and (when present) VRAM, plus GGUF metadata parsed directly from the file header (architecture, block count, embedding length, head counts, sliding window, KV LoRA rank).

![How the app estimates whether a local model will run on your device](https://lhdgeo5fms.ufs.sh/f/m0TBUtMLsaiEoykKp8xgeUAy1BckbI0oS8FO2ihTxv7GqEl9)

The app weighs the model's size and quantization against your memory and graphics card, works out how much can run on the GPU, and shows a runnability score before you download. If a model runs out of memory while loading, it retries at a smaller size before falling back to the CPU.

The final score is a weighted sum:

-   **Memory fitness** (25%): how much total memory is available relative to the model, KV cache, and compute overhead.
-   **GPU acceleration** (35%): whether the model fits fully in VRAM, partially (with KV or weight spill), or only via partial layer offload.
-   **KV cache headroom** (15%): free memory left for the KV cache after loading weights.
-   **Quantization quality** (25%): a per-quant quality score (for example F16 = 100, Q8\_0 = 90, Q5\_K\_M = 85, Q4\_K\_M = 75, Q3\_K\_S = 50, Q2\_K = 35, IQ1\_S = 15).

The numeric score maps to a fitness label:

-   **Excellent**: 80 to 100
-   **Good**: 60 to 79
-   **Marginal**: 40 to 59
-   **Poor**: 20 to 39
-   **Unrunnable**: below 20 (or the model does not fit at all, which caps the score at 10)

Architecture-aware adjustments are applied automatically: Gemma 2 and Cohere are scored with their sliding-window KV cap, and DeepSeek V2/V3 use the MLA (multi-head latent attention) compressed KV size. Apple Silicon and other unified-memory devices use the maximum of RAM and VRAM rather than the sum.

#### Mixture-of-Experts (MoE) awareness

Large MoE models (for example Mixtral or Qwen MoE) only run a small fraction of their weights for each token. The scorer detects MoE models from their GGUF metadata and estimates the **active weight ratio**: roughly the share of weights actually used per token (attention plus the experts that fire). That active size, not the full on-disk size, drives the compute and memory overhead estimate. In practice this means a big sparse MoE can score much better than its file size alone would suggest.

#### Partial GPU offload

The GPU acceleration score is not all-or-nothing. The browser estimates one of several outcomes for your hardware and scores accordingly:

-   **Full GPU**: weights and KV cache both fit in VRAM (highest score).
-   **Mostly GPU with spill**: weights fit but the KV cache or compute buffers spill into system RAM.
-   **Model in RAM, context on GPU** (or the reverse), for mixed setups.
-   **Partial layer offload**: only some transformer layers fit in VRAM and the rest run on CPU, scaled by how much of the model fits.

Budgets reserve roughly 10 percent of free RAM and VRAM as headroom so the estimate is conservative rather than optimistic.

#### Quantization and KV cache quality

Each quant type has a quality score (for example F16 = 100, Q8\_0 = 90, Q5\_K\_M = 85, Q4\_K\_M = 75, Q2\_K = 35, IQ1\_S = 15). Unsloth dynamic (`UD-`) quants and quantization-aware-trained (QAT) files are recognized and scored higher, because they hold up better at the same size.

When the browser recommends settings for a file, it sizes the context against the cost of different KV cache types and never recommends the full-precision `f16` KV cache: `q8_0` is the highest KV type it will suggest, since it saves memory at almost no quality cost.

### Downloading

Tap a file to add it to the download queue. The queue processes items sequentially through a single worker, writing to a `.tmp` file and renaming on completion. Downloads emit progress, byte counts, and a sampled speed (bytes per second).

Each queued item is in one of these states:

-   **Queued**: waiting for the worker
-   **Downloading**: bytes streaming with live progress
-   **Complete**: file moved to its final path
-   **Error / Cancelled**: failed or stopped by the user

Restarts on retry

Downloads are not resumed across cancellation or app restart. If a download is cancelled the temporary file is removed, and the next attempt starts from zero. If the destination file already exists and is at least 1 MB, the queue skips re-downloading unless you choose force redownload.

Downloaded GGUF files are stored under the app's models directory in a per-repo folder (`owner--repo/filename.gguf`). When a download finishes the browser shows a quick action to create a model profile pointing at the new file.

### Vision model files

Multimodal models ship an extra `mmproj` (multimodal projection) file alongside the main GGUF. The browser detects these by filename and lists them separately. Download both the main GGUF and the mmproj, then point the model profile's **MMPROJ Path** at the mmproj file and enable image input.

## Pull to a Remote Ollama Server

The browser can also push a HuggingFace GGUF directly to a remote [Ollama](/docs/ollama) instance instead of downloading it to this device. The file lands on the Ollama host (not in the app's models directory), and is served from there through the Ollama provider.

### Choosing the destination

The model page has a destination picker with two sections: **On this device** (the default, labelled _Local library_) and **Remote Ollama**, which lists every provider credential configured with the `ollama` provider. Selecting one of those Ollama credentials switches the browser into **Ollama mode**. If you have no Ollama providers configured yet, the picker links you to **Settings → Providers** to add one.

### What changes in Ollama mode

-   The **Recommended settings** tab is hidden and a notice is shown in its place. Hardware-aware runability scoring is disabled, because the model will run on the Ollama host's hardware, which the app cannot inspect.
-   The files panel still lists every `.gguf` file in the repo, and tapping a file queues it. The local destination path, mmproj auto-creation, and create-model-on-finish flows do not apply.

### How the model reference is built

When you queue a file in Ollama mode, the app constructs a HuggingFace model reference of the form:

`hf.co/<owner>/<repo>:<quantization>`

The quantization tag is parsed from the chosen GGUF filename (for example `Q4_K_M`). If the file has no detectable quantization tag, the colon and tag are omitted and Ollama pulls the repo's default. This is the same `hf.co/...` ref form that `ollama pull` understands natively, so the remote server resolves and fetches the GGUF itself.

### How the pull runs

The pull is dispatched through the `ollama_pull_model` Tauri command. It posts to the credential's `api/pull` endpoint with `stream: true`, and the streamed status, completed, and total byte counts are forwarded into the regular download queue as a synthetic item (the queue kind is set to `ollama`).

That means an Ollama pull appears in the same queue panel as local GGUF downloads, with live progress, a sampled speed, and a cancel control. Cancelling sends a cancellation signal to the streaming pull; the partial state of the model on the Ollama host is up to Ollama itself.

The file is not on your device

A successful Ollama pull does not produce a file in the app's models directory. There is nothing to point a llama.cpp profile at. To use the pulled model, chat against it through the Ollama provider you pulled it to (see the [Ollama provider doc](/docs/ollama)).

Resolution is up to Ollama

Ollama resolves the `hf.co/owner/repo:quant` reference on the host. Repos that Ollama cannot resolve (for example, some sharded multi-file GGUFs, or files behind gated access without matching credentials on the host) will fail at the remote end. The app surfaces the error returned by the Ollama server.

## Installed Models

The **Installed Models** page lists every GGUF file the app has indexed in its models directory. It opens each file briefly to read GGUF metadata and shows:

-   Filename, model ID, and absolute file path
-   Architecture and parameter size
-   Context length (if the GGUF advertises one)
-   File size
-   Whether the file is an LLM or an mmproj projector

You can sort by parameters, architecture, context length, file size, or filename, and search by name or architecture. From each row you can copy the file path, delete the file, or refresh the inventory.

## llama.cpp (Local Engine)

The local engine wraps the llama.cpp C++ library via Rust bindings (no separate server process). It runs entirely on desktop builds; the mobile builds of LettuceAI do not currently ship a local inference engine.

### Setting up a local model

1.  Download a GGUF model from the browser.
2.  Go to **Settings → Models → Add Model**.
3.  Choose **Local (llama.cpp)** as the provider and pick your GGUF file.
4.  Adjust advanced settings if needed, or apply one of the quick presets (see below).

The first load may take a moment while weights are mapped, layers are allocated, and the context is built. Subsequent loads of the same file are usually faster.

### Chat templates

Chat templates format your messages before they reach the model. The engine resolves the template in this order on every request:

1.  **Override**: if the model profile has a non-empty **Chat Template Override** (a raw Jinja template), it is used.
2.  **Embedded GGUF**: otherwise, if the GGUF file has an embedded chat template, that is used.
3.  **Preset name**: only if both of the above are missing, the engine tries the **Chat Template Preset** (a name passed through to llama.cpp, for example `chatml`).

The preset dropdown in the model profile ships these named options:

-   `auto`: do not send a preset hint (relies on the embedded template).
-   `chatml`, `llama2`, `llama3`, `mistral-v1`, `vicuna`, `gemma`

Templates are applied through the OpenAI-compatible chat template path with `use_jinja = true`, so models that ship Jinja templates with tool-call support work out of the box.

### Sampler profiles

Sampler profiles set baseline temperature, top-p, top-k, min-p, typical-p, and penalty values. Any field you fill in on the profile overrides the corresponding default.

| Profile | Defaults |
| --- | --- |
| Balanced | temp 0.80, top-p 0.95, top-k 40, min-p 0.05, frequency penalty 0.15 |
| Creative | temp 0.95, top-p 0.98, top-k 80, min-p 0.02, presence penalty 0.25 |
| Stable | temp 0.55, top-p 0.90, top-k 32, min-p 0.08, typical-p 0.97, frequency penalty 0.20 |
| Reasoning | temp 0.35, top-p 0.90, top-k 24, typical-p 0.95, frequency penalty 0.10 |

### Sampler order

The engine builds a chained sampler in a configurable order. The default chain is:

`penalties, grammar, top_k, top_p, min_p, dry, typical, temp`

A final distribution stage is appended automatically (a random distribution sampler when temperature > 0, otherwise a greedy sampler). You can reorder the stages from the **Sampler Order** editor on the model profile. Unknown or duplicate stages are dropped, and any default stage you omit is appended at the end.

### Quick presets

The Performance section of the model profile exposes four single-tap presets that pre-fill batch size, KV cache type, KQV offload, and flash attention:

-   **Balanced**: batch 512, KV q8\_0, KQV on, flash attention auto
-   **Throughput**: batch 1024, KV f16, KQV on, flash attention enabled
-   **VRAM Saver**: batch 512, KV q4\_k, KQV on, flash attention enabled
-   **CPU + RAM**: batch 256, KV q8\_0, KQV off, flash attention auto

### GPU and compute settings

| Setting | Range | What it does |
| --- | --- | --- |
| Context Length | 0 to 262,144 | Maximum prompt + generation tokens kept in the KV cache. 0 falls back to the model default. |
| GPU Layers | 0 to 512 | Number of transformer layers offloaded to GPU. 0 runs entirely on CPU. Disabled on CPU-only backends. |
| Threads | 1 to 256 | CPU threads for token generation. |
| Batch Threads | 1 to 256 | CPU threads for prompt (batch) evaluation. |
| Batch Size | 1 to 8192 | Tokens processed per batch during prefill. Larger means faster prompt processing if memory allows. |
| Offload KQV | Auto / On / Off | Place the KV cache and KQV ops on GPU. Forced off on CPU-only backends. |
| Flash Attention | Auto / Enabled / Disabled | Memory-efficient attention kernel. Auto lets the engine decide per backend. |

### KV cache type

The KV cache stores keys and values for past tokens. A smaller cache type lowers VRAM use at a small quality cost. The model profile dropdown ships these choices:

-   `auto`: model default (recommended)
-   `f16`: full precision, highest VRAM
-   `q8_0` (recommended balance), `q8_1`
-   `q6_k`
-   `q5_k`, `q5_1`, `q5_0`
-   `q4_k`, `q4_1`, `q4_0`
-   `q3_k`
-   `q2_k`: maximum VRAM saving

### Full SWA Cache

Some models (for example Gemma) use sliding-window attention, where only a recent window of tokens is normally kept in the cache. The **Full SWA Cache** toggle (in the model's **Runtime** tab, under Layer Placement) keeps the entire attention window resident instead. This improves recall over long conversations, but at a high VRAM cost. Leave it off unless you have VRAM to spare. It is off by default.

### Faster generation with MTP (speculative decoding)

Multi-Token Prediction (MTP) is a speculative decoding feature that can speed up local generation. A small, fast "draft" predicts several tokens ahead and the main model verifies them in one pass, so more tokens come out per step when the guesses are right. There is no quality loss: any token the main model would not have produced is rejected.

Turn it on in the model's **Runtime** tab under **Multi-Token Prediction**. When enabled, two controls appear:

-   **Draft Tokens**: how many tokens the drafter proposes per step (1 to 8, default 4).
-   **MTP Draft File**: an optional external draft GGUF. Leave it on _Auto-discover_ to pick up a matching `mtp-*.gguf` sidecar next to your model, or use **Select from library** to choose a downloaded draft file.

The drafter is sourced automatically. If the model file already ships bundled MTP layers (so-called "nextn" layers), those are used directly and no separate file is needed. Otherwise the app looks for the external draft file described above. Gemma-style shared-assistant drafters are detected automatically. If MTP is enabled but no usable drafter is found, the model simply runs without it. MTP is also skipped automatically for requests that include images.

No draft model? No problem

MTP only helps when a compatible drafter is available. Many newer GGUF releases bundle the draft layers; others publish a small `mtp-*.gguf` sidecar in the same HuggingFace repo that you can download alongside the main file.

### Reasoning and thinking

For models that produce a hidden reasoning or "thinking" stream, LettuceAI separates that stream from the final answer automatically. It recognizes the common tag styles (`<think>`, `<thinking>`, `<reasoning>`) as well as Gemma channel-style tags, and no setup is required.

The model's **Reasoning** tab has a **Force send thinking state** toggle. Normally the app omits the thinking flag and lets the model decide. Some local models default to thinking when no value is sent; turning this on makes the app send the flag explicitly so reasoning behaves as expected. It is off by default.

### Automatic memory fallback

When a model nearly fits but the chosen context or GPU layers would run out of memory, the engine does not just fail. It retries with a smaller context or batch size, moves the KV cache from VRAM to system RAM, reduces how many layers are offloaded to the GPU, and as a last resort falls back to running on CPU. The goal is to get the model running instead of erroring out.

If you would rather the engine attempt exactly the settings you entered and fail otherwise, enable **Strict Mode** on the model. This bypasses the automatic fallbacks (lowering GPU layers, clamping context or batch, or switching to CPU). It is off by default and intended for advanced users tuning a known-good configuration.

### Sampling parameters

On top of the profile defaults, the model page exposes individual overrides for temperature, top-p, top-k, min-p, typical-p, frequency penalty, presence penalty, and the DRY repetition sampler (multiplier, base, allowed length, penalty last-N window, and sequence breakers). The DRY sampler is only active when its multiplier is > 0.

| Setting | Range | What it does |
| --- | --- | --- |
| Seed | 0 to 2,147,483,647 | Fixed seed for reproducible output. Empty means a random seed per run. |
| Min-P | 0 to 1 | Drops tokens whose probability is below this fraction of the top token's probability. |
| Typical-P | 0 to 1 | Locally typical sampling threshold. Keeps tokens close to the expected information content. |
| RoPE Freq Base | 0 to 1,000,000 | Base frequency for rotary embeddings. Used for context-length extension. |
| RoPE Freq Scale | 0 to 10 | Scale factor for RoPE. Values below 1 extend the effective context window. |

Advanced users only

RoPE settings extend context beyond what the model was trained for. Changing them without understanding the architecture can produce garbage output. Leave them empty (auto) unless you know what you are doing.

### Vision support

Multimodal models load through an mmproj projector. To enable vision on a llama.cpp profile:

1.  Download the matching `mmproj` file from the same HuggingFace repo.
2.  Set the model profile's **MMPROJ Path** to point at it.
3.  Enable **Image** in the model's input scopes.

Inline images in chat are decoded and normalized to PNG before being sent through the multimodal pipeline, so JPEG and other formats are accepted as input.

### Runtime report

After loading or running a model, LettuceAI stores a runtime report on the profile. It captures the values the engine actually used (which may differ from your requested settings if fallbacks kicked in), including:

-   Requested vs actual context length and batch size
-   Requested vs actual GPU layers offloaded
-   Actual KV cache type and KQV offload mode
-   Flash attention policy and backend path used
-   Whether smart GPU layer fallback, KQV fallback, or context fallback activated
-   Vision request vs vision actually active, plus image count
-   Tokens per second from the most recent generation

The report appears in the model's **Runtime** tab under **Last runtime report**, with a plain-language status such as "Run succeeded", "CPU fallback recovered", or "Run failed". If the engine had to fall back to a CPU-safe context to get the model running, an **Apply working config** button lets you save those recovered settings so future runs reuse them directly.

## Local Runtime Defaults & Model Storage

From **Settings → Models** you can open **Local Runtime Defaults** ("Engine setup and calculator baselines"). It holds settings that apply across all your local models rather than to one profile.

### Calculator baselines

The **llama.cpp Defaults** section sets the assumptions the runability calculator uses when it scores models in the browser:

-   **Default Context Length**: the context size assumed when scoring. Leave it empty to use 8192.
-   **Default KV Cache Quant**: the KV cache type assumed when scoring. `Auto` behaves like `f16`.

These only change how scores are estimated; they do not change how any individual model actually runs.

### Where models are stored

The **Storage** section shows the **LLM Models Folder**, the folder on this computer where downloaded GGUF files are kept. Use **Change** to pick a different folder (for example a larger drive), or **Reset to default** to go back.

When you choose a new folder, the app asks what to do with your existing models:

-   **Move them to the new folder** (recommended): your models are relocated and keep working automatically. The app also updates the saved paths so every model profile still points at the right file.
-   **Leave them where they are**: only new downloads use the new folder; existing files stay put.

Pick an empty folder

A move is done safely: files are copied first and only deleted from the old location once the copy succeeds. If the destination already contains a model with the same name, the move is cancelled and you are asked to pick an empty folder.

## Model Import & Export

A model profile (name, provider, and every advanced setting) can be exported and imported in two formats:

-   **JSON**: a plain object that other LettuceAI installs can read.
-   **USC** (Unified System Card): wraps the JSON payload with a schema header (`schema.name = "USC"`, `kind = "model_profile"`). The same USC wrapper is also used for lorebooks, chat templates, and prompt templates.

On import, the parser auto-detects whether the file is a plain JSON profile or a USC envelope and unwraps it. A new model profile is created from the payload.

Local files are not bundled

Exporting a llama.cpp profile does not include the GGUF or mmproj file. If you import the profile on a different device, you need to re-download those files and update the path fields.

## Tips

-   Start with a Q4\_K\_M or Q5\_K\_M file if you are unsure. They sit in the runability sweet spot for most desktops.
-   Check the runability score before downloading. An "Excellent" or "Good" label means the file should run on your hardware with current free memory.
-   If the model loads but generates slowly, try lowering GPU layers, shrinking the context length, or switching the KV cache to `q8_0`.
-   For vision models, remember to set the mmproj path and enable the image input scope. Without both, image messages will not reach the model.
-   Read the runtime report after the first run to see whether the engine had to fall back (smart layer fallback, KQV fallback, or context fallback). The actual values are what the model used, not what you asked for.

[

PreviousModels

](/docs/models)[

NextMulti-GPU Local Models

](/docs/multi-gpu)