# Local LLMs, explained and tested

> What running a large language model (LLM) on your own hardware involves, the pieces of the stack and the model types, then what I found testing them on a Strix Halo laptop and an RTX 3090.

Web page: https://www.farez.ai/local-llms

This page has two parts. The first explains what running a model locally involves, for anyone technical who hasn't tried it yet: the pieces of the stack, the kinds of model you'll see named, and the two speeds that matter. The second is [my experiments](#my-experiments): what worked, and what didn't, on my own laptop and desktop. If you already run models locally, skip straight there.

## What a local model is

When you use a hosted AI service, your prompt goes over the internet to someone else's servers, the model runs there, and you pay per token or by subscription. You never see the model itself.

A local model is one you download and run on your own hardware. "Open weights" means the trained numbers that make up the model, its weights, are published as files anyone can download. You load that file into a program on your machine, and nothing leaves it.

Why bother:

- **Privacy.** Prompts, code and documents stay on your machine.
- **Cost.** Once you own the hardware, a long job costs electricity, not a per-token bill.
- **Offline.** It works without a connection.
- **Overnight batch work.** You can queue up a pile of jobs and let them run unattended. That's the use I'm testing.

What you give up:

- **Quality.** The open models that fit on a laptop or a gaming GPU are behind the best hosted models, most of all on long, hard tasks.
- **Speed.** A hosted service runs on data-centre hardware. At home you get what your GPU can do.
- **Setup.** You pick a model file, an engine and the settings yourself, and some of it is trial and error.
- **Size.** The model has to fit in your GPU's memory, which caps how big a model you can run.

## The stack, bottom to top

Running a model locally means stacking a few pieces of software on your hardware. Each one can be swapped without changing the others.

**Hardware.** Models run fastest on a GPU, and the hard limit is the GPU's memory: the whole model has to fit in it, with room left over for the conversation. A desktop card like my RTX 3090 has 24 GB of its own memory, which is fast but small. My laptop has 128 GB of unified memory shared by the CPU and the GPU, and I set 96 GB of it aside for the GPU. It's slower than the card, but it can hold far bigger models.

**GPU backend.** The engine needs a way to run its maths on your particular GPU. That's the backend:

- **Vulkan** is a cross-vendor graphics and compute API. It works on AMD, Nvidia and Intel GPUs, so it's the easy option on an AMD laptop under Windows. I used it on the laptop.
- **CUDA** is Nvidia's compute platform, and only runs on Nvidia GPUs. I used it on the 3090.
- **ROCm** is AMD's compute stack, mainly used on Linux. My Linux runs will use it.

**Inference engine.** The program that loads the model and runs it. llama.cpp is the open-source C/C++ engine most people start with. It loads GGUF model files and can be built for many backends: Vulkan, CUDA, ROCm, Metal on Macs, and others. [Loom](https://pypi.org/project/hrx-loom/) is a much newer engine from AMD's ROCm organization. It only runs models that have been ported to it, and each port pins one exact model file.

**Model file.** A model is billions of numbers. GGUF is the single-file format llama.cpp uses to hold them, along with what's needed to run them. Most files are quantised: each weight is stored in fewer bits than the original release, so the file is smaller and quicker to read, and the model gets slightly less accurate. The end of the name says how. In "Qwen3.8 27B Q4_K_M", Q4_K_M is a common scheme at roughly 4 bits per weight. My tables also have Q4_0, MXFP4 and a smaller IQ3_XXS.

**Server.** llama.cpp comes with `llama-server`, and Loom with `loom-serve`. Both expose an OpenAI-compatible HTTP API: the same request shape the hosted services use. A tool written for a hosted API can often be pointed at your own machine just by changing the address it sends to.

**App or agent.** Whatever sends the requests: a script, an editor, a coding agent. In my tests it's a small harness that gives the model a task and a few file tools.

**Figure: The stack, and a request going through it** (animated on the web page)

1. **Six layers.** From the bottom: the hardware, a GPU backend (Vulkan, CUDA or ROCm), an inference engine (llama.cpp, Loom), the model file, a server with an OpenAI-compatible API, and the app or agent that calls it. Your prompt goes down; tokens come back up.
2. **Memory decides what fits.** The model file has to fit in the memory the GPU can use. My RTX 3090 has 24 GB; my laptop gives 96 GB of its 128 GB to the GPU. The sizes here are the model files from my speed tests: only the three smallest fit on the card.
3. **A request goes down the stack.** Your code sends an ordinary chat request over HTTP. The server hands it to the engine, the engine runs the model whose weights sit in GPU memory, and the backend turns that work into something the GPU can run.
4. **Prefill: read the whole prompt at once.** First the model reads your prompt. All of its tokens go through together in one parallel pass, which is why prefill runs at hundreds to thousands of tokens per second on my machines.
5. **Decode: write one token at a time.** Then it writes. Each new token needs another pass through the model, one after the other, so decode is far slower: tens of tokens per second, sometimes a couple of hundred. The tokens stream back up to your app as they are made.
6. **Swap a layer, keep the rest.** The layers are interchangeable. llama.cpp can be built for Vulkan, CUDA or ROCm; Loom is a different engine with its own server. Because both servers speak the same OpenAI-style API, the app on top does not change.

## The models you'll see named

Model names pack in a lot. The number with a B is the size, in billions of weights: a 9B model is small, a 120B model is big. More weights usually means a more capable model and a bigger file. The rest of the name tells you about its design.

**Dense or mixture-of-experts.** In a dense model, every token goes through every weight. In a mixture-of-experts (MoE) model, much of the network is split into "experts", and a small router picks a few of them for each token. A name like "30B-A3B" means 30B weights in total, about 3B of them active per token. All 30B still have to fit in memory, but each token only reads about a tenth of them, so the model writes much faster. Faster isn't the same as better, as my results show.

**Reasoning, or "thinking", models.** These write out a plan before they answer. That often helps on hard tasks, but the plan costs tokens and time, and a model left to itself can spend its whole budget thinking and never produce the answer. llama.cpp's `--reasoning-budget` caps the thinking. For Qwen3.8 27B in my tests, the cap was the difference between no code at all and one of the best results.

**Speculative decoding and MTP.** Two ways to write faster without changing who decides the answer. In speculative decoding, a small "drafter" model guesses the next few tokens and the big model checks them all at once, keeping the ones it agrees with. Multi-token prediction (MTP) does the same with extra heads built into the model, so it drafts for itself.

**Figure: Model architectures, token by token** (animated on the web page)

1. **Dense: every token uses every weight.** In a dense model each token passes through all of the weights. Qwen3.8 27B is dense, so every token it writes reads all 27B of them.
2. **Mixture of experts: a router picks a few.** A mixture-of-experts (MoE) model splits much of its network into experts, and a small router picks a few of them for each token. "30B-A3B" means 30B weights in total and about 3B active per token. The whole model still has to fit in memory.
3. **Fewer active weights, faster writing.** Writing is mostly a matter of reading weights from memory, so reading fewer per token is faster. On my laptop Qwen3.8 27B (dense) decodes at 13.0 tokens per second and Qwen3-30B-A3B (MoE) at 83.3, from files of similar size. Faster was not better in my tests, though.
4. **Reasoning models think before they answer.** Reasoning ("thinking") models write out a plan before the answer, and that plan costs tokens and time. Uncapped, a model can spend its whole budget thinking and never write the code. llama.cpp's --reasoning-budget caps it; I used 4,000 tokens.
5. **Speculative decoding: a drafter guesses ahead.** A small, fast drafter model guesses the next few tokens. The main model checks all of the guesses in one pass, keeps the ones it agrees with and replaces the first one it does not. It still has the final say on every token; it just gets there in fewer passes.
6. **Multi-token prediction: the model drafts for itself.** Some models ship extra multi-token prediction (MTP) heads that guess a few tokens ahead, so the model can draft for itself without a second model. With MTP on in llama.cpp, Qwen3.8 27B went from 13 to roughly 24 to 30 tokens per second in my coding runs.

## Two speeds: prefill and decode

Speed is counted in tokens per second. A token is a chunk of text, often a word or part of one. There are two speeds, and they're very different:

- **Prefill** is reading the prompt. All of the prompt's tokens are processed together, so it's fast: hundreds to thousands of tokens per second on my machines. It matters when prompts are long, like an agent sending its whole conversation back every turn.
- **Decode** is writing the answer, one token at a time. Each token needs another pass through the model's active weights, so it's much slower: tens of tokens per second, a couple of hundred at best on my machines. This is the number you feel as waiting.

Both matter for agent work. An agent's context grows every turn, which is prefill, and it writes code and thinking, which is decode. Here are the decode numbers from my own tests, the same ones that are in the tables in my [speed](/blog/strix-halo-vs-rtx-3090-local-llm-speed) and [Loom](/blog/amd-loom-vs-llama-cpp-strix-halo-windows) posts.

**Figure: My decode speeds at a glance** (animated on the web page)

1. **Decode on the laptop.** Tokens per second on the Z13 with llama.cpp and Vulkan: Qwen3-30B-A3B 83.3, Nemotron 3.5 Lightning 66.6, gpt-oss-120b 52.8, Qwen3.8-Flash-Next 27.5, Qwen3.8 27B 13.0. The two 30B-A3B mixture-of-experts models are the fastest; the dense Qwen3.8 27B is the slowest.
2. **The same models on the RTX 3090.** On the 3090 with CUDA: Qwen3-30B-A3B 202.3, Nemotron 3.5 Lightning 192.5, Qwen3.8 27B 43.1. Between 2.4 and 3.3 times the laptop's decode speed, but only for models that fit in 24 GB.
3. **Same laptop, different engine.** Output tokens per second over whole repair runs. Qwen3.8 27B (dense): llama.cpp Vulkan 11.6–11.9, llama.cpp + MTP 23.1–26.4, Loom 25.2–25.7. Ornith 1.5 9B (dense): llama.cpp Vulkan 33–34, Loom 50–51. Ornith 1.5 35B-A3B (MoE): llama.cpp Vulkan 54–56, Loom 40–45. Details and caveats are in my Loom vs llama.cpp post.

## My experiments

I make an explainer-video series, [engineering.fm](https://engineering.fm), whose animations are written as code ([Remotion](https://www.remotion.dev) scenes in React). I wanted to know whether open-weight models running on my own machines could take on real parts of that work, and some unattended code repair, well enough to queue it up and let it run overnight. Each experiment has its own post with the full tables, method and caveats. This is the summary, and I'll update it as I test more.

The machines: an ASUS ROG Flow Z13 laptop (Ryzen AI Max+ 395, Strix Halo, with 96 GB of its 128 GB set aside for the GPU, llama.cpp with Vulkan) and a desktop with an RTX 3090 (24 GB, llama.cpp with CUDA). Both on Windows 11.

### Strix Halo laptop vs RTX 3090

The 3090 is much faster on every model that fits in its 24 GB: roughly 4–6× at prefill and 2.4–3.3× at decode. The Z13 runs models the 3090 can't, like Qwen3.8-Flash-Next (77.9 GiB) and gpt-oss-120b. [Full speed tables →](/blog/strix-halo-vs-rtx-3090-local-llm-speed)

### Writing animation scenes

Qwen3.8-Flash-Next on the laptop and Qwen3.8 27B on the desktop both rewrote a full 44-second scene from a detailed spec and came out close to the original (scores of 0.94–0.97 out of 1). Capping the thinking mattered: uncapped, Qwen3.8 27B spent its whole budget planning and wrote no code; capped at 4,000 tokens on the 3090, it scored 0.948 in under 10 minutes. The fast mixture-of-experts models never produced a working full scene on the 3090. And the spec is the ceiling: a local model saves me coding tokens, not design work. [The scene test →](/blog/local-llm-animation-scenes)

### Unattended code repair

Flash-Next and Qwen3.8 27B fixed all three small JavaScript projects, checked against tests they never saw, in 41 to 193 seconds each. Qwen3-30B-A3B failed two of the three. With 91 tool definitions in the prompt instead of 5, Flash-Next still passed but took twice as long (88 s vs 41 s). [The repair test →](/blog/local-llm-unattended-code-repair)

### AMD's Loom engine

On Qwen3.8 27B, Loom gave about twice the throughput of plain llama.cpp on Vulkan, roughly level with llama.cpp + MTP. On the dense Ornith 1.5 9B it was about 1.5× faster; on the mixture-of-experts Ornith 1.5 35B-A3B it was slower, at about 0.75×. [Loom vs llama.cpp →](/blog/amd-loom-vs-llama-cpp-strix-halo-windows)

### Caveats

These are single runs, not averages, and everything so far ran on Windows. Linux, with ROCm, is next. Each post lists the caveats that apply to its experiment.

Last updated: 2026-11-16
