farez.ai

Local LLMs, explained and tested

What running a large language model (LLM) on your own hardware involves, the pieces of the stack and the model types, then what I found testing them on a Strix Halo laptop and an RTX 3090.

Part 1 · How local models work

This page has two parts. The first explains what running a model locally involves, for anyone technical who hasn’t tried it yet: the pieces of the stack, the kinds of model you’ll see named, and the two speeds that matter. The second is my experiments: what worked, and what didn’t, on my own laptop and desktop. If you already run models locally, skip straight there.

What a local model is

When you use a hosted AI service, your prompt goes over the internet to someone else’s servers, the model runs there, and you pay per token or by subscription. You never see the model itself.

A local model is one you download and run on your own hardware. “Open weights” means the trained numbers that make up the model, its weights, are published as files anyone can download. You load that file into a program on your machine, and nothing leaves it.

Why bother:

  • Privacy. Prompts, code and documents stay on your machine.
  • Cost. Once you own the hardware, a long job costs electricity, not a per-token bill.
  • Offline. It works without a connection.
  • Overnight batch work. You can queue up a pile of jobs and let them run unattended. That’s the use I’m testing.

What you give up:

  • Quality. The open models that fit on a laptop or a gaming GPU are behind the best hosted models, most of all on long, hard tasks.
  • Speed. A hosted service runs on data-centre hardware. At home you get what your GPU can do.
  • Setup. You pick a model file, an engine and the settings yourself, and some of it is trial and error.
  • Size. The model has to fit in your GPU’s memory, which caps how big a model you can run.

The stack, bottom to top

Running a model locally means stacking a few pieces of software on your hardware. Each one can be swapped without changing the others.

Hardware. Models run fastest on a GPU, and the hard limit is the GPU’s memory: the whole model has to fit in it, with room left over for the conversation. A desktop card like my RTX 3090 has 24 GB of its own memory, which is fast but small. My laptop has 128 GB of unified memory shared by the CPU and the GPU, and I set 96 GB of it aside for the GPU. It’s slower than the card, but it can hold far bigger models.

GPU backend. The engine needs a way to run its maths on your particular GPU. That’s the backend:

  • Vulkan is a cross-vendor graphics and compute API. It works on AMD, Nvidia and Intel GPUs, so it’s the easy option on an AMD laptop under Windows. I used it on the laptop.
  • CUDA is Nvidia’s compute platform, and only runs on Nvidia GPUs. I used it on the 3090.
  • ROCm is AMD’s compute stack, mainly used on Linux. My Linux runs will use it.

Inference engine. The program that loads the model and runs it. llama.cpp is the open-source C/C++ engine most people start with. It loads GGUF model files and can be built for many backends: Vulkan, CUDA, ROCm, Metal on Macs, and others. Loom is a much newer engine from AMD’s ROCm organization. It only runs models that have been ported to it, and each port pins one exact model file.

Model file. A model is billions of numbers. GGUF is the single-file format llama.cpp uses to hold them, along with what’s needed to run them. Most files are quantised: each weight is stored in fewer bits than the original release, so the file is smaller and quicker to read, and the model gets slightly less accurate. The end of the name says how. In “Qwen3.8 27B Q4_K_M”, Q4_K_M is a common scheme at roughly 4 bits per weight. My tables also have Q4_0, MXFP4 and a smaller IQ3_XXS.

Server. llama.cpp comes with llama-server, and Loom with loom-serve. Both expose an OpenAI-compatible HTTP API: the same request shape the hosted services use. A tool written for a hosted API can often be pointed at your own machine just by changing the address it sends to.

App or agent. Whatever sends the requests: a script, an editor, a coding agent. In my tests it’s a small harness that gives the model a task and a few file tools.

App or agentscript · editor · agentServerllama-server · loom-serveInference enginellama.cpp · LoomModel fileGGUF · e.g. Q4_K_MGPU backendVulkan · CUDA · ROCmHardwareGPU + its memorypromptgoes downtokenscome up
01/06
0:00 / 0:37

The models you’ll see named

Model names pack in a lot. The number with a B is the size, in billions of weights: a 9B model is small, a 120B model is big. More weights usually means a more capable model and a bigger file. The rest of the name tells you about its design.

Dense or mixture-of-experts. In a dense model, every token goes through every weight. In a mixture-of-experts (MoE) model, much of the network is split into “experts”, and a small router picks a few of them for each token. A name like “30B-A3B” means 30B weights in total, about 3B of them active per token. All 30B still have to fit in memory, but each token only reads about a tenth of them, so the model writes much faster. Faster isn’t the same as better, as my results show.

Reasoning, or “thinking”, models. These write out a plan before they answer. That often helps on hard tasks, but the plan costs tokens and time, and a model left to itself can spend its whole budget thinking and never produce the answer. llama.cpp’s --reasoning-budget caps the thinking. For Qwen3.8 27B in my tests, the cap was the difference between no code at all and one of the best results.

Speculative decoding and MTP. Two ways to write faster without changing who decides the answer. In speculative decoding, a small “drafter” model guesses the next few tokens and the big model checks them all at once, keeping the ones it agrees with. Multi-token prediction (MTP) does the same with extra heads built into the model, so it drafts for itself.

Dense model"The"all of the model's weightsEvery token passes through every weight. Qwen3.8 27B reads all 27B per token.Simple and strong, but each token costs the whole model.
01/06
0:00 / 0:39

Two speeds: prefill and decode

Speed is counted in tokens per second. A token is a chunk of text, often a word or part of one. There are two speeds, and they’re very different:

  • Prefill is reading the prompt. All of the prompt’s tokens are processed together, so it’s fast: hundreds to thousands of tokens per second on my machines. It matters when prompts are long, like an agent sending its whole conversation back every turn.
  • Decode is writing the answer, one token at a time. Each token needs another pass through the model’s active weights, so it’s much slower: tens of tokens per second, a couple of hundred at best on my machines. This is the number you feel as waiting.

Both matter for agent work. An agent’s context grows every turn, which is prefill, and it writes code and thinking, which is decode. Here are the decode numbers from my own tests, the same ones that are in the tables in my speed and Loom posts.

Decode speed on the Z13tokens per second, llama-bench, llama.cpp VulkanQwen3-30B-A3B83.3Nemotron 3.5 Lightning66.6gpt-oss-120b52.8Qwen3.8-Flash-Next27.5Qwen3.8 27B13.0
01/03
0:00 / 0:19

Part 2 · My experiments

What I've tested so far

I make an explainer-video series, engineering.fm, whose animations are written as code (Remotion scenes in React). I wanted to know whether open-weight models running on my own machines could take on real parts of that work, and some unattended code repair, well enough to queue it up and let it run overnight. Each experiment has its own post with the full tables, method and caveats. This is the summary, and I’ll update it as I test more.

The machines: an ASUS ROG Flow Z13 laptop (Ryzen AI Max+ 395, Strix Halo, with 96 GB of its 128 GB set aside for the GPU, llama.cpp with Vulkan) and a desktop with an RTX 3090 (24 GB, llama.cpp with CUDA). Both on Windows 11.

Strix Halo laptop vs RTX 3090

The 3090 is much faster on every model that fits in its 24 GB: roughly 4–6× at prefill and 2.4–3.3× at decode. The Z13 runs models the 3090 can’t, like Qwen3.8-Flash-Next (77.9 GiB) and gpt-oss-120b. Full speed tables →

Writing animation scenes

Qwen3.8-Flash-Next on the laptop and Qwen3.8 27B on the desktop both rewrote a full 44-second scene from a detailed spec and came out close to the original (scores of 0.94–0.97 out of 1). Capping the thinking mattered: uncapped, Qwen3.8 27B spent its whole budget planning and wrote no code; capped at 4,000 tokens on the 3090, it scored 0.948 in under 10 minutes. The fast mixture-of-experts models never produced a working full scene on the 3090. And the spec is the ceiling: a local model saves me coding tokens, not design work. The scene test →

Unattended code repair

Flash-Next and Qwen3.8 27B fixed all three small JavaScript projects, checked against tests they never saw, in 41 to 193 seconds each. Qwen3-30B-A3B failed two of the three. With 91 tool definitions in the prompt instead of 5, Flash-Next still passed but took twice as long (88 s vs 41 s). The repair test →

AMD’s Loom engine

On Qwen3.8 27B, Loom gave about twice the throughput of plain llama.cpp on Vulkan, roughly level with llama.cpp + MTP. On the dense Ornith 1.5 9B it was about 1.5× faster; on the mixture-of-experts Ornith 1.5 35B-A3B it was slower, at about 0.75×. Loom vs llama.cpp →

Caveats

These are single runs, not averages, and everything so far ran on Windows. Linux, with ROCm, is next. Each post lists the caveats that apply to its experiment.

Last updated November 16, 2026 · also available as Markdown