This page has two parts. The first explains what running a model locally involves, for anyone technical who hasn’t tried it yet: the pieces of the stack, the kinds of model you’ll see named, and the two speeds that matter. The second is my experiments: what worked, and what didn’t, on my own laptop and desktop. If you already run models locally, skip straight there.
What a local model is
When you use a hosted AI service, your prompt goes over the internet to someone else’s servers, the model runs there, and you pay per token or by subscription. You never see the model itself.
A local model is one you download and run on your own hardware. “Open weights” means the trained numbers that make up the model, its weights, are published as files anyone can download. You load that file into a program on your machine, and nothing leaves it.
Why bother:
- Privacy. Prompts, code and documents stay on your machine.
- Cost. Once you own the hardware, a long job costs electricity, not a per-token bill.
- Offline. It works without a connection.
- Overnight batch work. You can queue up a pile of jobs and let them run unattended. That’s the use I’m testing.
What you give up:
- Quality. The open models that fit on a laptop or a gaming GPU are behind the best hosted models, most of all on long, hard tasks.
- Speed. A hosted service runs on data-centre hardware. At home you get what your GPU can do.
- Setup. You pick a model file, an engine and the settings yourself, and some of it is trial and error.
- Size. The model has to fit in your GPU’s memory, which caps how big a model you can run.
The stack, bottom to top
Running a model locally means stacking a few pieces of software on your hardware. Each one can be swapped without changing the others.
Hardware. Models run fastest on a GPU, and the hard limit is the GPU’s memory: the whole model has to fit in it, with room left over for the conversation. A desktop card like my RTX 3090 has 24 GB of its own memory, which is fast but small. My laptop has 128 GB of unified memory shared by the CPU and the GPU, and I set 96 GB of it aside for the GPU. It’s slower than the card, but it can hold far bigger models.
GPU backend. The engine needs a way to run its maths on your particular GPU. That’s the backend:
- Vulkan is a cross-vendor graphics and compute API. It works on AMD, Nvidia and Intel GPUs, so it’s the easy option on an AMD laptop under Windows. I used it on the laptop.
- CUDA is Nvidia’s compute platform, and only runs on Nvidia GPUs. I used it on the 3090.
- ROCm is AMD’s compute stack, mainly used on Linux. My Linux runs will use it.
Inference engine. The program that loads the model and runs it. llama.cpp is the open-source C/C++ engine most people start with. It loads GGUF model files and can be built for many backends: Vulkan, CUDA, ROCm, Metal on Macs, and others. Loom is a much newer engine from AMD’s ROCm organization. It only runs models that have been ported to it, and each port pins one exact model file.
Model file. A model is billions of numbers. GGUF is the single-file format llama.cpp uses to hold them, along with what’s needed to run them. Most files are quantised: each weight is stored in fewer bits than the original release, so the file is smaller and quicker to read, and the model gets slightly less accurate. The end of the name says how. In “Qwen3.8 27B Q4_K_M”, Q4_K_M is a common scheme at roughly 4 bits per weight. My tables also have Q4_0, MXFP4 and a smaller IQ3_XXS.
Server. llama.cpp comes with llama-server, and Loom with loom-serve. Both expose an OpenAI-compatible HTTP API: the same request shape the hosted services use. A tool written for a hosted API can often be pointed at your own machine just by changing the address it sends to.
App or agent. Whatever sends the requests: a script, an editor, a coding agent. In my tests it’s a small harness that gives the model a task and a few file tools.