What Actually Makes A Local Model Slow

People blame model size for local AI feeling slow, then download a smaller model, then discover it is still slow. Size matters, but it is rarely the thing that decides whether a tool is pleasant to use.

Here is what actually governs the wait, roughly in order of how often it is the real culprit.

The First Run Is Loading, Not Thinking

The most common complaint about local AI is that the first rewrite takes twenty seconds and later ones take three. Nothing improved in between. The first run included loading the model from disk into memory.

A 2.4GB model has to be read off storage and placed in RAM before a single token can be generated. On an NVMe drive that is a few seconds. On a slower drive, or a machine already under memory pressure, it can be much longer.

This is why a tool that keeps the model resident feels dramatically faster than one that loads it per request, even with identical hardware and an identical model. It is also why the honest way to report progress is to say “getting the model ready” rather than showing a spinner that implies work is happening on your text.

If your first rewrite of the day is slow and the next five are fine, you are seeing load time and there is nothing wrong.

Memory Pressure Turns Into Disk Access

A model needs its weights in RAM. If the machine does not have room, Windows starts paging, and paging during inference is catastrophic because the access pattern is large and repeated.

The symptom is distinctive: generation that starts at a reasonable pace and then stutters, with disk activity climbing. It is much worse than simply having a slower processor, because you have added a storage round trip to a compute loop.

The fix is not a faster CPU. It is either a smaller model or fewer things open. A 4B model needing roughly 3GB of RAM is comfortable on a 16GB machine and marginal on an 8GB machine that is also running a browser with thirty tabs and a video call.

How much RAM to run a local LLM covers the sizing properly. The short version is to leave headroom rather than fitting exactly.

Input Length Costs More Than People Expect

Generating output is not the only work. Before the model writes anything it has to process everything you gave it, which includes your text, the instruction, and any examples attached to the context.

This is why pasting three paragraphs feels slower than pasting three sentences even though the output is the same length, and why a context stuffed with a dozen style examples adds a delay to every single rewrite.

The practical consequence is that trimming your input is often the highest-leverage speed fix available, and it costs nothing. If you are rewriting one paragraph, paste one paragraph.

It is also an argument for keeping example sets small and representative rather than exhaustive. Every example is paid for on every run.

Quantization Is The Lever Most People Skip

Model weights can be stored at lower precision, which makes the file smaller and the arithmetic cheaper. Four-bit quantization is the common default for local use, and the quality cost for rewriting work is small.

The reason this matters for speed is that inference on a CPU, in engines such as llama.cpp, is frequently limited by memory bandwidth rather than raw arithmetic. Smaller weights mean less data moved per token, so a quantized model can be substantially faster even on the same hardware.

Quantization explained for writers covers the tradeoff without the mathematics. The summary is that for rewriting and tone adjustment, four-bit is nearly always the right choice, and chasing higher precision buys you very little on text you are going to edit anyway.

Parameter Count Is Real, Just Overrated

Having ranked everything else first, size does matter. A 4B model is meaningfully slower per token than a 1.7B one, and on a constrained laptop that difference decides whether you keep using the tool.

What people get wrong is assuming the bigger model is proportionally better at rewriting. For tone and clarity work it often is not. Rewriting is a constrained task with the content already supplied, which is exactly the shape of problem where small models hold up well. Small language models beat big ones for rewriting makes that case in full.

Try both on your own text before assuming you need the larger one.

A Wrivio Context that keeps rewrites fast could say:

Rewrite this as a clear, direct message in the same register. Match the original length within 10 percent. Keep every name, date, figure, and commitment exactly as written. Do not add an introduction or a closing line.

Press Ctrl+Shift+Space, paste one paragraph rather than a whole thread, and check the diff. A length constraint caps the output, which caps the wait.

Common Questions

Why is my first local rewrite so much slower than the rest?

The first one includes loading the model from disk into memory, which can take several seconds for a multi-gigabyte file. Later rewrites reuse the loaded model, so they only pay for generation.

Does a faster CPU fix a slow local model?

Sometimes, but memory is usually the tighter constraint. If the machine is paging to disk because the model does not fit comfortably in RAM, a faster processor changes very little.

Does the length of my input affect speed?

Yes, more than most people expect. The model processes your entire input before generating anything, so long pastes and large example sets add delay to every run.

Is a 4B model always better than a 1.7B one?

Not for rewriting. The content is already supplied and the task is constrained, which is where smaller models hold up well, so the larger model often costs you speed for a quality difference you cannot see.

Download Wrivio for Windows to try both local tiers on your own text and find out which one is actually worth the wait on your machine.