When a local LLM answers too slowly, most people think of a graphics card with more VRAM. But now there's a claim that just swapping the program that runs the model makes it faster. It's an open-source engine called Magnitude.
The developer says that across 11 models compared against Ollama, answer generation speed was 2.19x faster on average on a Mac M4 Pro and 2.17x faster on an NVIDIA DGX Spark.
But in an independent test, the same model was 2.17x faster when copying text, then actually got slower when making up a new story. So what's going on?
Change the Engine, Not the Model
Popular engines for running LLMs include Ollama and llama.cpp. Magnitude, the one getting attention now, is an engine too.
It isn't a program for training models. It's a runtime that helps a model compute more efficiently on your hardware.
✦ What Magnitude Does at First Download
A quick tuning pass, then the result is saved
When you first download a model, Magnitude tests compute settings that fit your current machine. It tries different combinations, like what size of job the GPU handles and how the work is split, then picks the fast one and saves it.
According to the developer, this tuning takes up to about a minute. It's one of the factors that speeds things up.
If you've used an engine like LM Studio, you've probably been confused every time you load a model about how to set each value for your machine. There are quite a lot of adjustable values, cache settings included.
Think of Magnitude as doing that setup for you, tailored to your machine.
✦ Speculative Decoding in Three Beats
Guess ahead, check, keep what matches
The other factor that boosts speed is 'speculative decoding.' While the big model computes the next word one at a time, a small model or a dedicated draft generator proposes candidate tokens for what comes next.
The big model checks those candidates and accepts the right ones, which makes things faster by that much. If the candidates predicted the next tokens well, the gain is bigger. If they miss, the effect shrinks.
Which Stage Actually Got 2x Faster?
✦ Two Stages of Answering
Which one the "2x" is about
When you ask an LLM a question, it reads your question and the conversation so far, then starts producing the answer token by token. Processing the input context is called prefill, and generating the answer is called decode.
What directly shapes that slow, character-by-character feel of an answer is mostly decode speed.
Magnitude ran a benchmark comparison against Ollama, the most representative engine. In the benchmark, both engines used speculative decoding, but Ollama used the draft generator bundled with each model, while Magnitude used its own.
✦ Magnitude's Published Averages
11 models, speed relative to Ollama
Here are Magnitude's average results from the 11-model comparison it published. On a Mac M4 Pro 48GB, prefill was 1.32x and decode 2.19x. On an NVIDIA DGX Spark 128GB, prefill was 1.34x and decode 2.17x.
So the 2x figure is about decode speed, the speed of generating the answer. It doesn't mean prefill, reading a long question, got twice as fast too.
✦ Model by Model: Decode Speed
Tokens per second, Ollama vs. Magnitude
Individual models show bigger gaps. On a Mac M4 Pro, Qwen 3.5 4B went from 45.2 tokens per second to 134, about 2.97x. On the NVIDIA DGX Spark, the same model went from 61.1 to 296 tokens, a 4.84x result.
But on the DGX Spark, Nemotron 3.5 dropped from 94.9 tokens per second to 72.2, so Magnitude was actually about 24% slower.
Why Fast at Copying but Slow at Writing Something New?
✦ How the Official Benchmark Worked
A speed test, not a quality test
It also helps to know how the official benchmark worked. They fed in about 64,000 tokens of text from Moby-Dick and had the model repeat the final part. Output was capped at 256 tokens, and both engines had speculative decoding turned on.
This experiment compares how fast a model reads a long context and then prints a fixed passage again. It doesn't evaluate answer quality or coding success rates.
✦ Independent Retest: The Result Flipped
Qwen3.6 35B-A3B, Ollama vs. Magnitude, tokens per second
On copy tasks like this, the next sentence is already sitting in the input, so the draft generator is likely to guess the candidates right. Makes sense, right?
Niklas Lenz, who ran an independent test, reran a similar copy test on the NVIDIA DGX Spark with Qwen3.6 35B-A3B.
Ollama hit 87.2 tokens per second and Magnitude hit 188.9, about a 2.17x gap.
But when the same model was asked to write a new story after a long context, the result flipped. Ollama was at 63.4 tokens per second, Magnitude at 50.4. Magnitude ran at about 0.79x of Ollama, in other words about 21% slower.
In a test that makes a new story without a long context, Magnitude was slightly slower too.
Why the difference? You've probably guessed already. On a task like copying, where the next token is easy to predict, the draft generator's predictions get more accurate.
When it has to write a new story, the next expression has far more variety, so the drafts are bound to miss more often.
In Lenz's test, about 75% of the candidates Magnitude's draft generator proposed hit on the copy task, but only about 43% on the new story.
There was also a test that ran the same model file through Ollama again. On the copy task, Magnitude was far faster at 188.9 vs. 45.6 tokens, and on the new story it was about even at 50.4 vs. 46.1.
So going by the results so far, Magnitude clearly has an edge on certain tasks. But we can't expect a reliable speedup on the general tasks we usually want local LLMs for.
So, Is It Worth Installing?
✦ Getting Started with Magnitude
From install to connecting an agent
Magnitude is a free, open-source tool that works on macOS, Windows and Linux. Anyone can download and run it, and it also offers a way to connect with agents like Codex and OpenCode.
If you'd like to try installing it, check the links below.
Official website: https://magnitude.dev
Official repository: https://github.com/magnitudedev/magnitude
If you haven't worked with tools like this much, tell the AI agent you use (Codex, Claude Code, etc.) the links above and ask it to install Magnitude in your environment. It'll set it up in no time.
Anyway, going by these results, the line 'swap the engine and it's automatically twice as fast' is a bit of an exaggeration. Still, if you keep using local LLMs, Magnitude is well worth a try.
Honestly, right now there's not much you can do to speed up a local LLM other than swapping the graphics card. Quantization costs you quality in exchange for speed, and graphics cards are just too expensive. So a free runtime like this is a nice opportunity.
Especially if your usual work involves a lot of repetition from the prompt you've already entered or the conversation so far, you might get a significant speedup. Give it a try. Hehe♡
That's all for this post. See you next time.
Related posts
- Ollama's New Q2 Quantization Support: Quantization Explained
- Mercury 3 Launches at 1,000+ Tokens/Sec. How Fast Is That?
- Medical AI MAI-DxO Beat Doctors. Will Doctors Disappear?
🌐 Korean version: 너무 느린 로컬 LLM... 실행 프로그램만 Magnitude로 바꿔도 2배 빨라진다는데! 사실일까?
No comments:
No comments yet. Want to leave the first one?
Post a Comment