In the last episode Qwen3.8 27B was the most accurate model, with two problems: it was slow (11 minutes for a year of spending), and once it had the right answer and got cut off before saying it. This episode changes no questions and no models, only settings. On a 48GB Mac, how do you make it faster, handle long contexts, and stop it from quitting on its own?
The setup
- Machine: an Apple Silicon Mac (M5 Pro) with 48GB of memory.
- Inference: oMLX 0.7.2, with my memory guard left at 40 GB.
- Speed: measured straight against oMLX's API, not through the harness. Streaming, greedy decoding, thinking off, at least 640 output tokens. One short prompt and one of about 25K tokens.
- Long context: oMLX's own source code stitched into about 120K tokens, with a code phrase hidden in the middle. The model has to find the phrase and say what the software is.
- Memory: the oMLX process's footprint, sampled every 5 seconds.
Each configuration ran once, with only one model in memory at a time.
Two kinds of MTP
Speculative decoding is simple: a tiny draft guesses the next few tokens, the big model checks them all in one pass, and every correct guess is free speed. Qwen3.8 can do this two ways:
- An external draft head: a separate small file (about 0.2 GB) attached to the main model. This is what I had.
- MTP layers built into the model: the draft layers live inside the model file. oMLX calls this Lightning MTP, and it has its own switch.
At first I saw the Lightning MTP switch was off and assumed I'd never had speculative decoding on. In fact the external draft head had been working all along: the 25 to 30 tok/s I was used to already included it. Lightning MTP couldn't run on my model at all, because many community 4-bit builds strip out the built-in MTP layers, and mine had none left. You can count them before downloading:
python3 -c "import json; m=json.load(open('model.safetensors.index.json'))['weight_map']; print(sum('mtp' in k for k in m))"Zero means they're gone. I switched to Qwen3.8-27B-oQ4e-mtp, made by the author of oMLX (17 GB, 29 MTP tensors kept), and oMLX turned Lightning MTP on as soon as it found the model. The drafts also got better: the average tokens per round went from 2.0–2.3 to 2.85–3.08.
TurboQuant: saves memory, not time, with a catch
TurboQuant compresses the context cache (the KV cache) to 4 bits. That saves memory, so the same RAM holds a longer context. It doesn't make generation faster.
The catch: it can't be combined with the external draft head. When I turned both on for the old model, oMLX didn't just disable one of them. It refused to load any of that model's settings, with one line in the log:
vlm_mtp_enabled and turboquant_kv_enabled cannot both be True; choose one speculative path per modelI had a backup and restored it right away. The lesson: back up oMLX's settings before you change them, and check the log for Failed to load settings afterwards. With the new model's built-in MTP layers, Lightning MTP and TurboQuant run together fine.
Results
| Configuration | Short prompt | 25K tokens | 120K tokens | Peak memory |
|---|---|---|---|---|
| A: the old one (external draft head) | 28.8–30.6 tok/s | 24.3 tok/s | ✓ | not measured |
| B: new model + Lightning MTP | 32.9 tok/s | 30.0 tok/s | ✓ | 31 GB |
| C: new model + Lightning MTP + TurboQuant | 34.9 tok/s | 27.3 tok/s | ✓ | 28 GB |
B is fastest on long contexts, but I picked C. My oMLX starts refusing requests at around 33 to 34 GB, and B already reached 31 GB at 120K tokens. The harness now allows 131K tokens in plus 32K out, so B would likely hit the wall near that limit. C is 9% slower there and buys 3 GB of headroom. Against where I started, C is about 15% faster on short prompts and 12% faster on long ones.
TurboQuant saved less memory than I expected, because Qwen3.8 uses hybrid attention: only 16 of its 64 layers keep a full context cache, so the cache is small to begin with.
The real wait: reading
On long coding tasks the slow part isn't writing, it's reading. Qwen reads a long context at about 300 tokens per second, so the first read of 120K tokens takes around 7 minutes, while the reply itself takes under a minute. oMLX caches what it has already read, so within one session only the first turn is slow. Ornith reads about five times faster: 92K tokens in 58 seconds.
I also tested the case that worried me most. Right after a 120K-token request, with its cache still in memory, I sent 135K tokens of different content. That kind of request has been refused on this machine before. This time it went through, and the model found the code phrase.
Why it quit on its own: the client's limits
When Qwen had the right answer and couldn't say it, oMLX wasn't the problem: it already allowed a 200K context. The harness was. It was set to a 49K context and 16K output per step, so a model that thought a little longer ran into the output limit.
The rule is simple: line up the limits at every layer, so that oMLX's context limit is at least the harness's context plus its output. Then the harness compacts the conversation before oMLX ever refuses a request. My settings now:
| oMLX context / output | Harness context / output | |
|---|---|---|
| Qwen3.8 27B | 204,800 / 32,768 | 131,072 / 32,768 |
| Ornith 35B-A3B | 131,072 / 32,768 | 98,304 / 32,768 |
Settings you can copy
The model's entry in oMLX (only the keys I changed):
"Qwen3.8-27B-oQ4e-mtp": {
"mtp_enabled": true,
"turboquant_kv_enabled": true,
"max_context_window": 204800,
"max_tokens": 32768
}After editing, reload settings from oMLX's admin page and check the log to confirm they took effect.
Caveats
Each configuration ran once, and the short-prompt differences are within noise. Speed was measured against oMLX directly, not through the harness, which adds several thousand to over ten thousand tokens of system prompt and tool definitions every turn, so real use is slower. I didn't test the extreme case of 131K in and 32K out at the same time, didn't test whether TurboQuant changes answer quality, and didn't try DFlash, oMLX's other speculative decoding method. All of this comes from one machine.