The last episode ended with: "The fix isn't a smarter model. The model should read and decide, and code should count." My plan for episode two followed from that: give Ornith a few calculation tools.
I changed my mind. Tools are me patching the model's weak spot, and what I actually want to know is how far a local model gets on its own. If some local model gets smart enough not to need those tools, I want to see it happen. So I pulled the original episode three forward: no tools at all, just a different model.
The result is a little ironic. The bigger model can't do mental math either, but it worked out by itself that code should do the counting.
The setup
- Same environment: the same 48GB Apple Silicon Mac, the same DeepSeek Harness 0.2.1-rc.2 and dsh-genui plugin, with oMLX updated to 0.7.2. Nothing leaves the machine.
- Same data: fake Sam Lin's notes, with the same traps: a transfer to savings, a jacket bought and refunded, and a paycheck plus credit-card payments that would double any naive total.
- Only the questions that broke: "Where did my money go this month?" (27 rows in one month) and "Has my spending been going up or down over the last few months?" (about 300 rows across twelve months). The other five questions already worked last time, so I didn't repeat them.
- Same rules: one fresh session per question, read-only mode, thinking at its highest setting, the question typed as written. I didn't say "use code" and added no tools: the model only had the harness's built-in file reading, search and terminal.
The three models:
- Ornith 1.5 35B-A3B: last episode's model. Mixture-of-experts, about 3B parameters active per token, so it's fast.
- Qwen3.8 27B: a dense model that uses all 27B parameters for every token, so it's much slower. 4-bit, with MTP speculative decoding for speed (more on that next episode).
- LFM2.5 24B-A2B: a new mixture-of-experts model with very few active parameters, pitched as good at tool calling. It went through a qualifying round first.
Results
| Model | One month of spending | A year's trend |
|---|---|---|
| Ornith 35B-A3B | ✓ 2:26 | ✗ 1:19, four months given, September $41 short |
| Qwen3.8 27B | ✓ 5:22 | ✓ 11:24, all six months from May to October right ✗ second run: right numbers, no answer third run thrown out (see below) |
| LFM2.5 24B-A2B | ✗ wrong all three times | ✗ nine months given, seven wrong |
The difference isn't who's better at arithmetic. I went through the log of every run. Ornith never wrote a line of code and did every addition in its head. Qwen wrote a short Python script in the terminal every single time, to parse the ledger and total each month. Nobody asked it to.
The moment worth seeing
On the run where Qwen got the trend right, it first added up September in its head and got $4,753.95. Then it wrote a script, ran it, and got $4,931.06. The two numbers didn't match.
"The script's September total ($4,931.06) differs from my hand sum ($4,753.95) — let me debug which rows it's counting."
It compared the two row by row and found the problem:
"Found it — my hand sum had missed two September rows (Trader Joe's $44.29 + Whole Foods $132.82 = $177.11). The script is correct."
$4,931.06 − $4,753.95 is exactly $177.11. Its head math failed the same way Ornith's did last episode: a few rows dropped out of a few hundred. The difference is that it expected to make mistakes, checked itself with code, and trusted the code when the two disagreed.
The stretches of italic in that screenshot are the plugin rendering text between two dollar signs as a math formula, the same rough edge I mentioned last time.
Last episode I said code should count, and I assumed I'd be the one wiring that up. The 27B did it on its own.
Ornith: fast and wrong
Ornith answered the trend question in 1m 19s, and its conclusion, "roughly flat, slightly down", is reasonable. It wrote a bar chart that never rendered, which left the monthly list below. Note the "≈" in front of every number: it knew these were estimates. Three of the four months were actually exact. It put September at $4,890 when the answer is $4,931.06. Counting last episode's three attempts, Ornith has now answered this question four times and never got every number right.
Qwen fails too
Right numbers, no answer. On the second trend run Qwen wrote its script again, and all twelve monthly totals in its reasoning were correct. But it thought for so long before writing the reply that it hit the harness's per-step output limit of 16,384 tokens, and the turn simply ended. What I saw was a reply with no answer in it. That's not the model getting it wrong. That's my limit being too small. I've raised it to 32,768, and the next episode covers why.
Slow. On the same questions Qwen took 2 to 9 times as long as Ornith: over 5 minutes for a month, over 11 minutes for a year. It's worth the wait, as long as you're not in a hurry.
One more thing: Qwen called the trend "going down", while my answer key says "no clear trend". Its numbers were all right and the verdict is a judgment call, so I scored the numbers.
LFM2.5: out
LFM2.5 is pitched as a tool-calling model, but in this harness it couldn't get the call format right. It stuffed all of a tool's arguments into a single string, the harness kept answering "missing required property", and on one run it repeated the same mistake 58 times until the ten-minute timeout. It misread tables too: the health log has a column for wake-up time, and it read "08:33" as 8.33 hours of sleep. It was quick and correct on looking up projects and holdings, and wrong whenever it had to calculate or read a table. I stopped testing it there.
One open question: 30-odd numbers, one wrong
Finally I asked something with no answer key behind it: "Take a look at this folder and tell me what you know." This used another build of the same Qwen 27B (one that ships with its own MTP layers, more next episode), with thinking set only to Low.
It went through the whole folder and wrote a summary across money, health and projects: net worth, the brokerage account, value, weight and gain for all ten holdings, this month's spending, two months of sleep and workouts, and the status of six projects. I checked more than 30 numbers. One was wrong: it counted 12 workouts in September when there were 13 (the 10-minute fallback sessions count, and it missed one). Along the way it said it again: "my manual calculation was off by one — the script gives $3,671.91 for October."
That kind of accuracy with thinking set to Low suggests the accuracy comes from checking with code, not from thinking longer. It still costs time: that turn took 16 minutes, a good part of it spent drawing the interface.
Read-only doesn't mean "only this folder", again
Last episode I wrote that the agent's read-only mode blocks writes, not reads, after Ornith went browsing my code directory. It happened again this time, in a way that's worth more attention.
On Qwen's third trend run, it read the test folder normally and then called the harness's "skills" feature. My computer has skill files written for other AI tools, and one of them describes my real notes, path included. The harness hands those skills to the model automatically. The model loaded that skill, followed the path, read my real ledger, and used my real numbers to answer a question about a fake person.
The damage was limited. The model is local, so nothing left the machine. Read-only mode blocked writes, so not a word of my real notes changed. I threw that run away, and there's no real data anywhere in this post. But it shows something easy to miss: the working folder you give an agent isn't everything it can see. Features like skills and memory can lead a model to places you never meant to show it.
When I test now, I turn the skills feature off and check every run's log for paths outside the working folder.
How I split the work now
- Ornith: looking things up, checking current status, summarizing a note. Answers in under a minute.
- Qwen 27B: anything where the numbers have to be right and the data is private, like my real ledger. Hand it over, do something else, come back in ten minutes.
- Both: I still check the result myself.
That last line is the biggest thing these two episodes taught me. A local model is no different from any other AI: it makes mistakes, it's overconfident, and now and then it reads something it shouldn't. The better ones just check their own work.
Next
#3: making a 27B fast and stable on a 48GB Mac. The two kinds of MTP speculative decoding, the memory headroom TurboQuant buys, and the output limit that left a model with the right answer and no way to say it.
Caveats, as usual: one to three runs per question (only two valid trend runs for Qwen), only two arithmetic questions, and fake data in my own formats. The speeds aren't strictly comparable either: Qwen gets speculative decoding and Ornith has no draft model to use. This episode shows what these models can do, not what they do reliably.