Mo-Search Local-first desktop search

News · 2026-10-17

AI models on our real code: local vs cloud.

We build MoSearch, MoContext and MoVault with AI help every day, so we test the models on the same work: reviewing real changes, planning a feature, and making C# and C++ changes in our own codebase. Our code is not public, so these are real problems no model has trained on or been tuned to score well on, as can happen with public benchmarks. Here is how today's cloud models compare with models running free on a desktop PC.

New since October 14: two C++ benches, three more models, 83 more runs. What changed →

The test machine: one mini PC, 128 GB, no separate graphics card

Minisforum MS-S1 Recipe

Every local result on this page ran on a Minisforum MS-S1 MAX: an AMD Ryzen AI Max+ 395 (“Strix Halo”) with Radeon 8060S graphics and 128 GB of LPDDR5X-8000 shared by the CPU and GPU. In AMD’s Adrenalin app we give the GPU 96 GB of it, enough for a 27B model at Q6_K with a 256K-token context, or two mid-size models loaded at once. The models run in AMD’s Lemonade server (llama.cpp on Vulkan) under Windows 11, driven by the OpenCode coding agent. Nothing leaves the machine.

How we tune before we benchmark. Each setting change is timed on the same 6,000-token code prompt with a 256-token reply, three runs, comparing runs two and three (the first is a warm-up), then confirmed on a 20,000-token prompt. Writing speed is capped by memory bandwidth (about 256 GB/s), so tuning mostly moves prompt reading. One setting paid off: the micro-batch size, which helped the dense 27B model at 128 (+7%) and the 35B mixture-of-experts model at 1024 (+20%), opposite directions, so measure each model. Flash attention, full GPU offload and the Windows power mode were already at their best; the ROCm backend did not see the GPU, so we stay on Vulkan. Only then does each model run the eight benchmarks below, with fixed answer keys.

Composite
Single bench
AA weight

All benches

e.g. sol · 6.1 sol · sol, grok

Local · free, runs all dayCloud · paid per use

Local AI Speed on our benches

Average tokens per second over each model's bench runs, from llama.cpp's own counters. Reading is taking in the prompt and files; writing is generating, thinking included. These are long agent sessions, so they run slower than short-prompt tests. Sorted by combined speed.

Fun (and a little scary) things the models did

  • The one-line test file. Gemma 4 26B “fixed” a feature by rewriting a 1,000-line test file onto a single line. The build failed.
  • “All tests pass.” Claude Haiku 4.5 wrote a test that failed, then reported that every test passed. Nemotron 3.5 Bolt later did the same with two failing tests.
  • The fix that does nothing. Nemotron 3.5 Lightning was asked to make a second program follow the dark/light switch. It never sent that program a message, then said the change works immediately.
  • Two throttles, one bug. Asked why progress updates were never throttled, Nemotron 3.5 Lightning added a second, working throttle and left the broken one in place.
  • The missing line that wasn’t. Nemotron 3.5 Super held up a C++ change for a line it said was missing. The line was there, unchanged, just outside the part of the file the change touched.
  • Same chat, new model. Once, by accident, we switched models inside a chat that had just fixed a bug, and the new model went looking for its predecessor’s exact fix. We threw that run out, and we now check every session log for a single model.
  • Help wanted at 3 AM. Nemotron 3.5 Bolt tried to build by starting Visual Studio with an option that does not exist. Visual Studio opened a help window and waited for someone to click Close; the model waited too, idle, until our 2-hour limit stopped it. We now block Visual Studio in benchmark runs; the re-run took 20 minutes.
  • While I’m in here… Gemma 4 31B, working on the theme fix, also renamed an unrelated license message.
  • Confident and wrong. Swift-Qwen3.8 cited “documented .NET behavior” for its only finding. We ran it: .NET doesn’t do that.
  • “Ready to commit.” On our hardest review, GPT-6 Luna at medium, high and max effort all called a change with a blocking bug ready; max took 8 minutes to say so. Grok 4.6 found the blocker; the newer Grok 4.7 argued it was fine.
  • Thinking less took longer. Qwen3.8 at low thinking wrote one and a half to two times more text than at medium, so it was no faster.
  • A model that remembered the answers. Codex kept memories across our benchmark runs, so later GPT runs started out knowing what earlier runs had found. One “discovery” vanished when we re-ran it with memories off. Every affected run has been redone.
  • The blue screen. With nobody at the PC, Windows turned the screen off in the middle of a long local run; with the GPU at full load, the graphics driver missed its power deadline and Windows crashed. Long runs now keep the screen on.
  • And an honest one. GPT-5.6 Terra at xhigh effort could not run its tests because Windows blocked a freshly built file, and said exactly that instead of claiming success.

The benches

Every bench is a frozen snapshot of our own repository at a real moment in development, with history removed so a model cannot look up the answer. Each answer key comes from the real review or the real fix that shipped.

  • 1 · Review, hard. Coming in v26.1 A 13-file change adding MoContext's Folder / Whole index access ceiling (+880 lines). The key is the one blocker and three issues our real review found; all were fixed before the change was committed.
  • 2 · Review, medium. MoContext's daily automatic backup (7 files, +1,360 lines). Three issues from the real review, plus two bonus issues that AI reviewers found later.
  • 3 · Plan. Write the worker plan for that auto-backup from the orchestrator's prompt, scored out of 19 against the real plan review.
  • 4 · Code, apply a review. Apply three review fixes in C#, build, and pass the 166-test suite.
  • 5 · Code, small feature. A real item we are fixing: an unreachable backup folder should back off for an hour, not ten minutes. Change the code, add a test that would catch a regression, pass the suite.
  • 6 · Code, C++/MFC. Symptom only: "a running AutoIndex keeps its old colors when you switch MoSearch between dark and light". The model has to find the cross-process cause in MoSearch's C++ and fix it; scored against the fix we shipped.
  • 7 · Review, C++. Coming in v26.1 A small MoSearch change (8 files) so the Settings dialog's OK either saves or says it could not. Small and local, but the main issue needs reading dialog code the change does not touch. Scored out of 3 against our real review.
  • 8 · Code, C++ bug. Symptom only: "progress updates that should be throttled happen on every folder and file". The model finds the faulty comparison in MoSearch's C++ and fixes it; scored out of 6 against the fix we shipped.

Benches 7 and 8 were added in October 2026. Every local setup has now run all eight, and so have most cloud setups. A blue number after a setup's name counts the benches it has not run yet, so its average covers fewer benches.

How it is scored, and what to keep in mind

  • Quality is points earned against each bench's fixed answer key. Wrong claims cost points; a failing build or test caps a coding score at 50%. A run that did not finish, because it hit our time limit or the agent stopped with nothing to score, shows as DNF and counts as 0%. Composites average a setup's benches; time is their sum.
  • Who scored it. Claude (Anthropic's Opus 5.5) scored every run, including runs by Claude models, against those fixed keys. The keys came first, from our real reviews.
  • Real findings. Some review findings are real issues in our code. We fix them outside the benchmark, so the table words them plainly rather than in detail.
  • Small samples. One or two runs per setup. Reviews vary a lot from run to run, so close scores are within noise.
  • Times. Local times are measured. Cloud times are wall-clock readings taken while running each model in the editor.
  • Clean runs. Each cloud run starts in a new chat with the tool’s memory off, and we check its session log before scoring: one model, no outside tools, nothing read from outside the bench.
  • AA weight blends in the Artificial Analysis Intelligence Index (v4.3.2, read from their public model pages 2026-10-10 to 12), scaled so the best tested setup is 100%. Estimated entries (quantized local models, nearest effort setting) count at half weight and show hollow. Our ranking on the first six benches and the AA index agree closely (rank correlation 0.85), so the knob mostly moves the setups where they disagree. Nemotron 3.5 Super and Bolt use the scores from AA’s October 15 article on them.
  • Local setup. One desktop PC: AMD Ryzen AI Max+ 395 (Strix Halo) with 128 GB, 96 GB of it given to the GPU, running Lemonade (llama.cpp, Vulkan) and the OpenCode agent. Nemotron 3.5 Bolt and K2 Horizon need a newer llama.cpp (build b11702) than Lemonade includes, so those two ran on that build. Cloud models ran in VS Code and Cursor.
  • Local names are short: model, quant, setting, as in “Qwen3.6-35B (Q6, thinking)”. The setting is left out when it is the model’s default. When we rerun a model on a newer llama.cpp, the build joins its name (“b11702”) and the new run is the one we rank. The older run stays in the chart and table, dimmed and marked older build, so you can see what the newer build changed. Runs on different builds are never averaged together.

Setups

e.g. sol · 6.1 sol · sol, grok