From 75 Graph Splits to 3: SpacemiT's llama.cpp Fork Brings Gated Delta Net to the K1 and Cuts Model Loading 2.8×
On 18 November 2026 — the same day SpacemiT was showing the K1, K3 and V100 at the 5G+Industrial Internet Congress in Wuhan — two pull requests merged into the vendor's llama.cpp fork (spacemit-com/llama.cpp, branch mtmd-backend). They attack two completely different bottlenecks in running modern LLMs on the company's RISC-V boards: PR #71 is a two-line compute-path unlock that stops the scheduler from shattering a model's graph into dozens of fragments on the K1; PR #55 is an 81-line memory-path fix that stops you waiting two minutes for a 21 GiB model to load on the K3. Both carry measured numbers from real boards, and both are worth reading precisely because of what they admit they did not verify.
Why gated delta net matters on these boards
Gated delta net (GDN) is the linear-attention block that the Qwen3.5-family hybrid models interleave with ordinary full-attention layers. In llama.cpp it is the GATED_DELTA_NET operator. SpacemiT's fork implements it on its SPACEMIT ggml backend, which targets two core classes: A60 cores on the K1 (flagged use_ime1) and A100 cores on the K3 (flagged use_ime2). Until this week the operator was gated on use_ime2 only — K3 fine, K1 out of luck. On a K1, every GDN layer forced the backend to fall back to the CPU path, and the scheduler cut the graph around each linear-attention layer. The PR's own wording: for Qwen3.5-2B, 75 graph splits per decode; 38 during prompt processing.
PR #71: the two-line unlock
The fix changes the gating condition to use_ime2 || use_ime1 — consistent with how the other IME flags already work. Two lines added, one removed. It depends on PR #59 (merged 9 November, “strip-mine gated delta net over head size”): without that fix, a head size of 128 trips the S_v <= vlmax(e32m8) assertion at VLEN=256 — the K1's 256-bit vector length simply cannot hold the tile the K3 path was using.
The author measured on a real K1 board (toolchain v1.2.4, real TCM) against the pre-PR commit:
| Check | Before (base) | After (PR #71) | Delta |
|---|---|---|---|
| Decode graph splits (Qwen3.5-2B) | 75 | 3 | −96% |
| Prompt-processing splits (Qwen3.5-2B) | 38 | 2 | −95% |
| llama-bench pp512, Qwen3.5-2B Q4_0 (−t 4 −ub 128 −fa on) | 12.08 t/s | 13.50 t/s | +11.7% |
| llama-bench tg128, Qwen3.5-2B Q4_0 | 2.94 t/s | 3.74 t/s | +27.2% |
| llama-bench pp512, Qwen3.5-4B | 5.66 t/s | 6.25 t/s | +10.4% |
| llama-bench tg128, Qwen3.5-4B | 1.34 t/s | 1.77 t/s | +32.1% (splits 99 → 3) |
Correctness was checked, not assumed: test-backend-ops -b SPACEMIT0 -o GATED_DELTA_NET passed 70/70 on the board, including extra cases the PR author ran locally for head sizes 96/128/256, KDA, K>1 and the chunked path — all of which the pre-PR build reported as unsupported. For Qwen3.5-2B Q4_0 over 15×512-token runs, mean KLD against the CPU reference was 0.000742, top-1 agreement 98.88% unchanged, perplexity 7.7666 vs 7.7641. The 64-token greedy output of Qwen3.5-4B is identical; for the 2B model the first 38 words match, then it picks a synonym (“surged” vs “swelled”) and diverges — the kind of honest variance report you want to see rather than a suspicious “bit-identical”.
PR #55: the two-minute load problem
The second bottleneck is less glamorous and equally real. Loading Qwen3.6-35B-A3B in Q4_K_M (21.1 GiB, 22,663,387,424 bytes, an Unsloth quant mixing Q4_K/Q5_K/Q6_K/Q8_0) took about 120 seconds on a K3 Pico-ITX — of which roughly 106 s was single-threaded weight repacking inside set_tensor, with fifteen of the sixteen cores idle. The PR adds spacemit_repack_parallel_for, distributing row blocks over up to 8 threads, for the Q4_K/Q5_K/Q6_K kernels only, behind a build flag (-DGGML_SPACEMIT_REPACK_MT, default off). During review it grew a dynamic backend selector that reads spert::backend_info().core_arch_id on first invocation (A60, id 0xA03C → spacemit-k1; the K3 board reported 0xa064) and dispatches per-expert tiles for 3-D MoE tensors, reusing the existing repack kernels untouched.
| Measurement | Serial (off) | Parallel (on) | Ratio |
|---|---|---|---|
35B load, pass 1 (wall time, llama-bench -p 16 -n 0 -r 1) | 122.4–124.5 s | 43.9–44.9 s | 0.36× |
| 35B load, reversed order pass | 133.4 s | 43.8 s | 0.344× |
| 35B load, final v4 with backend dispatch | 125.7 s | 54.6 s | 0.43× |
| Inference regression | — | — | pp512 1.001×, tg128 1.003× |
Repacked bytes are bit-identical: the verification harness hashes full repacked buffers across Q4_K/Q4_0/Q4_1/Q5_K/Q6_K/Q8_0, 2-D and 3-D MoE tensors, and every hash matches the serial path; 64-token greedy decoding is unchanged. One subtle bug the review caught is worth quoting because it generalizes: the expert stride must be computed from the kernel's actual layout size, not the tensor's allocated size — Q4_K allocations pad every 256 columns to 160 bytes while the layout only uses 152, so an allocation-based stride silently misaligns every expert after the first.
The GDN series so far
Neither PR is an isolated event; they are the visible tail of a month of work on the fork's GDN path:
| PR | Merged | What it did |
|---|---|---|
| #48 “Perf/k3 gdn decode state” | 16 Oct | K3 decode-state work for GDN |
| #53 “optimize MoE matmul and add dense Q4_0 F16 GEMM” | 9 Nov | MoE matmul path (later reused by the parallel repack) |
| #54 “optimize recurrent GDN with RVV and TCM” | 9 Nov | Recurrent GDN kernels on RVV + tightly-coupled memory |
| #59 “strip-mine gated delta net over head size” | 9 Nov | Head-size tiling — the hard prerequisite for head 128 at VLEN 256 |
| #64 “select TCM-fitting M/N split for prefill” | 13 Nov | Prefill tiling to fit TCM |
| #71 “enable gated delta net on A60 (K1)” | 18 Nov | Unlocks the K1 compute path |
| #55 “parallel repack” | 18 Nov | Load-time fix, both boards |
What the PRs do not claim
| Open item | Why it matters |
|---|---|
| The repack flag is default off | -DGGML_SPACEMIT_REPACK_MT must be enabled at build time; stock builds keep the serial path |
| K1 branch of the parallel repack not tested on hardware | The author had no K1 board; the K1 dispatch path (A60) is unverified in real runs |
spacemit-k3-x100 backend binds only cores 4–7 | Documented default, so the dispatched version (54.6 s) is slower than 8 plain std::thread workers (44–45 s); whether to claim more cores via StreamConfig is unresolved |
| 2-D tensors still repack serially | Only 3-D MoE expert tensors take the parallel path |
Other *_32_bl kernels untouched | The same parallelization applies but is future work |
| All throughput numbers are self-reported | llama-bench commands are published and reproducible, but no third-party ran them |
Why this matters
The Wuhan congress same-day story is the marketing frame: K1 as a robot's “cerebellum”, K3 at 60 TOPS, V100 servers in single-cabinet inference racks for small enterprises. What PRs #71 and #55 show is the unglamorous half of that pitch actually being built — edge inference on RISC-V is being won in vendor backends one graph split and one repack thread at a time. Two details deserve the label “structural”. First, the K1's limitation was never the ISA: the vector unit exists, the kernels exist — a gating condition kept the operator off the board, and the scheduler's fragmentation (75 splits) was the visible symptom. Second, both PRs were written with disclosed AI assistance and validated on real boards with reproducible commands — the fork's review culture (hash tables, KLD runs, split counts) is quietly becoming as much a part of the product as the kernels themselves.
Practical read for K1/K3 users: if you run Qwen3.5-family models, pull a post-18-November build and expect the tg128 gains on K1; if you load 30B-class MoE models on K3, build with the repack flag and expect roughly a 2.3–2.9× faster load depending on the final configuration — and check whether your build's backend core binding matches the docs' four-core default before comparing numbers.
Sources
- PR #71, “ggml-spacemit : enable gated delta net on A60 (K1)”, merged 18 Nov 2026 — github.com/spacemit-com/llama.cpp/pull/71
- PR #55, “ggml-spacemit : 优化repack模式,改为并行repack”, merged 18 Nov 2026 — github.com/spacemit-com/llama.cpp/pull/55
- PR #59, “strip-mine gated delta net over head size”, merged 9 Nov 2026 (hard dependency of #71) — github.com/spacemit-com/llama.cpp/pull/59
- SpacemiT at the 2026 5G+Industrial Internet Congress, Wuhan, 极目新闻, 18 Nov 2026 (K1/K3/V100 showcase context) — via 163.com/dy/article/LC243FCC053469LG.html
- llama.cpp upstream context — github.com/ggml-org/llama.cpp
2026 年 11 月 18 日,进迭时空在武汉 5G+工业互联网大会展出 K1/K3/V100 的同一天,官方 llama.cpp 分支(spacemit-com/llama.cpp,mtmd-backend 分支)合并了两笔 PR,分别打向两个完全不同的瓶颈:PR #71(+2/−1 行)把 GATED_DELTA_NET 算子从仅限 K3 的 A100 核(use_ime2)开放到 K1 的 A60 核(条件改为 use_ime2 || use_ime1);PR #55(+81 行)把 K3 上 21.1 GiB 模型约 120 秒的加载时间砍到约 44 秒。
PR #71 实测(真实 K1 板,toolchain v1.2.4):Qwen3.5-2B 每次解码的 graph 分片从 75 降到 3(预填充 38→2);llama-bench tg128 从 2.94 提到 3.74 t/s(+27%),4B 模型 tg128 1.34→1.77 t/s(+32%,分片 99→3);GATED_DELTA_NET 后端测试 70/70 通过(含 head 96/128/256、KDA、K>1、chunked 路径),Qwen3.5-2B Q4_0 对 CPU 参考的 KLD 0.000742、top-1 一致率 98.88% 不变。依赖 #59(head size 128 在 VLEN=256 下会触发 S_v <= vlmax(e32m8) 断言,11-09 已合并)。PR #55:把 Q4_K/Q5_K/Q6_K 的 repack 行块分给最多 8 线程(构建开关 -DGGML_SPACEMIT_REPACK_MT,默认关),120→44 秒(2.8–2.9 倍),repack 后字节逐位一致,推理速度不变(pp512 1.001×);评审中加入了按 spert::backend_info().core_arch_id 自动选后端(A60=0xA03C→spacemit-k1)的 3-D MoE 按专家 tile 分发。
诚实点:+27–32% 只对含 GDN 层的 Qwen3.5 系模型有效,全注意力模型无收益;repack 开关默认关闭;并行 repack 的 K1 分支因作者没有 K1 板未做硬件验证;spacemit-k3-x100 后端默认只绑 cpu 4–7(派发版 54.6 s 慢于 8 线程版的 44–45 s);2-D 张量仍走串行;吞吐数字全部为 PR 作者自报(llama-bench 命令已给出可复现)。这是进迭时空一个月 GDN 优化链(#48→#53/#54/#59→#64→#71/#55)的收口:K1 的限制从来不是 ISA,而是一个让算子不上板的开关条件。
18 ноября 2026 года — в тот же день, когда SpacemiT показывала K1/K3/V100 на конгрессе 5G+промышленный интернет в Ухане, — в вендорский форк llama.cpp (spacemit-com/llama.cpp, ветка mtmd-backend) влились два PR, бьющих по совершенно разным узким местам: PR #71 (+2/−1 строки) открывает оператор GATED_DELTA_NET для ядер A60 на K1 (условие меняется на use_ime2 || use_ime1), а PR #55 (+81 строка) сокращает загрузку модели 21.1 ГиБ с ~120 с до ~44 с на K3.
Измерения #71 на реальной плате K1 (toolchain v1.2.4): разбиения графа при декодировании Qwen3.5-2B сокращаются с 75 до 3 (prefill 38→2); tg128 вырастает с 2,94 до 3,74 ток/с (+27%), для 4B — 1,34→1,77 ток/с (+32%, разбиений 99→3); тест GATED_DELTA_NET 70/70, KLD 0,000742 к CPU-референсу, совпадение top-1 98,88% без изменений. Зависимость — #59 от 9 ноября (head size 128 при VLEN=256 ломал assert S_v <= vlmax(e32m8)). PR #55: repack для Q4_K/Q5_K/Q6_K распараллелен до 8 потоков (флаг -DGGML_SPACEMIT_REPACK_MT, по умолчанию выключен), 2,8–2,9× быстрее, байты побитово идентичны, скорость инференса не изменилась; при ревью добавлен выбор бэкенда по spert::backend_info().core_arch_id (A60=0xA03C→spacemit-k1).
Честные оговорки: прирост +27–32% касается только моделей с GDN-слоями (семейство Qwen3.5); флаг repack по умолчанию выключен; ветка K1 параллельного repack не проверена на железе (у автора не было платы K1); бэкенд spacemit-k3-x100 по умолчанию привязан к ядрам 4–7 (54,6 с против 44–45 с у 8 потоков); 2-D тензоры всё ещё обрабатываются серийно; все числа — самостоятельные замеры авторов PR (команды llama-bench опубликованы). Итог месячной серии GDN-оптимизаций: ограничение K1 никогда не было в ISA — это было условие-переключатель, не пускавшее оператор на плату.
El 18 de noviembre de 2026, el mismo día en que SpacemiT exhibía K1/K3/V100 en el congreso 5G+Internet Industrial de Wuhan, se fusionaron en su fork de llama.cpp (spacemit-com/llama.cpp, rama mtmd-backend) dos PR que atacan cuellos de botella totalmente distintos: el PR #71 (+2/−1 líneas) habilita el operador GATED_DELTA_NET en los núcleos A60 del K1 (la condición pasa a use_ime2 || use_ime1), y el PR #55 (+81 líneas) reduce de ~120 s a ~44 s la carga de un modelo de 21,1 GiB en el K3.
Mediciones de #71 sobre una placa K1 real (toolchain v1.2.4): las particiones de grafo por decodificación en Qwen3.5-2B bajan de 75 a 3 (prefill 38→2); tg128 sube de 2,94 a 3,74 t/s (+27 %), y en el 4B de 1,34 a 1,77 t/s (+32 %, particiones 99→3); test GATED_DELTA_NET 70/70, KLD 0,000742 frente a la referencia CPU, top-1 del 98,88 % sin cambios. Depende de #59 (9 de noviembre: head size 128 con VLEN=256 rompía el assert S_v <= vlmax(e32m8)). PR #55: repack de Q4_K/Q5_K/Q6_K paralelizado hasta 8 hilos (flag -DGGML_SPACEMIT_REPACK_MT, desactivado por defecto), 2,8–2,9× más rápido, bytes bit-idénticos y velocidad de inferencia intacta (pp512 1,001×); en revisión se añadió selección de backend vía spert::backend_info().core_arch_id (A60=0xA03C→spacemit-k1).
Honestidad: las ganancias del +27–32 % solo valen para modelos con capas GDN (familia Qwen3.5); el flag de repack está apagado por defecto; la rama K1 del repack paralelo no se probó en hardware (el autor no tenía placa K1); el backend spacemit-k3-x100 se une por defecto a los núcleos 4–7 (54,6 s frente a 44–45 s con 8 hilos); los tensores 2-D siguen en serie; todas las cifras son mediciones propias de los autores (comandos llama-bench publicados). Cierre de un mes de trabajo GDN: la limitación del K1 nunca fue el ISA, sino una condición que impedía que el operador llegara a la placa.
Le 18 novembre 2026, alors même que SpacemiT présentait K1/K3/V100 au congrès 5G+Internet industriel de Wuhan, deux PR ont été fusionnés dans son fork de llama.cpp (spacemit-com/llama.cpp, branche mtmd-backend), visant deux goulots d'étranglement très différents : le PR #71 (+2/−1 lignes) active l'opérateur GATED_DELTA_NET sur les cœurs A60 du K1 (la condition devient use_ime2 || use_ime1), et le PR #55 (+81 lignes) ramène de ~120 s à ~44 s le chargement d'un modèle de 21,1 Gio sur le K3.
Mesures de #71 sur une vraie carte K1 (toolchain v1.2.4) : les découpages de graphe par décodage passent de 75 à 3 pour Qwen3.5-2B (prefill 38→2) ; tg128 grimpe de 2,94 à 3,74 t/s (+27 %), et de 1,34 à 1,77 t/s pour le 4B (+32 %, découpages 99→3) ; test GATED_DELTA_NET 70/70, KLD 0,000742 contre la référence CPU, top-1 à 98,88 % inchangé. Dépendance : #59 (9 novembre — head size 128 avec VLEN=256 déclenchait l'assert S_v <= vlmax(e32m8)). PR #55 : le repack de Q4_K/Q5_K/Q6_K parallélisé jusqu'à 8 threads (flag -DGGML_SPACEMIT_REPACK_MT, désactivé par défaut), 2,8–2,9× plus rapide, octets bit-identiques et vitesse d'inférence inchangée (pp512 1,001×) ; en revue, sélection dynamique du backend via spert::backend_info().core_arch_id (A60=0xA03C→spacemit-k1).
Honnêteté : les gains de +27–32 % ne concernent que les modèles à couches GDN (famille Qwen3.5) ; le flag de repack est désactivé par défaut ; la branche K1 du repack parallèle n'a pas été testée sur matériel (l'auteur n'avait pas de carte K1) ; le backend spacemit-k3-x100 se lie par défaut aux cœurs 4–7 (54,6 s contre 44–45 s avec 8 threads) ; les tenseurs 2-D restent sériels ; tous les chiffres sont des mesures des auteurs (commandes llama-bench publiées). aboutissement d'un mois de travail GDN : la limite du K1 n'a jamais été l'ISA, mais une condition qui empêchait l'opérateur d'atteindre la carte.
Am 18. November 2026 — am selben Tag, an dem SpacemiT K1/K3/V100 auf dem 5G+Industrie-Internet-Kongress in Wuhan zeigte — sind zwei PRs in den Vendor-Fork von llama.cpp (spacemit-com/llama.cpp, Branch mtmd-backend) gemerged worden, die zwei völlig verschiedene Engpässe angreifen: PR #71 (+2/−1 Zeilen) schaltet den Operator GATED_DELTA_NET für die A60-Kerne des K1 frei (Bedingung wird zu use_ime2 || use_ime1), und PR #55 (+81 Zeilen) senkt die Ladezeit eines 21,1-GiB-Modells auf dem K3 von ~120 s auf ~44 s.
Messungen zu #71 auf einer echten K1-Platine (toolchain v1.2.4): Die Graph-Aufteilungen pro Decode sinken bei Qwen3.5-2B von 75 auf 3 (Prefill 38→2); tg128 steigt von 2,94 auf 3,74 t/s (+27 %), beim 4B von 1,34 auf 1,77 t/s (+32 %, Aufteilungen 99→3); GATED_DELTA_NET-Test 70/70, KLD 0,000742 gegenüber der CPU-Referenz, Top-1-Übereinstimmung 98,88 % unverändert. Abhängigkeit: #59 (9. November — Head Size 128 mit VLEN=256 brach die Assert S_v <= vlmax(e32m8)). PR #55: Repack für Q4_K/Q5_K/Q6_K auf bis zu 8 Threads parallelisiert (Flag -DGGML_SPACEMIT_REPACK_MT, standardmäßig aus), 2,8–2,9× schneller, Bytes bitidentisch, Inferenzgeschwindigkeit unverändert (pp512 1,001×); im Review wurde eine Backend-Auswahl über spert::backend_info().core_arch_id ergänzt (A60=0xA03C→spacemit-k1).
Ehrlichkeit: die +27–32 % gelten nur für Modelle mit GDN-Schichten (Qwen3.5-Familie); das Repack-Flag ist standardmäßig aus; der K1-Zweig des parallelen Repacks wurde mangels K1-Platine nicht auf Hardware geprüft; das Backend spacemit-k3-x100 bindet standardmäßig nur die Kerne 4–7 (54,6 s gegenüber 44–45 s mit 8 Threads); 2-D-Tensoren bleiben seriell; alle Zahlen sind Selbstmessungen der PR-Autoren (llama-bench-Befehle veröffentlicht). Abschluss eines Monats GDN-Arbeit: Die Grenze des K1 war nie die ISA, sondern eine Schaltbedingung, die den Operator von der Platine fernhielt.
در ۱۸ نوامبر ۲۰۲۶ — همان روزی که SpacemiT در کنگرهٔ 5G+اینترنت صنعتی ووهان K1/K3/V100 را به نمایش گذاشت — دو PR در فورک llama.cpp این شرکت (spacemit-com/llama.cpp، شاخهٔ mtmd-backend) ادغام شدند که به دو گلوگاه کاملاً متفاوت حمله میکنند: PR شمارهٔ ۷۱ (+۲/−۱ خط) عملگر GATED_DELTA_NET را روی هستههای A60 برد K1 آزاد میکند (شرط به use_ime2 || use_ime1 تغییر میکند) و PR شمارهٔ ۵۵ (+۸۱ خط) زمان بارگذاری مدل ۲۱٫۱ گیگابایتی روی K3 را از ~۱۲۰ ثانیه به ~۴۴ ثانیه میرساند.
اندازهگیریهای #71 روی برد واقعی K1 (toolchain v1.2.4): تقسیمهای گراف در هر decode برای Qwen3.5-2B از ۷۵ به ۳ میرسد (prefill از ۳۸ به ۲)؛ tg128 از ۲٫۹۴ به ۳٫۷۴ توکن بر ثانیه (+۲۷٪) و برای مدل 4B از ۱٫۳۴ به ۱٫۷۷ (+۳۲٪، تقسیمها ۹۹→۳) میرسد؛ تست GATED_DELTA_NET با نتیجهٔ ۷۰/۷۰، KLD برابر ۰٫۰۰۰۷۴۲ نسبت به مرجع CPU و تطابق top-1 برابر ۹۸٫۸۸٪ بدون تغییر. وابستگی: #59 (۹ نوامبر — head size 128 با VLEN=256 باعث شکست assert میشد). PR شمارهٔ ۵۵: repack برای Q4_K/Q5_K/Q6_K تا ۸ رشته موازی شده (کلید ساخت -DGGML_SPACEMIT_REPACK_MT، بهصورت پیشفرض خاموش)، ۲٫۸–۲٫۹ برابر سریعتر، بایتها دقیقاً یکسان و سرعت استنتاج بدون تغییر؛ در بازبینی، انتخاب خودکار بکاند از طریق spert::backend_info().core_arch_id اضافه شد (A60=0xA03C→spacemit-k1).
صداقت: بهبودهای +۲۷–۳۲٪ فقط برای مدلهای دارای لایهٔ GDN (خانوادهٔ Qwen3.5) کاربرد دارد؛ کلید repack بهصورت پیشفرض خاموش است؛ شاخهٔ K1 در repack موازی بهدلیل نبود برد K1 روی سختافزار آزمایش نشده؛ بکاند spacemit-k3-x100 بهصورت پیشفرض فقط به هستههای ۴ تا ۷ متصل میشود (۵۴٫۶ ثانیه در برابر ۴۴–۴۵ ثانیه با ۸ رشته)؛ تانسورهای 2-D هنوز سریالیاند؛ همهٔ اعداد اندازهگیریهای خودِ نویسندگان PR است (دستورات llama-bench منتشر شده). حاصل یک ماه کار روی GDN: محدودیت K1 هرگز ISA نبود، بلکه یک شرط کلیدی بود که اجازهٔ رسیدن عملگر به برد را نمیداد.