Running Real LLM Inference on SpacemiT K1: ONNX Runtime MatMulNBits via IME

Breakthrough: Real LLM Decode on K1 via IME

The SpacemiT K1 SoC packs 8 X60 cores at 1.6 GHz with a 2 TOPS AI engine, but actually running large language model inference efficiently on RISC-V hardware has been a challenge. A community developer known as opensolvers has demonstrated real LLM decoding on the Orange Pi RV2 (K1-based) using ONNX Runtime's MatMulNBits operator accelerated by the X60's Integer Matrix Extension (IME) -- the smt.vmadot instruction.

This is not a synthetic benchmark. It runs complete generation through real ONNX graphs of Qwen2.5-0.5B, SmolLM2-360M, and TinyLlama-1.1B.

Benchmark Results

All numbers are greedy decode (ms/token, lower is better) through a patched ORT 1.29.0 with accuracy_level=4 selecting the CompInt8 IME path. Pinned to cluster 0 (taskset -c 0-3).

Int8 is roughly 1.7-2x slower than int4 on the same model, which is expected given the ~2x weight traffic. Both paths now hit the IME hardware instead of falling back to a multi-second generic path.

The Journey: From 16 Seconds to 80 Milliseconds

The most dramatic improvement was on Qwen2.5-0.5B:

Prefill on the BlkLen=128 Qwen path dropped from ~18s to ~3.9s (~4.7x) once CompInt8 was selected.

How It Works: Five Key Steps

1. Select the Right Compute Path -- accuracy_level=4

ONNX Runtime picks the compute variant in matmul_nbits.cc based on the accuracy_level attribute:

• accuracy_level=4 selects SQNBIT_CompInt8 -- the IME smt.vmadot path
• Any other value selects SQNBIT_CompFp32 -- a fallback that dequantizes to fp32 and uses generic SGEMM, completely bypassing IME

Many generated ONNX graphs had zero accuracy_level attributes. Roofline analysis showed the workload was 178-341x above the STREAM bandwidth floor -- clear evidence of the wrong path, not a slow kernel. Simply setting accuracy_level=4 yielded a 9.1x speedup with no kernel change. A hand-written M<4 RVV "fast path" inside the IME kernel was tested but measured 28% slower than the stock smt.vmadot tile -- configuration beat micro-optimization.

2. Pack for Decode -- Q4x16 m1pack (BlkLen=32)

Stock CompInt8 still left decode bandwidth on the table. The solution packs weights once into Q4_0x16 panels (llama.cpp layout) and runs dedicated M=1 assembly plus M>=4 gather paths. Isolated microbenchmarks reach ~10 GOP/s single-thread.

3. Match Real Exports -- BlkLen=128 CompInt8

AMD's Qwen2.5-0.5B package uses block_size=128, not 32. The Q4x16 panel path does not apply. A BlkLen=128 CompInt8 path was added (column-major pack + RVV nibble gather + IME) so accuracy_level=4 actually runs IME on that graph. A truncated RVV unpack that only processed 64 of 128 nibbles initially produced garbage logits -- a subtle bug that was caught and fixed.

4. Int8 Needed Its Own Path -- SQ8Bit IME

bits=8 MatMulNBits had no X60 CompInt8 backend -- Qwen int8 sat at multi-second per token. The team added SQ8Bit CompInt8: pack signed B' = B-128, flat scales, width-16 block sums, kernel via smt.vmadot. This alone brought Qwen int8 from multi-second to ~1.2s (1t) / ~0.4s (4t).

5. Close the Gap -- Q8x16 M1 Panels

Mirroring the int4 panel trick for int8: Llama i8i8 layout (16 columns x k-block with fp16 scales + tiles), M1 assembly gemm_m1_panel_q8x16, M>=4 gather from panels, and a resolve helper for BlkSum/scales. Qwen int8 drops to ~241/159 ms.

Reproduction

The complete benchmark source is available on GitHub:

# On the X60 board: apply m1pack IME patches, rebuild ORT bash apply-ime-m1pack.sh make -C $ORT_BUILD onnxruntime_mlas onnxruntime # Force CompInt8 on any MatMulNBits graph python3 patch_accuracy_level.py model.onnx model_acc4.onnx # Real decode bash run-real-llm-ort.sh # Qwen-shaped defaults bash run-smollm2-ort.sh # SmolLM2-360M bash run-tinyllama-ort.sh # TinyLlama-1.1B Toolchain: ONNX Runtime 1.29.0, X60 smt.vmadot (assembler-only), -march=rv64gcv_zvl256b_zfh_zvfh.

Key Takeaways

1. Results on real models first -- synthetic FFN found the path, but Qwen/SmolLM2/TinyLlama are the real scoreboard.
2. Prove the code runs -- probes + roofline analysis beat weeks of kernel tuning on a dead path.
3. Config, then pack, then panels -- accuracy_level=4 selects CompInt8; block length must match the pack format; M1 panels recover decode performance.
4. Int8 needed its own ship path -- SQ8Bit + Q8x16, not "hope int4 helps."
5. Grep the ONNX -- missing attributes and broken embed wiring look like slow kernels until you read the bytes.

What This Means for the K1 Ecosystem

The SpacemiT K1 was already known for its 50K DMIPS compute capability and OpenWrt 23.05 support. This work demonstrates that the K1's IME hardware can deliver practical LLM inference at usable speeds -- ~80 ms/token for a 0.5B parameter model is fast enough for interactive applications. Combined with the K1's low power consumption, this opens possibilities for edge AI deployment where a dedicated GPU is impractical.

The CH32V series MCUs from WCH (like the CH32V003 at $0.1 or CH32V307 with gigabit Ethernet) handle the peripheral and sensing layer, while the K1 can serve as the edge AI inference engine. This complementary architecture is exactly what the RISC-V ecosystem needs for real-world IoT and edge computing solutions.

---

Source: SpacemiT Forum Topic 1612 by opensolvers. Full technical writeup: OpenSolvers -- ONNX Runtime on X60. Benchmark code: github.com/opensolvers/benchmarks.