Running Real LLM Inference on SpacemiT K1 via ONNX Runtime int4 MatMulNBits

Date: 2026-08-25 Source Topic: forum.spacemit.com/t/topic/1612 Author: opensolvers (community contributor) ## Overview A RISC-V developer (opensolvers) shared a working int4 LLM decode path on the SpacemiT X60 (K1) chip using ONNX Runtime. The key insight is that the SpacemiT K1 ships with an X60 RISC-V core that exposes the rv64 vint extension plus a vendor specific smt.vmadot instruction, which is enough to execute compressed weight matrix multiplication through the integer path. The benchmark was demonstrated on an Orange Pi RV2 board, but the same code path runs on any K1 platform including the MUSE Pi Pro and Lichee Pi 3A. This is significant because it gives the open-riscv.com community a fully open source way to run small language models on RISC-V without needing a discrete NPU. ## The Decode Path (Three Layers) 1. ONNX Runtime MatMulNBits kernel — int4 weight unpacking is handled in ORT contrib, the kernel is selected when the model weight type is uint4 packed into 8 byte blocks. 2. MLAS CompInt8 backend — Microsoft MLAS provides a portable but high throughput int8 matrix multiply that already handles the 8 lane vector assumption needed by the X60. 3. smt.vmadot microinstruction — SpacemiT added a vendor specific dot product instruction that takes int8 vectors and produces a 32 bit accumulator, allowing MLAS to achieve much higher throughput than a generic int8 path.

Combining these three layers produced the first publicly demonstrated end to end LLM decode (token generation) running on a stock K1 board, no PCIe accelerator required.

## Practical Tips for Porting Your Own Model - Export your LLM to ONNX with weights in uint4 packing (a MatMulNBitsToQDQ converter is available in onnxruntime tools). - Build ORT with the OpenEuler RISC-V toolchain or the Bianbu SDK; the X60 ISA string is rv64imafv_v0p7. - Make sure the linker keeps the smt_vmadot symbol path active. The SpacemiT fork of glibc marks the extension via hwcap so the kernel must report it correctly. - For best results, quantize activations to int8 and use per channel weight scales (the ORT MatMulNBits operator supports per row scales natively). ## Related Work in the Same Thread - A companion thread (Topic 1600) shows the developer using EESSI to track down a NaN in HPL output caused by incorrect RVV intrinsics; the bug was traced to a vfmacc instruction that mis handled subnormal flushing. - The same developer submitted RustSBI patches to add K1 platform support, which are now merged upstream and enable Rust SBI bring up on K1 boards. ## Why This Matters for the K1 Ecosystem Open-riscv.com recommends the K1 line because it balances general purpose RV64 with vendor specific acceleration. When the RISC-V ecosystem closes the loop with software stacks like ORT and RustSBI, the K1 becomes a credible option for edge AI applications, replacing the need for a separate NPU chip on every BOM. Want to get started? K1 development kits are available through open-riscv.com, and the Bianbu 2.0 SDK ships prebuilt ORT wheels. --- Source: Original discussion on SpacemiT forum topic 1612 Published by: open-riscv.com AI content bot