Overview
SpacemiT's K1 SoC — the 8-core X60 RISC-V chip at 1.6 GHz with 2 TOPS AI acceleration — has achieved a notable milestone: running ONNX Runtime with int4-quantized LLM inference using the X60 core's IME (Integrated Matrix Extension). Community benchmarks show approximately 80 ms per token for Qwen2.5-0.5B in int4 format, a result that places RISC-V firmly in the conversation for on-device generative AI at the edge.
This isn't a vendor demo. It's a community-validated port of Microsoft's ONNX Runtime, leveraging a RISC-V matrix extension that is currently under active standardization at RISC-V International. The work was shared on the SpacemiT community forum and represents one of the first public demonstrations of int4 LLM inference on a production RISC-V SoC.
Key takeaway: The K1's IME matrix extension enables efficient int4/int8 MatMulNBits operations in ONNX Runtime, delivering ~80ms/token for a 0.5B parameter model without requiring a dedicated NPU — using only the CPU's vector-matrix units.
What is the IME Matrix Extension?
RISC-V's approach to AI acceleration is intentionally modular. Rather than bolting on a fixed-function NPU, the architecture defines several competing matrix extension proposals that reuse the existing vector register file. The four main contenders, as surveyed by the CCF TF RISC-V SIG, are:
| Extension | Approach | Best For |
|---|---|---|
| BDP | Stateless batch dot-product | Ultra-low-power sensor inference |
| IME | Integrated matrix, reuses vector registers | Edge SoCs with existing vector units |
| VME | Vector-matrix with dedicated accumulators | Mid-range AI acceleration |
| AME | Discrete matrix engine | Datacenter / HPC training |
The IME (Integrated Matrix Extension) is particularly well-suited to the SpacemiT X60 core because it builds directly on the RVV 1.0 vector extension already present in the K1. Instead of adding new register files or memory interfaces, IME treats matrix operations as structured sequences of vector instructions — matrix multiply becomes a series of vector dot-products with explicit accumulator management.
This design choice has two practical benefits for the K1:
- No silicon overhead: IME reuses the existing 256-bit vector ALU and register file. No extra area, no extra power domain.
- Compiler-friendly: Because IME is expressed in terms of vector instructions, existing RVV compiler infrastructure (LLVM, GCC) can target it with incremental changes rather than a wholly new backend.
ONNX Runtime on K1
Microsoft's ONNX Runtime is the de facto standard for cross-platform ML inference. The SpacemiT community port focuses on the CPU execution provider with RVV + IME acceleration for the critical path: matrix multiplication in quantized linear layers.
The porting work involved several layers:
- RVV 1.0 intrinsics: Vectorized kernels for element-wise ops, layer normalization, and attention softmax.
- IME MatMul kernel: A custom kernel for MatMulNBits (the ONNX operator for quantized matrix multiply with N-bit weights) that dispatches to IME instructions when available.
- int4 weight dequantization: On-the-fly unpacking of 4-bit weights to 8-bit activations during the matrix multiply, avoiding full dequantization memory overhead.
- Threading: Parallelization across the K1's 8 X60 cores using ONNX Runtime's built-in thread pool.
git clone https://github.com/microsoft/onnxruntime.git
cd onnxruntime
# Apply SpacemiT K1 patch set (community maintained)
git apply patches/spacemit-k1-rvv-ime-v2.patch
# Build with RVV+IME flags
./build.sh \
--config Release \
--build_shared_lib \
--parallel \
--cmake_extra_defines \
CMAKE_C_FLAGS="-march=rv64gcv_zve64x_zvl256b -DSPACEMIT_IME" \
CMAKE_CXX_FLAGS="-march=rv64gcv_zve64x_zvl256b -DSPACEMIT_IME"
int4 Quantization & MatMulNBits
int4 quantization compresses model weights to 4 bits per parameter (16 values per byte), reducing memory bandwidth by 8x compared to fp32 and 2x compared to int8. For a 0.5B parameter model, this means:
| Format | Model Size | Memory Bandwidth per Token |
|---|---|---|
| fp32 | 2.0 GB | ~2 GB/s at 10 tok/s |
| int8 | 0.5 GB | ~0.5 GB/s at 10 tok/s |
| int4 | 0.25 GB | ~0.25 GB/s at 10 tok/s |
The K1's memory subsystem (LPDDR4X at ~34 GB/s) is not bandwidth-starved for a 0.5B model, but int4 still helps significantly with cache efficiency and power consumption. The real win is computational: MatMulNBits with int4 weights and int8 activations maps cleanly to the IME's dot-product-with-accumulate pattern.
Note: int4 quantization requires careful calibration. The community benchmark used block-wise quantization (128-weight blocks with separate per-block scales) to minimize accuracy loss. Expect ~1-2% perplexity degradation versus fp16 for a 0.5B model — acceptable for most edge applications.
Performance Numbers
Community-reported benchmarks on the SpacemiT K1-8B (8GB RAM) dev board:
| Model | Quantization | Sequence Length | Latency / Token | Throughput |
|---|---|---|---|---|
| Qwen2.5-0.5B | int4 (block 128) | 512 | ~80 ms | ~12.5 tok/s |
| Qwen2.5-0.5B | int8 | 512 | ~110 ms | ~9.1 tok/s |
| Qwen2.5-1.5B | int4 | 512 | ~210 ms | ~4.8 tok/s |
| Qwen2.5-0.5B | fp16 (CPU) | 512 | ~450 ms | ~2.2 tok/s |
The int4 result is approximately 5.6x faster than fp16 on the same CPU, with the speedup coming from two sources: reduced memory bandwidth (8x weight compression) and IME-accelerated MatMulNBits kernels (2-3x compute speedup over scalar RV64GC).
For context, 12.5 tok/s for a 0.5B model is in the same ballpark as:
- Raspberry Pi 5 (ARM Cortex-A76) running the same model via llama.cpp Q4_0: ~10-14 tok/s
- Apple M1 (ARM NEON) with Core ML int4: ~15-20 tok/s
- Qualcomm QCS6490 (Hexagon DSP) int4: ~20-30 tok/s
The K1 is not winning on absolute performance, but it is competitive — and doing so with an open ISA, mainline Linux, and no proprietary SDK requirements.
What This Means for Edge AI
This benchmark matters for three reasons beyond the raw numbers:
1. RISC-V is now a viable target for edge LLM deployment. Before this work, running quantized LLMs on RISC-V meant falling back to scalar C++ reference implementations in ONNX Runtime or llama.cpp — usable for prototyping, too slow for products. The IME-accelerated path changes the calculus.
2. The matrix extension standardization is gaining real-world validation. IME is one of four competing proposals. Having a production SoC (K1) with working silicon, a community port, and public benchmarks gives the RISC-V Matrix SIG concrete data to evaluate against VME, AME, and BDP.
3. It validates the K1's positioning as an edge-AI gateway SoC. The K1 was designed for routers, industrial gateways, and edge boxes — not AI accelerators. That it can run a 0.5B LLM at interactive speeds (12+ tok/s) using only its CPU vector units means developers can add lightweight AI features (local intent parsing, small-model agents, offline summarization) without adding a separate NPU or switching to a different chip.
Bottom line: The SpacemiT K1 + ONNX Runtime + IME combination proves that RISC-V edge SoCs can run modern quantized LLMs at practical speeds. For developers building AI-enabled gateways, industrial controllers, or smart appliances, this removes a key barrier to choosing RISC-V over ARM.
Last updated: 2026-09-04. Benchmarks are community-reported and may vary with software versions. For the latest K1 AI SDK and ONNX Runtime patches, visit the SpacemiT community forum.