MNN LLM Inference on SpacemiT K3 — IME2 W4B64 Quantization and Benchmarks

On September 13, 2026, Alibaba's MNN inference engine team published a deep-dive implementation note covering end-to-end LLM optimization on the SpacemiT K3 RISC-V AI CPU. The work focuses on asymmetric W4B64 quantization accelerated by K3's IME2 matrix extension, with reproducible llm_bench numbers for Qwen3 and Qwen3.5 models. For engineers evaluating K3 as a local-LLM host, this is the most detailed public recipe yet for squeezing usable throughput out of the A100 AI cores.

Why K3 + MNN Matters for Local LLMs

SpacemiT K3 is marketed as an RVA23 AI CPU: eight X100 general-purpose cores plus eight A100 AI compute cores on the same ISA. Unlike a discrete NPU, the A100 cores run standard RISC-V Linux threads and share the RVV 1.0 programming model with the X100 cores. That means an inference engine can treat the chip as a uniform RISC-V host and simply dispatch matrix-heavy kernels to the A100 cluster via the IME (Integrated Matrix Extension) instruction set.

MNN's K3 target exploits this unified model. It adds an IME2 backend that compiles into the A100 tensor units while keeping ARM/x86/standard-RVV paths untouched. The result is a layered build where enabling K3 support is a single CMake flag.

SpacemiT K3 Hardware Baseline

The A100 cluster is organized into two clusters of four scalar cores each, with two vector cores and one shared IME2 Tensor Core per pair of vector cores. A single core cannot continuously feed the shared Tensor Core; MNN therefore uses worker pairs to keep the matrix unit busy.

Benchmark Results

MNN ran all tests with 8 threads, prefill at pp512 (512 input tokens) and decode at tg128 (128 generated tokens). Each benchmark repeated 5 times inside the process and was averaged across 3 fresh processes.

The Qwen3.5 models use a mixed LinearAttention/full-Attention architecture, so their numbers are not directly comparable to Qwen3 on a single-kernel basis. The article explicitly notes that Flash Attention was enabled for Qwen3 (-fa 1) but disabled for Qwen3.5 (-fa 0).

Build MNN with K3 IME2 Support

The only K3-specific switch is -DMNN_RVV_SPACEMIT_IME2=ON. With it, MNN builds two additional object libraries:

• MNNSpacemitIme2Runtime — runtime glue compiled with standard -march
• MNNSpacemitIme2 — vendor kernel compiled with -march=..._xsmtvdotii

When IME2 is enabled, the standard RVV fast-path registration is replaced by K3's own registration entry, so the two paths do not collide.

# Typical K3 build command
cmake -S . -B build-k3 \
  -DCMAKE_BUILD_TYPE=Release \
  -DMNN_RVV_SPACEMIT_IME2=ON \
  -DMNN_BUILD_LLM=ON \
  -DMNN_BUILD_CONVERTER=OFF \
  -DMNN_OPENCL=OFF \
  -DMNN_VULKAN=OFF \
  -DMNN_CUDA=OFF
cmake --build build-k3 -j$(nproc)

Reproducing the Benchmarks

MNN uses its own llm_bench tool. The article publishes the exact command lines for prefill and decode.

Qwen3 prefill (Flash Attention on):

./llm_bench -m <MNN_CONFIG> \
  -p 512 -n 0 -rep 5 -t 8 \
  -load false -fa 1 -kv false

Qwen3.5 prefill (Flash Attention off):

./llm_bench -m <MNN_CONFIG> \
  -p 512 -n 0 -rep 5 -t 8 \
  -load false -fa 0 -kv false

Decode for all models:

./llm_bench -m <MNN_CONFIG> \
  -p 1 -n 128 -rep 5 -t 8 \
  -load false -fa 0 -kv true

The IME2 path is selected at build time; no runtime environment variable is required.

Key Optimizations

1. Asymmetric W4B64 IME2 kernel

Weights are quantized to 4 bits per value in 64-element blocks (W4B64) with per-block FP16 scale and offset. Activations are kept at INT8. Because IME2's native instructions operate on same-width integer tiles, the kernel splits each INT8 activation into signed-high and unsigned-low 4-bit halves and issues two IME2 INT4 dot-product instructions (vmadotsu and vmadotu), then merges the partial sums. A block-scale variant (*.hp) reduces scaling overhead.

2. Fused dynamic quantization and A packing

Instead of scanning activations twice (once for absmax/scale, once for packing), the optimized path computes the dynamic scale, packs the activation tile and produces the kernel sum in a single pass.

3. Prefill: strided M4 + direct-C4 epilogue

Prefill uses an M4 kernel that computes four rows of activation at once. Workers are assigned strided row blocks to avoid fine-grained task dispatch. Where shape constraints allow, the kernel writes C4 output directly, skipping an intermediate buffer and layout conversion.

4. Decode: M1 asym-pair + worker-pair TCM pipeline

Decode is memory-bound: each token streams most of the weights through LPDDR5. MNN uses a worker-pair copy/compute double-buffer over the 3 MiB TCM:

Worker A: copy B tile 0 -> compute tile 0 -> copy tile 2 -> ...
Worker B:                copy tile 1 -> compute tile 1 -> ...

This raised Qwen3-1.7B decode from ~22.5 tok/s to ~24.9 tok/s. The implementation only enables TCM pipelining when the packed-B is at least 2 MiB and K is at least 2048; smaller matrices stay on the DRAM path to avoid synchronization overhead.

5. Attention and KV Cache

Quantized Linear layers dominate runtime (about 93.9% for Qwen3-1.7B decode). Attention uses a standard RVV direct-matvec path, plus a gated K3 fused Attention path that keeps QK, online softmax and PV in one tiled loop. KV Cache updates are parallelized for contiguous FP32 key/value data.

Source Code Map

Honest Limitations

• The published benchmarks are for specific models, W4B64 quantization, 8 threads and fixed input lengths. Other shapes may behave differently.
• K3's headline 60 TOPS is for 4:2 structured sparse INT4; the W4B64 dense weights tested here run closer to the dense INT4 ceiling.
• The IME2 instruction set and K3 architecture paper are still marked as preview / v0.6; production code should be revalidated against the final vendor documents.
• Decode throughput is ultimately constrained by LPDDR5 bandwidth, not just peak TOPS. Simply adding more workers can hurt performance once bandwidth is saturated.