K3 Acceleration Bug Hunt: SpaceMIT EP Computes Attention Q.KT Wrong on EmbeddingGemma-2

A detailed bug report appeared on the SpacemiT forum this week (Oct 10) from a researcher running EmbeddingGemma-2 text embeddings locally on a K3 Pico-ITX board. It is one of the best-debugged execution-provider issues published in the community so far, and the debugging method is worth learning even if you never hit this bug.

The Symptom

ONNX Runtime 1.24.0+spacemit.a3 on Bianbu 4.0.1 (kernel 6.18.3, riscv64 RVA23)
With CPUExecutionProvider: 182 ms per sentence, and output bit-exact identical to a desktop fp32 reference. Retrieval works.
With SpaceMITExecutionProvider (the 8 AI cores, cpu8-15): it runs faster (132 ms) but the output is garbage - cosine similarity vs the fp32 reference is only 0.6956. Semantic retrieval is unusable.

The Localization Method

The reporter used the EP built-in debug switches to export every intermediate tensor and compared them one by one against the CPU EP reference:

SPACEMIT_EP_DUMP_SUBGRAPHS=1 (145 subgraphs exported) SPACEMIT_EP_DUMP_TENSORS=<dir> (533 tensors exported)

For each tensor they computed both order-sensitive and order-insensitive error metrics (the latter to rule out layout/transposition artifacts), then walked the graph in order looking for the first genuine divergence:

Graph order	Tensor	Sensitive err	Insensitive err	Verdict
#57	embedding_projection/MatMul	5.6e-7	5.6e-7	OK
#67	attn/qkv_proj/MatMul	6.7e-7	6.7e-7	OK
#112	attn/MatMul_qk	1.00	0.33	WRONG

The culprit: the attention Q.KT MatMul inside layer 0. The adjacent qkv projection computes fine. Errors then amplify downstream - reaching 3.1 by layer 8 and 5-17x in the MLP blocks of layers 16-20.

Ruled Out

The report is unusually thorough in what it eliminates:

Decomposing the com.microsoft::RotaryEmbedding node into primitive ops - error unchanged, so rotary is not the cause
SPACEMIT_EP_DENSE_ACCURACY_LEVEL at all settings (int8/fp16/fp32 tiers) - output identical
Disabling MatMulTransposeFusion / PackedAttentionFusion passes - no effect
Pushing all 24 QKT MatMuls back to CPU - still wrong (cosine 0.6815), so the corruption may not even be in the accelerated node itself

Bonus Findings

The same report documents three additional issues: the apt-packaged libspacemit_ep.so 2.0.7 has a different md5 than the official release and segfaults on session creation (the official release works, matching modelzoo benchmark numbers), the xslim 2.1.1 optimizer has its own quirks, and int8 models currently fail to compile on the EP with a RotaryEmbedding error.

Takeaways for K3/K1 Developers

Always validate against CPU EP before trusting SpaceMIT EP output - cosine vs a reference is a five-line check that would have caught this immediately.
The tensor-dump workflow is the fastest path to a reproducible bug report. Forum reports at this quality level get fixed; vague ones do not.
If your workload is latency-sensitive but correctness-critical (embeddings for RAG, search), run the CPU EP path until SpacemiT ships a fix - 182 ms vs 132 ms is a modest penalty for usable results.

This also explains several earlier community reports of "wrong results" and "TCM buffer" issues across the SpaceMIT EP stack - the provider is powerful but still maturing, and systematic per-op validation is currently essential.

Hardware context: the SpacemiT K3 pairs 8 X100 cores at 2.4 GHz with 8 AI cores delivering 60 TOPS, RVA23-compliant, with local 30B-parameter LLM inference. The K1 (8x X60, 2 TOPS) shares the same software stack for lighter gateway + edge-AI duties.

Source: K3 Pico-ITX EmbeddingGemma-2 SpaceMIT EP QKT bug report (SpacemiT forum #1696)