RISC-V's per-CPU Fast Path Goes Fully Preemptible: Recoverable GPR Critical Sections in Patch v10
Every network packet bumped into a queue, every page freed, every scheduler tick increments a
counter that does not need a global lock because it is per CPU. On x86, Arm, PowerPC and s390
that increment is a single atomic instruction against a per-CPU base that the architecture keeps
somewhere cheap. On RISC-V, according to the cover letter of a patch series first posted
on 8 November 2026, it has been something else entirely: the architecture has no
arch/riscv/include/asm/percpu.h, so this_cpu_*() read-modify-write operations use
the generic fallbacks that disable local interrupts, and the address of the variable is
recomputed each time from the CPU number and __per_cpu_offset[].
This article was updated on 13 November 2026 for version 10, posted on 12 November on linux-mm. v10 is not a touch-up: it reworks the entire scheme to be fully preemptible using recoverable general-purpose-register critical sections borrowed from arm64, grows the series from three patches to nine, and publishes a per-operation benchmark table for the first time. The v9 design and its history are kept below, because v10 is easiest to read as an answer to the one weakness v9 still had — it still switched preemption off.
The Gap: An Architecture Without a per-CPU Header
Per-CPU variables exist so that hot counters never contend. The correctness contract is narrower than most people assume: the update must be atomic with respect to everything else on the same CPU, including interrupt handlers, and it must not be migrated to another CPU in the middle. The generic implementation buys that guarantee the crude way — it disables interrupts across the read-modify-write:
/* shape of the generic fallback RISC-V has been using (illustrative, not verbatim) */
this_cpu_inc(v):
local_irq_save(flags) /* interrupts off for the whole RMW */
*per_cpu_ptr(v, raw_smp_processor_id()) += 1
local_irq_restore(flags)
/* shape of the v9 RISC-V path (illustrative, not verbatim) */
this_cpu_inc(v):
preempt_disable() /* interrupts stay enabled */
amoadd.w zero, 1, (this_cpu_base + &v)
preempt_enable()
Two costs follow. First, an interrupt-disabled window around every counter update, on a core where the
update itself only needed preemption disabled; the longer those windows, the more the interrupt latency
tail moves, and an interrupt handler that bumps its own per-CPU counter cannot preempt the thread that is
already inside a generic this_cpu_* block. Second, address computation: without a cached
base, each access has to derive the current offset from the CPU number and the
__per_cpu_offset[] table.
vmstat, network stack statistics, memory
accounting and a long list of hot paths that run millions of times per second on a server. A per-CPU
operation that is merely correct shows up as a constant tax on every workload, which is exactly
the kind of tax that never appears in a benchmark table but does appear in a profile.What v10 Actually Implements
Version 10 is nine patches, twelve files, +805/−16 lines, on the same
linux-next base (ddfe51b0a0f5). The header is now 701 lines, and a new
arch/riscv/kernel/entry-common.c carries the exception-path fixups that make the preemptible
scheme work. The series structure, verbatim from the cover letter:
| Patch | What it does |
|---|---|
| 1/9 | Stores the per-CPU offset into thread_info — the cached
base from v9, now the first patch |
| 2/9 | Adds the infrastructure for preemptible this_cpu operations:
the recoverable-GPR entry/fixup plumbing |
| 3/9 – 7/9 | Implements the preemptible operation classes one by one — read/write, void RMW, add_return, xchg, cmpxchg — so each is independently reviewable |
| 8/9 | cmpxchg128 fallback using the already resolved per-CPU
pointer |
| 9/9 | Selects HAVE_CMPXCHG_LOCAL, opening the
this_cpu_try_cmpxchg() fast path for vmstat and per-CPU counters |
The Rework: Preemptible Without preempt_disable
v9 kept preemption disabled around every operation — cheap to reason about, but it still blocks
task migration for the duration of the update. Reviewers David Laight and Jisheng
Zhang pushed the design further, and v10 adopts their suggestion: recoverable GPR
critical sections, the scheme arm64 already uses. Each operation records the registers used
to form its per-CPU address; if the task is interrupted and resumes on a different CPU, the exception
return path repairs those registers so the operation completes against the correct
per-CPU base. The result is that this_cpu_read() and this_cpu_write() no longer
touch preemption at all, and the RMW classes stay atomic-correct across migration by fixup rather than by
blocking migration.
The cover letter is explicit about what is shared with arm64 and what is not: only the protocol
is common. RISC-V keeps its own cached offset in thread_info, its own GPR encoding and
exception entry code, and its own AMO/LR/SC operation sequences; factoring those
into generic code would share little beyond the field layout while coupling architecture entry paths, so
the implementation stays under arch/riscv.
One more entry-path requirement follows from the design: independent NMI-like entry paths must participate in the entry-state protocol directly. The save, clear and restore operations therefore live in common RISC-V assembly macros so SBI Supervisor Software Events (SSE) can preserve an interrupted per-CPU operation — SSE support itself is developed separately and is explicitly not a dependency of this series.
Why 8- and 16-Bit Needed Zabha
Native-width updates map cleanly onto the A extension. Subword updates do not: the A extension only
defines AMOs for words and doublewords, so a byte or halfword increment historically had to be emulated
— either with a wider bitwise AMO (valid only for the bitwise operations) or with a word-sized
LR/SC loop. The ratified Zabha extension exists precisely to
delete that emulation, and its specification states the reasons better than any summary:
“In systems with large-scale or Non-Uniform Memory Access (NUMA) configurations, emulation based on LR/SC introduces issues related to scalability and fairness, particularly under conditions of high contention. Emulation of narrower AMOs through wider AMO* instructions on non-idempotent IO memory regions may result in unintended side effects. Utilizing wider AMO* instructions for emulating narrower AMOs risks activating extraneous breakpoints or watchpoints. In the absence of native support for subword atomics, compilers often resort to inlining code sequences to provide the required emulation.”
Zabha 1.0 was ratified at the RISC-V Board of Directors meeting on
25 April 2024, in the same batch as the B extension and the Zaamo/Zalrsc split, and it
depends on Zaamo. It adds
AMO[ADD|AND|OR|XOR|SWAP|MIN[U]|MAX[U]].[B|H], requires naturally aligned addresses like the
rest of the A extension, sign-extends the value into rd, and deliberately omits byte and
halfword LR/SC “due to low utility”.
v10 tightens two subword details flagged in review. Masked LR/SC
cmpxchg operands are kept XLEN-wide so that on RV64 the high byte and high
halfword of the containing word are not affected by LR.W sign extension during the compare.
And in the AMOCAS.Q path, both halves of the new value are forced into fixed
registers so the implicit register pair remains valid even when either half is zero.
| Update width | Mechanism (v9, carried into v10) | Instruction count |
|---|---|---|
| Native width (XLEN) | Single relaxed AMO (amoadd.w/d, amoand,
amoor, amoswap, …) | 1 |
| 8-bit / 16-bit, Zabha present | amoadd.b /
amoadd.h and friends, emitted under .option arch,+zabha | 1 |
| 8-bit / 16-bit, no Zabha | 32-bit LR/SC
sequence on the containing word | A loop |
this_cpu_read / write (v10) | Plain load/store through the cached per-CPU base — no preemption control, fixup on migration | 1 |
LR/SC row. That is still better than disabling
interrupts, but it means “RISC-V now does atomic byte counters” and “your board now does
atomic byte counters” are two different claims.The Offset Cache and the Context Switch
The offset cache lives in thread_info, which means it is per-task state that has to
survive three transitions:
- CPU bringup and hotplug — a CPU that comes online later must have its offset filled in before any per-CPU operation runs on it;
- Context switch — the new task must inherit the correct offset for the CPU it is
about to run on, which is a copy from
prevtonextin the switch path (credited to Thomas Gleixner); - Secondary idle task initialisation, which moved into the common startup path so it
covers parallel bringup and CPU hotplug, removing a duplicate initialisation that had been sitting in
smp_callin()(flagged by Sashiko).
v10 adds one more placement decision: the cached offset field is put first in
thread_info, and the exception/fixup infrastructure and the operation classes are split into
independently reviewable patches — both review-driven changes aimed at making the series bisectable
and each piece auditable on its own.
HAVE_CMPXCHG_LOCAL: Unlocking the vmstat Fast Path
The final patch is a single line of Kconfig and arguably the one with the widest blast radius.
Selecting HAVE_CMPXCHG_LOCAL declares that the architecture can perform a local
cmpxchg with preemption disabled rather than interrupts disabled. The generic per-CPU layer
then stops having to take the conservative route, and vmstat and per-CPU counters can
use the this_cpu_try_cmpxchg() fast path. That is the layer that pages, slab and
network statistics ride on, which is why a Kconfig symbol is the interesting part of the series rather
than an afterthought.
The Numbers, With Their Measurement Conditions Stated
Two evidence sets now exist, and they measure different things. For v9, the cover letter attributed a
precise figure: on a SiFive P550, Davidlohr Bueso measured about 15–20% lower
this_cpu_inc() cost in an ad-hoc microbenchmark.
For v10, the cover letter publishes its own table for the first time. Conditions, verbatim:
CONFIG_PREEMPT_LAZY=y, GCC 15, the same linux-next base and configuration, an in-kernel
microbenchmark pinned to one CPU, each operation run 2,000,000 times for 15
rounds, reporting the median empty-loop-adjusted time:
| Operation | v9 (ns/op) | v10 (ns/op) | Change |
|---|---|---|---|
this_cpu_read(u64) | 5.668 | 0.333 | −94.1% |
this_cpu_write(u64) | 5.668 | 0.395 | −93.0% |
this_cpu_add(u64) | 8.002 | 8.004 | +0.0% |
this_cpu_add_return(u64) | 8.335 | 8.004 | −4.0% |
this_cpu_xchg(u64) | 8.278 | 8.004 | −3.3% |
this_cpu_cmpxchg(u64) | 18.713 | 16.133 | −13.8% |
Under the same configuration, vmlinux .text shrank from 14,369,618 to
14,212,882 bytes (−156,736 bytes, −1.09%), and
percpu_counter_add_batch() went from 330 to 228 bytes.
preempt_disable() wrapper, while the RMW rows barely move because roughly 8 ns is the
cost of the AMO instruction itself. Both tables are single microbenchmarks on pinned cores; the series
still publishes no workload-level results and no interrupt- or preemption-latency measurements, and v10
is unmerged — a tenth revision signals review engagement, not acceptance. Treat the
numbers as evidence the direction is real, not as a performance promise.From v8 to v10: What Review Changed
| Revision | Change | Why it matters |
|---|---|---|
| v8 → v9 | PIO helper patch dropped | Already merged upstream |
Value-returning AMOs; .option arch,+zabha; typed always-inline helpers around
arch_cmpxchg_relaxed() | Matches the existing RISC-V cmpxchg implementation;
avoids macro variable capture | |
Copy the cached offset prev → next in the context switch |
Thomas Gleixner’s suggestion; the offset is per-task state | |
Secondary idle task init moved into the common startup path; HAVE_CMPXCHG_LOCAL split
into its own patch | Covers parallel bringup and hotplug (Sashiko); keeps the series bisectable | |
| v9 → v10 | Rework the operations to remain preemptible using recoverable GPR critical sections | David Laight and Jisheng Zhang; removes the last
preempt_disable() dependency |
Cached offset first in thread_info; fixup infrastructure and operation classes split
into independent patches; GPR save/clear/restore in reusable asm macros (SBI SSE participates; SSE support
itself is not a dependency) | Auditability and a defined contract for NMI-like entry paths | |
Masked LR/SC operands kept XLEN-wide; AMOCAS.Q new-value
halves in fixed registers; TYPEOF_UNQUAL() wrappers; parenthesized
this_cpu_write() values | Correctness of the subword compare on RV64; register-pair validity; macro hygiene |
What to Watch
- Merge status. Until this lands, everything above describes a patch, not a kernel.
The observable milestone is the series reaching
for-nextand then a merge window. - Zabha in shipping silicon. This is the variable that decides whether the subword path
is one instruction or a loop on the board in front of you. If you are selecting a RISC-V SoC for a
counter-heavy workload, the ISA string is now worth reading for
zabhaspecifically. - Whether the latency argument shows up in data. v10’s design makes the preemption- and interrupt-latency story testable — there is no longer a disabled window to argue about. A measured worst-case latency comparison would be a stronger story than any throughput figure, and nobody has published one yet.
- The same pattern elsewhere. RISC-V has been accumulating “implementation exists, fast path does not” gaps in exactly this layer; earlier examples covered on this site include the Zalrsc-without-Zaamo atomics work and the move of efficient unaligned access from a compile-time requirement to a runtime check in Linux 7.4.
The least interesting thing about this series is the percentage. The interesting thing is what the fast path turned out to be: a 701-line header, a per-task cached offset, a recoverable-GPR entry contract shared in spirit with arm64, and one Kconfig symbol — none of which could have been written much earlier, because they needed a ratified subword-atomics extension and an agreed answer to “where does the offset live, and what happens if the task moves mid-operation”. That is what architecture support matures into: not new instructions, but the removal of reasons to take the slow road.
Sources: [PATCH v10 0/9] “riscv: optimize percpu access”, Yunhui Cui
(ByteDance), linux-mm, 12 November 2026, Message-ID
20261112104149.38589-1-cuiyunhui@bytedance.com; [PATCH v9 0/3], same author, linux-arch and
linux-kernel, 8 November 2026, Message-ID
20261108110406.97495-1-cuiyunhui@bytedance.com; RISC-V Unified Database, Zabha extension
(v1.0.0, ratified 2024-04-25); The RISC-V Instruction Set Manual, Volume I: Unprivileged
Architecture (20240411), chapter 16.1 “Zabha”. Cover-letter and specification text quoted
verbatim.
中文摘要
2026 年 11 月 8 日,字节跳动 Yunhui Cui 在 linux-arch / linux-kernel 发出
[PATCH v9 0/3] riscv: optimize percpu access。核心问题:RISC-V 一直没有自己的
arch/riscv/include/asm/percpu.h,于是 this_cpu_*() 的读改写操作走通用后备实现——整段关闭本地中断,
而且每次访问都要用 CPU 号去查 __per_cpu_offset[] 重新算偏移。这就是「支持了但没优化」的典型形态:
vmstat、网络统计、内存记账等每秒数百万次的热路径,一直在交一笔看不见的税。
v9 的三个补丁(v10 已重构为 9 个补丁,见下方更新):① 新建 574 行的 asm/percpu.h,用松弛(无 aq/rl)AMO +
preempt_disable() 实现 this_cpu 操作,替代关中断;② 把当前 CPU 的 per-CPU 偏移缓存进
thread_info,并保证在 CPU 上线、热插拔、上下文切换中始终有效(v9 按 Thomas Gleixner 建议在
switch 时从 prev 拷到 next,并把次要 idle 任务初始化移到公共启动路径以覆盖并行 bringup 与热插拔——该重复初始化由
Sashiko 指出);③ 一行 Kconfig 选中 HAVE_CMPXCHG_LOCAL,让 vmstat 与 per-CPU 计数器走上
this_cpu_try_cmpxchg() 快路径。总共 8 文件 +598/−16。
为什么 8/16 位需要 Zabha:A 扩展只定义字/双字 AMO,字节/半字长期靠 LR/SC 循环或更宽的位运算 AMO
模拟。Zabha 1.0 于 2024-04-25 批准(依赖 Zaamo),补上
AMO[ADD|AND|OR|XOR|SWAP|MIN|MAX].[B|H],规范原文给出的理由是:大规模/NUMA 下 LR/SC 模拟在高竞争时
有可扩展性与公平性问题;用更宽 AMO 模拟窄 AMO 在非幂等 IO 区域可能产生副作用;可能误触发断点/观察点;编译器内联模拟序列
会增大代码体积。
2026-11-12 更新(v10):字节跳动 Yunhui Cui 在 linux-mm 发出 [PATCH v10 0/9]——按 David Laight 与 Jisheng Zhang 的意见,整个方案改成完全可抢占:采用 arm64 式「可恢复 GPR 临界区」,每条 this_cpu 操作记录构成 per-CPU 地址的寄存器,任务若在操作中途被中断并在另一个 CPU 上恢复,异常返回路径会修复这些寄存器;读/写操作彻底不再关抢占。总计 9 个补丁、12 文件 +805/−16、percpu.h 701 行。自测基准(CONFIG_PREEMPT_LAZY=y、GCC 15、同一 linux-next 基线、单核绑定、每操作 200 万次 × 15 轮取中位数):this_cpu_read(u64) 5.668→0.333 ns(−94.1%)、write −93.0%、cmpxchg −13.8%、add 持平 0.0%;vmlinux .text 缩小 156,736 B(−1.09%)。诚实话:读/写的大降衡量的是被删掉的 preempt_disable 包装,不是新指令;RMW 各行卡在 AMO 本身约 8 ns 的成本上;两个表都是单核微基准,没有工作负载级与中断/抢占延迟数据,系列仍未合并;P550 的 15–20% 是 v9 阶段 Davidlohr Bueso 的单一微基准口径。
Кратко (RU)
8 ноября 2026 года Yunhui Cui (ByteDance) отправил в linux-arch и linux-kernel серию
[PATCH v9 0/3] riscv: optimize percpu access. Суть проблемы: у RISC-V нет собственного
arch/riscv/include/asm/percpu.h, поэтому операции this_cpu_*() используют
общие резервные реализации, запрещающие локальные прерывания на всё время
чтения-изменения-записи, а адрес каждый раз вычисляется из номера CPU и таблицы
__per_cpu_offset[]. Это классическое «поддержано, но не оптимизировано»: счётчики
vmstat, сетевой статистики и учёта памяти — горячий путь, выполняемый миллионы раз в
секунду.
Три патча v9 (v10 переработана в девять — см. обновление ниже): ① новый заголовок на 574 строки, где this_cpu_* строится на
расслабленных AMO (без aq/rl) плюс preempt_disable()
вместо отключения прерываний; ② кэширование смещения per-CPU в thread_info с сохранением
корректности при старте CPU, hotplug и переключении контекста (копирование prev→next
по предложению Thomas Gleixner; инициализация idle-задачи вторичных CPU перенесена в общий путь запуска —
дублирование в smp_callin() нашёл Sashiko); ③ одна строка Kconfig выбирает
HAVE_CMPXCHG_LOCAL, открывая быстрый путь this_cpu_try_cmpxchg() для
vmstat. Итого 8 файлов, +598/−16.
Зачем нужна Zabha для 8/16 бит: расширение A определяет AMO только для слов и
двойных слов. Zabha 1.0 ратифицирована 25 апреля 2024 года (зависит от Zaamo) и добавляет
AMO[ADD|AND|OR|XOR|SWAP|MIN|MAX].[B|H]. В спецификации прямо названы причины: проблемы
масштабируемости и справедливости LR/SC при высокой конкуренции в NUMA-системах; побочные эффекты при
эмуляции узких AMO широкими на неидемпотентной IO-памяти; риск ложных срабатываний точек останова и
watchpoint; рост размера кода из-за встроенных последовательностей эмуляции.
Обновление 12 ноября 2026 года (v10): Yunhui Cui отправил в linux-mm [PATCH v10 0/9] — по замечаниям David Laight и Jisheng Zhang вся схема переработана в полностью вытесняемую на основе «восстанавливаемых критических секций GPR» по образцу arm64: каждая операция запоминает регистры, формирующие per-CPU-адрес, и при возврате из исключения они исправляются, если задача продолжает работу на другом CPU; чтение и запись больше не отключают вытеснение вовсе. Итог: 9 патчей, 12 файлов, +805/−16, заголовок — 701 строка. Собственный замер (CONFIG_PREEMPT_LAZY=y, GCC 15, та же база linux-next, привязка к одному CPU, 2 млн итераций × 15 раундов, медиана): this_cpu_read(u64) 5.668 → 0.333 нс (−94.1%), write −93.0%, cmpxchg −13.8%, add — без изменений; .text vmlinux сократился на 156 736 байт (−1.09%). Честная оговорка: обвал read/write измеряет убраную обёртку, а не новую инструкцию; строки RMW упираются в ~8 нс самого AMO; оба замера — микробенчмарки на одном ядре без данных на нагрузках; серия по-прежнему не принята, а 15–20% на P550 — одиночный микробенчмарк этапа v9 от Davidlohr Bueso.
Resumen (ES)
El 8 de noviembre de 2026, Yunhui Cui (ByteDance) envió a linux-arch y linux-kernel la
serie [PATCH v9 0/3] riscv: optimize percpu access. El problema: RISC-V no tiene
arch/riscv/include/asm/percpu.h, así que las operaciones this_cpu_*() usan las
implementaciones genéricas que deshabilitan las interrupciones locales durante todo el
read-modify-write, y la dirección se recalcula cada vez a partir del número de CPU y
__per_cpu_offset[]. Es el caso típico de «soportado pero no optimizado»: los contadores de
vmstat, de la pila de red y de contabilidad de memoria se actualizan millones de veces por
segundo.
Tres parches de v9 (v10 se recompuso en nueve; véase la actualización al final): ① un nuevo encabezado de 574 líneas que implementa this_cpu_*
con AMO relajados (sin aq/rl) y preempt_disable()
en lugar de desactivar interrupciones; ② caché del desplazamiento per-CPU en thread_info,
mantenido válido en el arranque de CPU, hotplug y cambio de contexto (copia de prev a
next sugerida por Thomas Gleixner; la inicialización de la tarea idle secundaria se movió al
camino común, con el duplicado en smp_callin() detectado por Sashiko); ③ una línea de Kconfig
que selecciona HAVE_CMPXCHG_LOCAL y habilita la ruta rápida
this_cpu_try_cmpxchg() para vmstat. Total: 8 archivos,
+598/−16.
Por qué hace falta Zabha para 8/16 bits: la extensión A solo define AMO de palabra y
doble palabra. Zabha 1.0 se ratificó el 25 de abril de 2024 (depende de Zaamo) y añade
AMO[ADD|AND|OR|XOR|SWAP|MIN|MAX].[B|H]. La propia especificación enumera los motivos:
problemas de escalabilidad y equidad del emulado con LR/SC bajo alta contención en sistemas NUMA; efectos
secundarios al emular AMO estrechos con AMO más anchos en memoria IO no idempotente; riesgo de disparar
breakpoints y watchpoints espurios; y crecimiento del código por secuencias de emulación insertadas por el
compilador.
Actualización del 12 de noviembre de 2026 (v10): Yunhui Cui envió a linux-mm [PATCH v10 0/9] — siguiendo los comentarios de David Laight y Jisheng Zhang, todo el esquema se recompuso para ser completamente apropiable mediante «secciones críticas de GPR recuperables» al estilo arm64: cada operación registra los registros que forman su dirección per-CPU y el retorno de excepción los repara si la tarea continúa en otra CPU; las operaciones de lectura y escritura ya no desactivan la preeminencia en absoluto. Total: 9 parches, 12 archivos, +805/−16, el encabezado llega a 701 líneas. Medición propia (CONFIG_PREEMPT_LAZY=y, GCC 15, misma base linux-next, fijado a una CPU, 2 millones de iteraciones × 15 rondas, mediana): this_cpu_read(u64) 5.668 → 0.333 ns (−94.1%), write −93.0%, cmpxchg −13.8%, add sin cambios; el .text de vmlinux se reduce en 156 736 bytes (−1.09%). Advertencia honesta: el desplome de read/write mide la sobrecarga eliminada, no una instrucción nueva; las filas RMW chocan con los ~8 ns del propio AMO; ambas tablas son microbenchmarks en un solo núcleo sin datos a nivel de carga; la serie sigue sin fusionar, y el 15–20% del P550 es un único microbenchmark de la etapa v9 medido por Davidlohr Bueso.
Résumé (FR)
Le 8 novembre 2026, Yunhui Cui (ByteDance) a envoyé sur linux-arch et linux-kernel la
série [PATCH v9 0/3] riscv: optimize percpu access. Le problème : RISC-V n’a pas de
arch/riscv/include/asm/percpu.h, donc les opérations this_cpu_*() passent par les
implémentations génériques qui désactivent les interruptions locales pendant tout le
read-modify-write, et l’adresse est recalculée à chaque accès à partir du numéro de CPU et de
__per_cpu_offset[]. C’est le cas d’école du « supporté mais non optimisé » : les
compteurs de vmstat, de la pile réseau et de la comptabilité mémoire sont mis à jour des
millions de fois par seconde.
Trois correctifs de v9 (v10 a été recomposé en neuf — voir la mise à jour plus bas) : ① un nouvel en-tête de 574 lignes qui construit this_cpu_*
sur des AMO relâchés (sans aq/rl) plus
preempt_disable() au lieu de couper les interruptions ; ② la mise en cache de l’offset
per-CPU dans thread_info, maintenue valide au démarrage des CPU, au hotplug et au changement de
contexte (copie de prev vers next suggérée par Thomas Gleixner ; l’init de la
tâche idle secondaire déplacée dans le chemin commun, le doublon dans smp_callin() ayant été
signalé par Sashiko) ; ③ une ligne de Kconfig qui sélectionne HAVE_CMPXCHG_LOCAL et ouvre le
chemin rapide this_cpu_try_cmpxchg() pour vmstat. Total : 8 fichiers,
+598/−16.
Pourquoi Zabha pour le 8/16 bits : l’extension A ne définit des AMO que pour le mot
et le double mot. Zabha 1.0 a été ratifiée le 25 avril 2024 (elle dépend de Zaamo) et ajoute
AMO[ADD|AND|OR|XOR|SWAP|MIN|MAX].[B|H]. La spécification donne elle-même les raisons :
problèmes de passage à l’échelle et d’équité de l’émulation LR/SC sous forte contention en
NUMA ; effets de bord possibles lors de l’émulation d’AMO étroits par des AMO plus larges sur une
mémoire IO non idempotente ; risque de déclencher des breakpoints et watchpoints parasites ; et gonflement du
code par les séquences d’émulation insérées par le compilateur.
Mise à jour du 12 novembre 2026 (v10) : Yunhui Cui a envoyé sur linux-mm [PATCH v10 0/9] — suivant les retours de David Laight et Jisheng Zhang, tout le schéma a été recomposé pour être entièrement préemptible grâce à des « sections critiques de GPR récupérables » à la arm64 : chaque opération enregistre les registres qui forment son adresse per-CPU et le retour d’exception les répare si la tâche reprend sur un autre CPU ; les opérations de lecture et d’écriture ne désactivent plus du tout la préemption. Total : 9 correctifs, 12 fichiers, +805/−16, l’en-tête atteint 701 lignes. Mesure propre (CONFIG_PREEMPT_LAZY=y, GCC 15, même base linux-next, épinglé sur un CPU, 2 millions d’itérations × 15 rounds, médiane) : this_cpu_read(u64) 5.668 → 0.333 ns (−94,1 %), write −93,0 %, cmpxchg −13,8 %, add inchangé ; le .text de vmlinux réduit de 156 736 octets (−1,09 %). Mise au point honnête : l’effondrement de read/write mesure la surcharge supprimée, pas une nouvelle instruction ; les lignes RMW butent sur les ~8 ns de l’AMO lui-même ; les deux tableaux sont des microbenchmarks mono-cœur sans résultats à niveau de charge ; la série reste non fusionnée, et les 15–20 % du P550 correspondent à un unique microbenchmark de l’étape v9 mesuré par Davidlohr Bueso.
Kurzfassung (DE)
Am 8. November 2026 schickte Yunhui Cui (ByteDance) die Serie
[PATCH v9 0/3] riscv: optimize percpu access an linux-arch und linux-kernel. Das Problem:
RISC-V hat kein arch/riscv/include/asm/percpu.h, deshalb nutzen
this_cpu_*()-Operationen die generischen Fallbacks, die für die gesamte
Read-Modify-Write-Sequenz lokale Interrupts abschalten, und die Adresse wird bei jedem
Zugriff erneut aus CPU-Nummer und __per_cpu_offset[] berechnet. Das ist der Musterfall
„unterstützt, aber nicht optimiert“: Zähler von vmstat, Netzwerk-Stack und
Speicher-Abrechnung laufen millionenfach pro Sekunde durch diesen Pfad.
Drei Patches von v9 (v10 wurde zu neun umgebaut — siehe das Update unten): ① ein neuer 574-Zeilen-Header, der this_cpu_* auf
relaxed AMOs (ohne aq/rl) plus preempt_disable()
statt Interrupt-Sperre aufbaut; ② Caching des per-CPU-Offsets in thread_info, gültig über
CPU-Start, Hotplug und Kontextwechsel (Kopie von prev nach next auf Vorschlag von
Thomas Gleixner; die Initialisierung der sekundären Idle-Tasks wurde in den gemeinsamen Startpfad
verschoben, die Dublette in smp_callin() meldete Sashiko); ③ eine Kconfig-Zeile, die
HAVE_CMPXCHG_LOCAL selektiert und damit den this_cpu_try_cmpxchg()-Schnellpfad
für vmstat öffnet. Insgesamt 8 Dateien, +598/−16.
Warum Zabha für 8/16 Bit nötig war: Die A-Erweiterung definiert AMOs nur für Wort und
Doppelwort. Zabha 1.0 wurde am 25. April 2024 ratifiziert (abhängig von Zaamo) und ergänzt
AMO[ADD|AND|OR|XOR|SWAP|MIN|MAX].[B|H]. Die Spezifikation nennt die Gründe selbst:
Skalierbarkeits- und Fairnessprobleme der LR/SC-Emulation unter hoher Konkurrenz in NUMA-Systemen;
mögliche Nebenwirkungen, wenn schmale AMOs durch breitere auf nicht-idempotentem IO-Speicher emuliert
werden; das Risiko, überzählige Breakpoints und Watchpoints auszulösen; und wachsende Codegröße durch
eingebettete Emulationssequenzen.
Update vom 12. November 2026 (v10): Yunhui Cui schickte an linux-mm [PATCH v10 0/9] — auf Hinweis von David Laight und Jisheng Zhang wurde das gesamte Schema zu einer vollständig präemptiblen Lösung umgebaut, mit „recoverten GPR-Kritischen Abschnitten“ nach arm64-Vorbild: jede Operation merkt sich die Register, die ihre per-CPU-Adresse bilden, und die Exception-Rückkehr repariert sie, falls die Aufgabe auf einer anderen CPU fortsetzt; Lese- und Schreiboperationen schalten Preemption gar nicht mehr ab. Insgesamt 9 Patches, 12 Dateien, +805/−16, der Header umfasst 701 Zeilen. Eigener Benchmark (CONFIG_PREEMPT_LAZY=y, GCC 15, gleiche linux-next-Basis, an einen CPU gebunden, 2 Mio. Iterationen × 15 Runden, Median): this_cpu_read(u64) 5.668 → 0.333 ns (−94,1 %), write −93,0 %, cmpxchg −13,8 %, add unverändert; der vmlinux-.text schrumpft um 156 736 Byte (−1,09 %). Ehrliche Einschränkung: der Einbruch bei read/write misst die entfernte Hülle, keine neue Instruktion; die RMW-Zeilen stoßen an die ~8 ns des AMO selbst; beide Tabellen sind Mikrobenchmarks auf einem gebundenen Kern ohne Workload-Daten; die Serie ist weiterhin nicht gemergt, und die 15–20 % vom P550 stammen aus dem einzelnen Mikrobenchmark der v9-Phase von Davidlohr Bueso.
خلاصه (FA)
در ۸ نوامبر ۲۰۲۶، یونهویی تسویی (ByteDance) مجموعهٔ
[PATCH v9 0/3] riscv: optimize percpu access را به linux-arch و linux-kernel فرستاد.
مسئله: RISC-V فایل arch/riscv/include/asm/percpu.h را ندارد، پس عملیات
this_cpu_*() از پیادهسازیهای عمومی استفاده میکند که در تمام مدت خواندن-تغییر-نوشتن
وقفههای محلی را غیرفعال میکنند و آدرس هم هر بار از شمارهٔ CPU و
__per_cpu_offset[] دوباره محاسبه میشود. این همان حالت «پشتیبانیشده اما بهینهنشده» است:
شمارندههای vmstat، پشتهٔ شبکه و حسابداری حافظه میلیونها بار در ثانیه از این مسیر میگذرند.
سه وصلهٔ v9 (v10 به نه وصله بازسازی شده — بهروزرسانی پایین را ببینید): ① هدر تازهٔ ۵۷۴ خطی که this_cpu_* را بر پایهٔ
AMO های relaxed (بدون aq/rl) بههمراه
preempt_disable() میسازد، نه غیرفعالسازی وقفه؛ ② ذخیرهٔ آفست per-CPU در
thread_info و معتبر نگهداشتن آن در راهاندازی CPU، hotplug و تعویض زمینه (کپی از
prev به next به پیشنهاد Thomas Gleixner؛ انتقال مقداردهی idle-task ثانویه به
مسیر راهاندازی مشترک، که تکرارش در smp_callin() را Sashiko گزارش کرده بود)؛ ③ یک خط
Kconfig که HAVE_CMPXCHG_LOCAL را انتخاب میکند و مسیر سریع
this_cpu_try_cmpxchg() را برای vmstat باز میکند. در کل ۸ فایل،
+598/−16.
چرا Zabha برای ۸/۱۶ بیت لازم بود: افزونهٔ A تنها AMO برای کلمه و کلمهٔ دوتایی تعریف
میکند. Zabha 1.0 در ۲۵ آوریل ۲۰۲۴ تصویب شد (وابسته به Zaamo) و
AMO[ADD|AND|OR|XOR|SWAP|MIN|MAX].[B|H] را میافزاید. خودِ مشخصات دلایل را چنین میآورد:
مشکلات مقیاسپذیری و انصاف شبیهسازی LR/SC در رقابت شدید در سامانههای NUMA؛ اثرهای جانبی احتمالی هنگام
شبیهسازی AMO باریک با AMO پهنتر روی حافظهٔ IO غیرخودتوان؛ خطر فعالسازی ناخواستهٔ breakpoint و
watchpoint؛ و افزایش حجم کد بهدلیل توالیهای شبیهسازی که کامپایلر درونخطی میکند.
بهروزرسانی ۱۲ نوامبر ۲۰۲۶ (v10): یونهویی تسویی مجموعهٔ [PATCH v10 0/9] را به linux-mm فرستاد — بنا بر نظر David Laight و Jisheng Zhang، کل طرح به نسخهای کاملاً پیشگیریپذیر بازسازی شد با «بخشهای بحرانی GPR بازیافتپذیر» به سبک arm64: هر عملیات ثبتهایی را که آدرس per-CPU را میسازند یادداشت میکند و بازگشت از استثنا آنها را تعمیر میکند اگر تسک روی CPU دیگری ادامه یابد؛ عملیاتهای خواندن و نوشتن دیگر اصلاً پیشگیری را غیرفعال نمیکنند. در کل ۹ وصله، ۱۲ فایل، +805/−16، هدر ۷۰۱ خط. اندازهگیری خودی (CONFIG_PREEMPT_LAZY=y، GCC 15، پایهٔ یکسان linux-next، سنجاق به یک CPU، ۲ میلیون تکرار × ۱۵ دور، میانه): this_cpu_read(u64) از ۵٫۶۶۸ به ۰٫۳۳۳ نانوثانیه (−۹۴٫۱٪)، نوشتن −۹۳٫۰٪، cmpxchg −۱۳٫۸٪، جمع بدون تغییر؛ متن vmlinux ۱۵۶٬۷۳۶ بایت کوچکتر (−۱٫۰۹٪). هشدار صادقانه: سقوط read/write اندازهگیریِ لایهٔ حذفشده است نه دستور تازه؛ ردیفهای RMW به سقف ~۸ نانوثانیهای خود AMO میخورند؛ هر دو جدول میکروبنچمارکِ تکهستهایاند بدون دادهٔ بار کاری؛ مجموعه همچنان ادغامنشده است و ۱۵–۲۰٪ P550 مربوط به میکروبنچمارک تکی مرحلهٔ v9 از دیویدلور بوسو است.