RISC-V · Kernel · Per-CPU

RISC-V's per-CPU Fast Path Goes Fully Preemptible: Recoverable GPR Critical Sections in Patch v10

📅 2026-11-09 · Updated 2026-11-13 ⏱ 11 min read 🔗 Source: [PATCH v10 0/9] “riscv: optimize percpu access”, linux-mm, 12 November 2026

Every network packet bumped into a queue, every page freed, every scheduler tick increments a counter that does not need a global lock because it is per CPU. On x86, Arm, PowerPC and s390 that increment is a single atomic instruction against a per-CPU base that the architecture keeps somewhere cheap. On RISC-V, according to the cover letter of a patch series first posted on 8 November 2026, it has been something else entirely: the architecture has no arch/riscv/include/asm/percpu.h, so this_cpu_*() read-modify-write operations use the generic fallbacks that disable local interrupts, and the address of the variable is recomputed each time from the CPU number and __per_cpu_offset[].

This article was updated on 13 November 2026 for version 10, posted on 12 November on linux-mm. v10 is not a touch-up: it reworks the entire scheme to be fully preemptible using recoverable general-purpose-register critical sections borrowed from arm64, grows the series from three patches to nine, and publishes a per-operation benchmark table for the first time. The v9 design and its history are kept below, because v10 is easiest to read as an answer to the one weakness v9 still had — it still switched preemption off.

The Gap: An Architecture Without a per-CPU Header

Per-CPU variables exist so that hot counters never contend. The correctness contract is narrower than most people assume: the update must be atomic with respect to everything else on the same CPU, including interrupt handlers, and it must not be migrated to another CPU in the middle. The generic implementation buys that guarantee the crude way — it disables interrupts across the read-modify-write:

/* shape of the generic fallback RISC-V has been using (illustrative, not verbatim) */
this_cpu_inc(v):
    local_irq_save(flags)            /* interrupts off for the whole RMW   */
    *per_cpu_ptr(v, raw_smp_processor_id()) += 1
    local_irq_restore(flags)

/* shape of the v9 RISC-V path (illustrative, not verbatim) */
this_cpu_inc(v):
    preempt_disable()                /* interrupts stay enabled            */
    amoadd.w  zero, 1, (this_cpu_base + &v)
    preempt_enable()

Two costs follow. First, an interrupt-disabled window around every counter update, on a core where the update itself only needed preemption disabled; the longer those windows, the more the interrupt latency tail moves, and an interrupt handler that bumps its own per-CPU counter cannot preempt the thread that is already inside a generic this_cpu_* block. Second, address computation: without a cached base, each access has to derive the current offset from the CPU number and the __per_cpu_offset[] table.

Why this matters more than it looks: per-CPU counters are not an edge case. They are the implementation of vmstat, network stack statistics, memory accounting and a long list of hot paths that run millions of times per second on a server. A per-CPU operation that is merely correct shows up as a constant tax on every workload, which is exactly the kind of tax that never appears in a benchmark table but does appear in a profile.

What v10 Actually Implements

Version 10 is nine patches, twelve files, +805/−16 lines, on the same linux-next base (ddfe51b0a0f5). The header is now 701 lines, and a new arch/riscv/kernel/entry-common.c carries the exception-path fixups that make the preemptible scheme work. The series structure, verbatim from the cover letter:

PatchWhat it does
1/9Stores the per-CPU offset into thread_info — the cached base from v9, now the first patch
2/9Adds the infrastructure for preemptible this_cpu operations: the recoverable-GPR entry/fixup plumbing
3/9 – 7/9Implements the preemptible operation classes one by one — read/write, void RMW, add_return, xchg, cmpxchg — so each is independently reviewable
8/9cmpxchg128 fallback using the already resolved per-CPU pointer
9/9Selects HAVE_CMPXCHG_LOCAL, opening the this_cpu_try_cmpxchg() fast path for vmstat and per-CPU counters

The Rework: Preemptible Without preempt_disable

v9 kept preemption disabled around every operation — cheap to reason about, but it still blocks task migration for the duration of the update. Reviewers David Laight and Jisheng Zhang pushed the design further, and v10 adopts their suggestion: recoverable GPR critical sections, the scheme arm64 already uses. Each operation records the registers used to form its per-CPU address; if the task is interrupted and resumes on a different CPU, the exception return path repairs those registers so the operation completes against the correct per-CPU base. The result is that this_cpu_read() and this_cpu_write() no longer touch preemption at all, and the RMW classes stay atomic-correct across migration by fixup rather than by blocking migration.

The cover letter is explicit about what is shared with arm64 and what is not: only the protocol is common. RISC-V keeps its own cached offset in thread_info, its own GPR encoding and exception entry code, and its own AMO/LR/SC operation sequences; factoring those into generic code would share little beyond the field layout while coupling architecture entry paths, so the implementation stays under arch/riscv.

One more entry-path requirement follows from the design: independent NMI-like entry paths must participate in the entry-state protocol directly. The save, clear and restore operations therefore live in common RISC-V assembly macros so SBI Supervisor Software Events (SSE) can preserve an interrupted per-CPU operation — SSE support itself is developed separately and is explicitly not a dependency of this series.

What “recoverable” actually buys: correctness no longer depends on making migration impossible during the update; it depends on noticing, at exception return, that the operation was interrupted and fixing the address registers before the task continues. That is a bounded, testable property of the entry path — and it removes the last structural reason this architecture had to take the slow road.

Why 8- and 16-Bit Needed Zabha

Native-width updates map cleanly onto the A extension. Subword updates do not: the A extension only defines AMOs for words and doublewords, so a byte or halfword increment historically had to be emulated — either with a wider bitwise AMO (valid only for the bitwise operations) or with a word-sized LR/SC loop. The ratified Zabha extension exists precisely to delete that emulation, and its specification states the reasons better than any summary:

“In systems with large-scale or Non-Uniform Memory Access (NUMA) configurations, emulation based on LR/SC introduces issues related to scalability and fairness, particularly under conditions of high contention. Emulation of narrower AMOs through wider AMO* instructions on non-idempotent IO memory regions may result in unintended side effects. Utilizing wider AMO* instructions for emulating narrower AMOs risks activating extraneous breakpoints or watchpoints. In the absence of native support for subword atomics, compilers often resort to inlining code sequences to provide the required emulation.”

Zabha 1.0 was ratified at the RISC-V Board of Directors meeting on 25 April 2024, in the same batch as the B extension and the Zaamo/Zalrsc split, and it depends on Zaamo. It adds AMO[ADD|AND|OR|XOR|SWAP|MIN[U]|MAX[U]].[B|H], requires naturally aligned addresses like the rest of the A extension, sign-extends the value into rd, and deliberately omits byte and halfword LR/SC “due to low utility”.

v10 tightens two subword details flagged in review. Masked LR/SC cmpxchg operands are kept XLEN-wide so that on RV64 the high byte and high halfword of the containing word are not affected by LR.W sign extension during the compare. And in the AMOCAS.Q path, both halves of the new value are forced into fixed registers so the implicit register pair remains valid even when either half is zero.

Update widthMechanism (v9, carried into v10)Instruction count
Native width (XLEN)Single relaxed AMO (amoadd.w/d, amoand, amoor, amoswap, …)1
8-bit / 16-bit, Zabha presentamoadd.b / amoadd.h and friends, emitted under .option arch,+zabha1
8-bit / 16-bit, no Zabha32-bit LR/SC sequence on the containing wordA loop
this_cpu_read / write (v10)Plain load/store through the cached per-CPU base — no preemption control, fixup on migration1
Read the dependency, not the headline: the one-instruction subword path only exists on silicon that implements Zabha. Zabha was ratified in April 2024, so a large part of the installed RISC-V fleet — including cores designed before that, and cores that opted out — falls into the LR/SC row. That is still better than disabling interrupts, but it means “RISC-V now does atomic byte counters” and “your board now does atomic byte counters” are two different claims.

The Offset Cache and the Context Switch

The offset cache lives in thread_info, which means it is per-task state that has to survive three transitions:

v10 adds one more placement decision: the cached offset field is put first in thread_info, and the exception/fixup infrastructure and the operation classes are split into independently reviewable patches — both review-driven changes aimed at making the series bisectable and each piece auditable on its own.

HAVE_CMPXCHG_LOCAL: Unlocking the vmstat Fast Path

The final patch is a single line of Kconfig and arguably the one with the widest blast radius. Selecting HAVE_CMPXCHG_LOCAL declares that the architecture can perform a local cmpxchg with preemption disabled rather than interrupts disabled. The generic per-CPU layer then stops having to take the conservative route, and vmstat and per-CPU counters can use the this_cpu_try_cmpxchg() fast path. That is the layer that pages, slab and network statistics ride on, which is why a Kconfig symbol is the interesting part of the series rather than an afterthought.

The Numbers, With Their Measurement Conditions Stated

Two evidence sets now exist, and they measure different things. For v9, the cover letter attributed a precise figure: on a SiFive P550, Davidlohr Bueso measured about 15–20% lower this_cpu_inc() cost in an ad-hoc microbenchmark.

For v10, the cover letter publishes its own table for the first time. Conditions, verbatim: CONFIG_PREEMPT_LAZY=y, GCC 15, the same linux-next base and configuration, an in-kernel microbenchmark pinned to one CPU, each operation run 2,000,000 times for 15 rounds, reporting the median empty-loop-adjusted time:

Operationv9 (ns/op)v10 (ns/op)Change
this_cpu_read(u64)5.6680.333−94.1%
this_cpu_write(u64)5.6680.395−93.0%
this_cpu_add(u64)8.0028.004+0.0%
this_cpu_add_return(u64)8.3358.004−4.0%
this_cpu_xchg(u64)8.2788.004−3.3%
this_cpu_cmpxchg(u64)18.71316.133−13.8%

Under the same configuration, vmlinux .text shrank from 14,369,618 to 14,212,882 bytes (−156,736 bytes, −1.09%), and percpu_counter_add_batch() went from 330 to 228 bytes.

Honesty check. The −94% headline is a measurement of removed overhead, not a new instruction: read and write got fast because they no longer carry a preempt_disable() wrapper, while the RMW rows barely move because roughly 8 ns is the cost of the AMO instruction itself. Both tables are single microbenchmarks on pinned cores; the series still publishes no workload-level results and no interrupt- or preemption-latency measurements, and v10 is unmerged — a tenth revision signals review engagement, not acceptance. Treat the numbers as evidence the direction is real, not as a performance promise.

From v8 to v10: What Review Changed

RevisionChangeWhy it matters
v8 → v9PIO helper patch droppedAlready merged upstream
Value-returning AMOs; .option arch,+zabha; typed always-inline helpers around arch_cmpxchg_relaxed()Matches the existing RISC-V cmpxchg implementation; avoids macro variable capture
Copy the cached offset prev → next in the context switch Thomas Gleixner’s suggestion; the offset is per-task state
Secondary idle task init moved into the common startup path; HAVE_CMPXCHG_LOCAL split into its own patchCovers parallel bringup and hotplug (Sashiko); keeps the series bisectable
v9 → v10Rework the operations to remain preemptible using recoverable GPR critical sectionsDavid Laight and Jisheng Zhang; removes the last preempt_disable() dependency
Cached offset first in thread_info; fixup infrastructure and operation classes split into independent patches; GPR save/clear/restore in reusable asm macros (SBI SSE participates; SSE support itself is not a dependency)Auditability and a defined contract for NMI-like entry paths
Masked LR/SC operands kept XLEN-wide; AMOCAS.Q new-value halves in fixed registers; TYPEOF_UNQUAL() wrappers; parenthesized this_cpu_write() valuesCorrectness of the subword compare on RV64; register-pair validity; macro hygiene

What to Watch

The least interesting thing about this series is the percentage. The interesting thing is what the fast path turned out to be: a 701-line header, a per-task cached offset, a recoverable-GPR entry contract shared in spirit with arm64, and one Kconfig symbol — none of which could have been written much earlier, because they needed a ratified subword-atomics extension and an agreed answer to “where does the offset live, and what happens if the task moves mid-operation”. That is what architecture support matures into: not new instructions, but the removal of reasons to take the slow road.

Sources: [PATCH v10 0/9] “riscv: optimize percpu access”, Yunhui Cui (ByteDance), linux-mm, 12 November 2026, Message-ID 20261112104149.38589-1-cuiyunhui@bytedance.com; [PATCH v9 0/3], same author, linux-arch and linux-kernel, 8 November 2026, Message-ID 20261108110406.97495-1-cuiyunhui@bytedance.com; RISC-V Unified Database, Zabha extension (v1.0.0, ratified 2024-04-25); The RISC-V Instruction Set Manual, Volume I: Unprivileged Architecture (20240411), chapter 16.1 “Zabha”. Cover-letter and specification text quoted verbatim.

中文摘要

2026 年 11 月 8 日,字节跳动 Yunhui Cui 在 linux-arch / linux-kernel 发出 [PATCH v9 0/3] riscv: optimize percpu access。核心问题:RISC-V 一直没有自己的 arch/riscv/include/asm/percpu.h,于是 this_cpu_*() 的读改写操作走通用后备实现——整段关闭本地中断, 而且每次访问都要用 CPU 号去查 __per_cpu_offset[] 重新算偏移。这就是「支持了但没优化」的典型形态: vmstat、网络统计、内存记账等每秒数百万次的热路径,一直在交一笔看不见的税。

v9 的三个补丁(v10 已重构为 9 个补丁,见下方更新):① 新建 574 行的 asm/percpu.h,用松弛(无 aq/rl)AMO + preempt_disable() 实现 this_cpu 操作,替代关中断;② 把当前 CPU 的 per-CPU 偏移缓存进 thread_info,并保证在 CPU 上线、热插拔、上下文切换中始终有效(v9 按 Thomas Gleixner 建议在 switch 时从 prev 拷到 next,并把次要 idle 任务初始化移到公共启动路径以覆盖并行 bringup 与热插拔——该重复初始化由 Sashiko 指出);③ 一行 Kconfig 选中 HAVE_CMPXCHG_LOCAL,让 vmstat 与 per-CPU 计数器走上 this_cpu_try_cmpxchg() 快路径。总共 8 文件 +598/−16。

为什么 8/16 位需要 Zabha:A 扩展只定义字/双字 AMO,字节/半字长期靠 LR/SC 循环或更宽的位运算 AMO 模拟。Zabha 1.0 于 2024-04-25 批准(依赖 Zaamo),补上 AMO[ADD|AND|OR|XOR|SWAP|MIN|MAX].[B|H],规范原文给出的理由是:大规模/NUMA 下 LR/SC 模拟在高竞争时 有可扩展性与公平性问题;用更宽 AMO 模拟窄 AMO 在非幂等 IO 区域可能产生副作用;可能误触发断点/观察点;编译器内联模拟序列 会增大代码体积。

2026-11-12 更新(v10):字节跳动 Yunhui Cui 在 linux-mm 发出 [PATCH v10 0/9]——按 David Laight 与 Jisheng Zhang 的意见,整个方案改成完全可抢占:采用 arm64 式「可恢复 GPR 临界区」,每条 this_cpu 操作记录构成 per-CPU 地址的寄存器,任务若在操作中途被中断并在另一个 CPU 上恢复,异常返回路径会修复这些寄存器;读/写操作彻底不再关抢占。总计 9 个补丁、12 文件 +805/−16、percpu.h 701 行。自测基准(CONFIG_PREEMPT_LAZY=y、GCC 15、同一 linux-next 基线、单核绑定、每操作 200 万次 × 15 轮取中位数):this_cpu_read(u64) 5.668→0.333 ns(−94.1%)、write −93.0%、cmpxchg −13.8%、add 持平 0.0%;vmlinux .text 缩小 156,736 B(−1.09%)。诚实话:读/写的大降衡量的是被删掉的 preempt_disable 包装,不是新指令;RMW 各行卡在 AMO 本身约 8 ns 的成本上;两个表都是单核微基准,没有工作负载级与中断/抢占延迟数据,系列仍未合并;P550 的 15–20% 是 v9 阶段 Davidlohr Bueso 的单一微基准口径。

Кратко (RU)

8 ноября 2026 года Yunhui Cui (ByteDance) отправил в linux-arch и linux-kernel серию [PATCH v9 0/3] riscv: optimize percpu access. Суть проблемы: у RISC-V нет собственного arch/riscv/include/asm/percpu.h, поэтому операции this_cpu_*() используют общие резервные реализации, запрещающие локальные прерывания на всё время чтения-изменения-записи, а адрес каждый раз вычисляется из номера CPU и таблицы __per_cpu_offset[]. Это классическое «поддержано, но не оптимизировано»: счётчики vmstat, сетевой статистики и учёта памяти — горячий путь, выполняемый миллионы раз в секунду.

Три патча v9 (v10 переработана в девять — см. обновление ниже): ① новый заголовок на 574 строки, где this_cpu_* строится на расслабленных AMO (без aq/rl) плюс preempt_disable() вместо отключения прерываний; ② кэширование смещения per-CPU в thread_info с сохранением корректности при старте CPU, hotplug и переключении контекста (копирование prev→next по предложению Thomas Gleixner; инициализация idle-задачи вторичных CPU перенесена в общий путь запуска — дублирование в smp_callin() нашёл Sashiko); ③ одна строка Kconfig выбирает HAVE_CMPXCHG_LOCAL, открывая быстрый путь this_cpu_try_cmpxchg() для vmstat. Итого 8 файлов, +598/−16.

Зачем нужна Zabha для 8/16 бит: расширение A определяет AMO только для слов и двойных слов. Zabha 1.0 ратифицирована 25 апреля 2024 года (зависит от Zaamo) и добавляет AMO[ADD|AND|OR|XOR|SWAP|MIN|MAX].[B|H]. В спецификации прямо названы причины: проблемы масштабируемости и справедливости LR/SC при высокой конкуренции в NUMA-системах; побочные эффекты при эмуляции узких AMO широкими на неидемпотентной IO-памяти; риск ложных срабатываний точек останова и watchpoint; рост размера кода из-за встроенных последовательностей эмуляции.

Обновление 12 ноября 2026 года (v10): Yunhui Cui отправил в linux-mm [PATCH v10 0/9] — по замечаниям David Laight и Jisheng Zhang вся схема переработана в полностью вытесняемую на основе «восстанавливаемых критических секций GPR» по образцу arm64: каждая операция запоминает регистры, формирующие per-CPU-адрес, и при возврате из исключения они исправляются, если задача продолжает работу на другом CPU; чтение и запись больше не отключают вытеснение вовсе. Итог: 9 патчей, 12 файлов, +805/−16, заголовок — 701 строка. Собственный замер (CONFIG_PREEMPT_LAZY=y, GCC 15, та же база linux-next, привязка к одному CPU, 2 млн итераций × 15 раундов, медиана): this_cpu_read(u64) 5.668 → 0.333 нс (−94.1%), write −93.0%, cmpxchg −13.8%, add — без изменений; .text vmlinux сократился на 156 736 байт (−1.09%). Честная оговорка: обвал read/write измеряет убраную обёртку, а не новую инструкцию; строки RMW упираются в ~8 нс самого AMO; оба замера — микробенчмарки на одном ядре без данных на нагрузках; серия по-прежнему не принята, а 15–20% на P550 — одиночный микробенчмарк этапа v9 от Davidlohr Bueso.

Resumen (ES)

El 8 de noviembre de 2026, Yunhui Cui (ByteDance) envió a linux-arch y linux-kernel la serie [PATCH v9 0/3] riscv: optimize percpu access. El problema: RISC-V no tiene arch/riscv/include/asm/percpu.h, así que las operaciones this_cpu_*() usan las implementaciones genéricas que deshabilitan las interrupciones locales durante todo el read-modify-write, y la dirección se recalcula cada vez a partir del número de CPU y __per_cpu_offset[]. Es el caso típico de «soportado pero no optimizado»: los contadores de vmstat, de la pila de red y de contabilidad de memoria se actualizan millones de veces por segundo.

Tres parches de v9 (v10 se recompuso en nueve; véase la actualización al final): ① un nuevo encabezado de 574 líneas que implementa this_cpu_* con AMO relajados (sin aq/rl) y preempt_disable() en lugar de desactivar interrupciones; ② caché del desplazamiento per-CPU en thread_info, mantenido válido en el arranque de CPU, hotplug y cambio de contexto (copia de prev a next sugerida por Thomas Gleixner; la inicialización de la tarea idle secundaria se movió al camino común, con el duplicado en smp_callin() detectado por Sashiko); ③ una línea de Kconfig que selecciona HAVE_CMPXCHG_LOCAL y habilita la ruta rápida this_cpu_try_cmpxchg() para vmstat. Total: 8 archivos, +598/−16.

Por qué hace falta Zabha para 8/16 bits: la extensión A solo define AMO de palabra y doble palabra. Zabha 1.0 se ratificó el 25 de abril de 2024 (depende de Zaamo) y añade AMO[ADD|AND|OR|XOR|SWAP|MIN|MAX].[B|H]. La propia especificación enumera los motivos: problemas de escalabilidad y equidad del emulado con LR/SC bajo alta contención en sistemas NUMA; efectos secundarios al emular AMO estrechos con AMO más anchos en memoria IO no idempotente; riesgo de disparar breakpoints y watchpoints espurios; y crecimiento del código por secuencias de emulación insertadas por el compilador.

Actualización del 12 de noviembre de 2026 (v10): Yunhui Cui envió a linux-mm [PATCH v10 0/9] — siguiendo los comentarios de David Laight y Jisheng Zhang, todo el esquema se recompuso para ser completamente apropiable mediante «secciones críticas de GPR recuperables» al estilo arm64: cada operación registra los registros que forman su dirección per-CPU y el retorno de excepción los repara si la tarea continúa en otra CPU; las operaciones de lectura y escritura ya no desactivan la preeminencia en absoluto. Total: 9 parches, 12 archivos, +805/−16, el encabezado llega a 701 líneas. Medición propia (CONFIG_PREEMPT_LAZY=y, GCC 15, misma base linux-next, fijado a una CPU, 2 millones de iteraciones × 15 rondas, mediana): this_cpu_read(u64) 5.668 → 0.333 ns (−94.1%), write −93.0%, cmpxchg −13.8%, add sin cambios; el .text de vmlinux se reduce en 156 736 bytes (−1.09%). Advertencia honesta: el desplome de read/write mide la sobrecarga eliminada, no una instrucción nueva; las filas RMW chocan con los ~8 ns del propio AMO; ambas tablas son microbenchmarks en un solo núcleo sin datos a nivel de carga; la serie sigue sin fusionar, y el 15–20% del P550 es un único microbenchmark de la etapa v9 medido por Davidlohr Bueso.

Résumé (FR)

Le 8 novembre 2026, Yunhui Cui (ByteDance) a envoyé sur linux-arch et linux-kernel la série [PATCH v9 0/3] riscv: optimize percpu access. Le problème : RISC-V n’a pas de arch/riscv/include/asm/percpu.h, donc les opérations this_cpu_*() passent par les implémentations génériques qui désactivent les interruptions locales pendant tout le read-modify-write, et l’adresse est recalculée à chaque accès à partir du numéro de CPU et de __per_cpu_offset[]. C’est le cas d’école du « supporté mais non optimisé » : les compteurs de vmstat, de la pile réseau et de la comptabilité mémoire sont mis à jour des millions de fois par seconde.

Trois correctifs de v9 (v10 a été recomposé en neuf — voir la mise à jour plus bas) : ① un nouvel en-tête de 574 lignes qui construit this_cpu_* sur des AMO relâchés (sans aq/rl) plus preempt_disable() au lieu de couper les interruptions ; ② la mise en cache de l’offset per-CPU dans thread_info, maintenue valide au démarrage des CPU, au hotplug et au changement de contexte (copie de prev vers next suggérée par Thomas Gleixner ; l’init de la tâche idle secondaire déplacée dans le chemin commun, le doublon dans smp_callin() ayant été signalé par Sashiko) ; ③ une ligne de Kconfig qui sélectionne HAVE_CMPXCHG_LOCAL et ouvre le chemin rapide this_cpu_try_cmpxchg() pour vmstat. Total : 8 fichiers, +598/−16.

Pourquoi Zabha pour le 8/16 bits : l’extension A ne définit des AMO que pour le mot et le double mot. Zabha 1.0 a été ratifiée le 25 avril 2024 (elle dépend de Zaamo) et ajoute AMO[ADD|AND|OR|XOR|SWAP|MIN|MAX].[B|H]. La spécification donne elle-même les raisons : problèmes de passage à l’échelle et d’équité de l’émulation LR/SC sous forte contention en NUMA ; effets de bord possibles lors de l’émulation d’AMO étroits par des AMO plus larges sur une mémoire IO non idempotente ; risque de déclencher des breakpoints et watchpoints parasites ; et gonflement du code par les séquences d’émulation insérées par le compilateur.

Mise à jour du 12 novembre 2026 (v10) : Yunhui Cui a envoyé sur linux-mm [PATCH v10 0/9] — suivant les retours de David Laight et Jisheng Zhang, tout le schéma a été recomposé pour être entièrement préemptible grâce à des « sections critiques de GPR récupérables » à la arm64 : chaque opération enregistre les registres qui forment son adresse per-CPU et le retour d’exception les répare si la tâche reprend sur un autre CPU ; les opérations de lecture et d’écriture ne désactivent plus du tout la préemption. Total : 9 correctifs, 12 fichiers, +805/−16, l’en-tête atteint 701 lignes. Mesure propre (CONFIG_PREEMPT_LAZY=y, GCC 15, même base linux-next, épinglé sur un CPU, 2 millions d’itérations × 15 rounds, médiane) : this_cpu_read(u64) 5.668 → 0.333 ns (−94,1 %), write −93,0 %, cmpxchg −13,8 %, add inchangé ; le .text de vmlinux réduit de 156 736 octets (−1,09 %). Mise au point honnête : l’effondrement de read/write mesure la surcharge supprimée, pas une nouvelle instruction ; les lignes RMW butent sur les ~8 ns de l’AMO lui-même ; les deux tableaux sont des microbenchmarks mono-cœur sans résultats à niveau de charge ; la série reste non fusionnée, et les 15–20 % du P550 correspondent à un unique microbenchmark de l’étape v9 mesuré par Davidlohr Bueso.

Kurzfassung (DE)

Am 8. November 2026 schickte Yunhui Cui (ByteDance) die Serie [PATCH v9 0/3] riscv: optimize percpu access an linux-arch und linux-kernel. Das Problem: RISC-V hat kein arch/riscv/include/asm/percpu.h, deshalb nutzen this_cpu_*()-Operationen die generischen Fallbacks, die für die gesamte Read-Modify-Write-Sequenz lokale Interrupts abschalten, und die Adresse wird bei jedem Zugriff erneut aus CPU-Nummer und __per_cpu_offset[] berechnet. Das ist der Musterfall „unterstützt, aber nicht optimiert“: Zähler von vmstat, Netzwerk-Stack und Speicher-Abrechnung laufen millionenfach pro Sekunde durch diesen Pfad.

Drei Patches von v9 (v10 wurde zu neun umgebaut — siehe das Update unten): ① ein neuer 574-Zeilen-Header, der this_cpu_* auf relaxed AMOs (ohne aq/rl) plus preempt_disable() statt Interrupt-Sperre aufbaut; ② Caching des per-CPU-Offsets in thread_info, gültig über CPU-Start, Hotplug und Kontextwechsel (Kopie von prev nach next auf Vorschlag von Thomas Gleixner; die Initialisierung der sekundären Idle-Tasks wurde in den gemeinsamen Startpfad verschoben, die Dublette in smp_callin() meldete Sashiko); ③ eine Kconfig-Zeile, die HAVE_CMPXCHG_LOCAL selektiert und damit den this_cpu_try_cmpxchg()-Schnellpfad für vmstat öffnet. Insgesamt 8 Dateien, +598/−16.

Warum Zabha für 8/16 Bit nötig war: Die A-Erweiterung definiert AMOs nur für Wort und Doppelwort. Zabha 1.0 wurde am 25. April 2024 ratifiziert (abhängig von Zaamo) und ergänzt AMO[ADD|AND|OR|XOR|SWAP|MIN|MAX].[B|H]. Die Spezifikation nennt die Gründe selbst: Skalierbarkeits- und Fairnessprobleme der LR/SC-Emulation unter hoher Konkurrenz in NUMA-Systemen; mögliche Nebenwirkungen, wenn schmale AMOs durch breitere auf nicht-idempotentem IO-Speicher emuliert werden; das Risiko, überzählige Breakpoints und Watchpoints auszulösen; und wachsende Codegröße durch eingebettete Emulationssequenzen.

Update vom 12. November 2026 (v10): Yunhui Cui schickte an linux-mm [PATCH v10 0/9] — auf Hinweis von David Laight und Jisheng Zhang wurde das gesamte Schema zu einer vollständig präemptiblen Lösung umgebaut, mit „recoverten GPR-Kritischen Abschnitten“ nach arm64-Vorbild: jede Operation merkt sich die Register, die ihre per-CPU-Adresse bilden, und die Exception-Rückkehr repariert sie, falls die Aufgabe auf einer anderen CPU fortsetzt; Lese- und Schreiboperationen schalten Preemption gar nicht mehr ab. Insgesamt 9 Patches, 12 Dateien, +805/−16, der Header umfasst 701 Zeilen. Eigener Benchmark (CONFIG_PREEMPT_LAZY=y, GCC 15, gleiche linux-next-Basis, an einen CPU gebunden, 2 Mio. Iterationen × 15 Runden, Median): this_cpu_read(u64) 5.668 → 0.333 ns (−94,1 %), write −93,0 %, cmpxchg −13,8 %, add unverändert; der vmlinux-.text schrumpft um 156 736 Byte (−1,09 %). Ehrliche Einschränkung: der Einbruch bei read/write misst die entfernte Hülle, keine neue Instruktion; die RMW-Zeilen stoßen an die ~8 ns des AMO selbst; beide Tabellen sind Mikrobenchmarks auf einem gebundenen Kern ohne Workload-Daten; die Serie ist weiterhin nicht gemergt, und die 15–20 % vom P550 stammen aus dem einzelnen Mikrobenchmark der v9-Phase von Davidlohr Bueso.

خلاصه (FA)

در ۸ نوامبر ۲۰۲۶، یون‌هویی تسویی (ByteDance) مجموعهٔ [PATCH v9 0/3] riscv: optimize percpu access را به linux-arch و linux-kernel فرستاد. مسئله: RISC-V فایل arch/riscv/include/asm/percpu.h را ندارد، پس عملیات this_cpu_*() از پیاده‌سازی‌های عمومی استفاده می‌کند که در تمام مدت خواندن-تغییر-نوشتن وقفه‌های محلی را غیرفعال می‌کنند و آدرس هم هر بار از شمارهٔ CPU و __per_cpu_offset[] دوباره محاسبه می‌شود. این همان حالت «پشتیبانی‌شده اما بهینه‌نشده» است: شمارنده‌های vmstat، پشتهٔ شبکه و حسابداری حافظه میلیون‌ها بار در ثانیه از این مسیر می‌گذرند.

سه وصلهٔ v9 (v10 به نه وصله بازسازی شده — به‌روزرسانی پایین را ببینید): ① هدر تازهٔ ۵۷۴ خطی که this_cpu_* را بر پایهٔ AMO های relaxed (بدون aq/rl) به‌همراه preempt_disable() می‌سازد، نه غیرفعال‌سازی وقفه؛ ② ذخیرهٔ آفست per-CPU در thread_info و معتبر نگه‌داشتن آن در راه‌اندازی CPU، hotplug و تعویض زمینه (کپی از prev به next به پیشنهاد Thomas Gleixner؛ انتقال مقداردهی idle-task ثانویه به مسیر راه‌اندازی مشترک، که تکرارش در smp_callin() را Sashiko گزارش کرده بود)؛ ③ یک خط Kconfig که HAVE_CMPXCHG_LOCAL را انتخاب می‌کند و مسیر سریع this_cpu_try_cmpxchg() را برای vmstat باز می‌کند. در کل ۸ فایل، +598/−16.

چرا Zabha برای ۸/۱۶ بیت لازم بود: افزونهٔ A تنها AMO برای کلمه و کلمهٔ دوتایی تعریف می‌کند. Zabha 1.0 در ۲۵ آوریل ۲۰۲۴ تصویب شد (وابسته به Zaamo) و AMO[ADD|AND|OR|XOR|SWAP|MIN|MAX].[B|H] را می‌افزاید. خودِ مشخصات دلایل را چنین می‌آورد: مشکلات مقیاس‌پذیری و انصاف شبیه‌سازی LR/SC در رقابت شدید در سامانه‌های NUMA؛ اثرهای جانبی احتمالی هنگام شبیه‌سازی AMO باریک با AMO پهن‌تر روی حافظهٔ IO غیرخودتوان؛ خطر فعال‌سازی ناخواستهٔ breakpoint و watchpoint؛ و افزایش حجم کد به‌دلیل توالی‌های شبیه‌سازی که کامپایلر درون‌خطی می‌کند.

به‌روزرسانی ۱۲ نوامبر ۲۰۲۶ (v10): یون‌هویی تسویی مجموعهٔ [PATCH v10 0/9] را به linux-mm فرستاد — بنا بر نظر David Laight و Jisheng Zhang، کل طرح به نسخه‌ای کاملاً پیش‌گیری‌پذیر بازسازی شد با «بخش‌های بحرانی GPR بازیافت‌پذیر» به سبک arm64: هر عملیات ثبت‌هایی را که آدرس per-CPU را می‌سازند یادداشت می‌کند و بازگشت از استثنا آن‌ها را تعمیر می‌کند اگر تسک روی CPU دیگری ادامه یابد؛ عملیات‌های خواندن و نوشتن دیگر اصلاً پیش‌گیری را غیرفعال نمی‌کنند. در کل ۹ وصله، ۱۲ فایل، +805/−16، هدر ۷۰۱ خط. اندازه‌گیری خودی (CONFIG_PREEMPT_LAZY=y، GCC 15، پایهٔ یکسان linux-next، سنجاق به یک CPU، ۲ میلیون تکرار × ۱۵ دور، میانه): this_cpu_read(u64) از ۵٫۶۶۸ به ۰٫۳۳۳ نانوثانیه (−۹۴٫۱٪)، نوشتن −۹۳٫۰٪، cmpxchg −۱۳٫۸٪، جمع بدون تغییر؛ متن vmlinux ۱۵۶٬۷۳۶ بایت کوچک‌تر (−۱٫۰۹٪). هشدار صادقانه: سقوط read/write اندازه‌گیریِ لایهٔ حذف‌شده است نه دستور تازه؛ ردیف‌های RMW به سقف ~۸ نانوثانیه‌ای خود AMO می‌خورند؛ هر دو جدول میکروبنچمارکِ تک‌هسته‌ای‌اند بدون دادهٔ بار کاری؛ مجموعه همچنان ادغام‌نشده است و ۱۵–۲۰٪ P550 مربوط به میکروبنچمارک تکی مرحلهٔ v9 از دیویدلور بوسو است.