Watch
[v13] Add initial Xen Suspend-to-RAM support on ARM64 #3

Open
lore-7a2973d47aceb2a6ded7bdb9 wants to merge 14 commits from lore/67cfbad5a57e64f7cbbd9b92e05c1e0347f4e69763ad49f7300ffdb07cdbf400/v13 into lore/67cfbad5a57e64f7cbbd9b92e05c1e0347f4e69763ad49f7300ffdb07cdbf400/v13-base
Mykola Kvach commented
2026-10-27 22:06:43 +01:00
First-time contributor
Revision: v13
This is part 2 of the ARM Xen system suspend/resume patch series, based
on earlier work by Mirela Simonovic and Mykyta Poturai.

Part 1, covering guest suspend functionality, is already in mainline.

NOTE: Host-wide suspend/resume support is guarded by CONFIG_SYSTEM_SUSPEND,
which can currently only be selected when UNSUPPORTED is set, and thus the
host suspend backend is neither enabled by default nor built in supported
configurations. The separate HAS_HWDOM_SYSTEM_SUSPEND policy bit only changes
how ARM treats SHUTDOWN_suspend from the hardware domain; it does not enable
the host-wide suspend backend by itself.

This version is ported to Xen master and includes extensive improvements
based on reviewer feedback. The patch series restructures code to improve
robustness and maintainability, and implements initial ARM64 host-wide
Suspend-to-RAM support driven by control-domain PSCI SYSTEM_SUSPEND
requests. vPSCI also exposes SYSTEM_SUSPEND as a domain suspend operation
for all domains; attempt-time host-suspend policy failures are reported as
PSCI_DENIED rather than hidden through PSCI_FEATURES. A non-control domain
needs a control domain to resume it; otherwise its request is denied.

Key updates in this series:
 - Introduced architecture-specific suspend/resume infrastructure
 - Integrated GICv2/GICv3 suspend and resume, including memory-backed context
   save/restore with error handling
 - Added time and IRQ suspend/resume hooks, ensuring correct timer/interrupt
   state across suspend cycles
 - Implemented proper PSCI SYSTEM_SUSPEND invocation and version checks
 - Added vPSCI SYSTEM_SUSPEND policy for domain suspend and host-wide
   control-domain sequencing
 - Improved state management and recovery in error cases during suspend/resume
 - Added support for IPMMU-VMSA/SMMUv3 context save/restore
 - Added support for GICv3 eSPI registers context save/restore
 - Added support for ITS registers context save/restore
---

Link to CI: https://gitlab.com/xen-project/people/mykola_kvach/xen/-/pipelines/2996178037

Tested on H3ULCB (R-Car Gen3) and Orange Pi 5 (Rockchip RK3588S).
---

TODOs:
 - Enable "xl suspend" support on ARM
 - Add suspend/resume CI test for ARM (QEMU if feasible)
 - Add PCI suspend/resume and save/restore R-Car Gen4 PCIe OSID state
---

Detailed changelogs can be found in each patch.

Changes in v13:
- Rebase onto current master and update the arm_smccc_smc() call.
  The API update has no functional effect.
- Add a patch to defer freeing per-CPU areas until resume failure cleanup
  is complete. Remove the redundant RCU barrier before restarting CPUs.
- Keep the GICv2 CPU interface and Distributor control state unchanged on
  suspend entry. Document the requirements for preserving pending events.
- Check GICv3 active priorities before disabling the interfaces. Panic if
  the redistributor fails to enter sleep, as suspend cannot safely abort.
- Update SMMUv3 queue and interrupt handling based on the Linux
  suspend/resume fixes. Preserve unread events and check reset errors.
  Panic if rollback fails or if the SMMU cannot be disabled after a probe
  reset failure. The probe and reset fixes also apply with
  CONFIG_SYSTEM_SUSPEND disabled.
- Block host suspend when Xen uses R-Car Gen4 PCIe, since its OSID state
  is not restored.
- Deny SYSTEM_SUSPEND for non-control domains if no control domain exists
  to resume them.
- Fix MISRA issues in GICv2/GICv3/ITS, FF-A, SMMUv3 and the host suspend
  path. Type and name changes, explicit return-value discards and comment
  updates have no functional effect. The SMMUv3 error checks above change
  failure handling.
- Document the LPI pending-table lifetime requirement for CPU hotplug and
  clarify that IPMMU suspend/resume was tested only on H3ULCB (R-Car Gen3).

Changes in v12:
- Rebase onto the latest master.
- Updated patch 12; all other patches are unchanged.
  No functional changes.

Changes in v11:
- Keep SMMUv3 reset helpers in init text when CONFIG_SYSTEM_SUSPEND is
  disabled.
- Update host suspend policy blockers after review: make the runtime gate
  __ro_after_init, log the SMMUv3 MSI blocker only once, and wrap the Arm
  IOMMU blocker in CONFIG_SYSTEM_SUSPEND.

Changes in v10:
- Clarify the vPSCI SYSTEM_SUSPEND policy summary: keep SYSTEM_SUSPEND
  advertised once implemented and return PSCI_DENIED, rather than
  PSCI_NOT_SUPPORTED, for attempt-time host-suspend policy failures.
- Tighten GICv2/GICv3 suspend/resume based on review feedback: avoid
  reserved interrupt register ranges, check visible active-priority state,
  restore configuration before enable state, and re-enable the redistributor
  before restoring CPU/virtual interface state on abort paths.
- Refine ITS resume so MAPC is replayed only for ITS-backed collections and
  clarify the collection-ID assumptions.
- Rework IPMMU and SMMUv3 resume/suspend handling, including root-before-cache
  IPMMU restore ordering and disabling SMMU interrupt generation before
  suspend.
- Save and restore CNTHCTL_EL2 in the arm64 CPU resume context and simplify
  the resume trampoline/context hand-off.
- Re-apply boot CPU errata/workaround handling after SYSTEM_SUSPEND and move
  set_init_ttbr() declaration to asm/mmu/mm.h.
- Update patch 12 details: shorten SYSTEM_SUSPEND blocker logs, use %pd for
  control-domain logging, mark serial_suspend_available as __ro_after_init,
  and mention the xen/suspend.h struct domain forward declaration.

Changes in v9:
- Split the control-domain SYSTEM_SUSPEND flow so host availability,
  runtime blockers and domain-readiness checks are handled separately from
  the host suspend backend.
- Gate vPSCI SYSTEM_SUSPEND on cached host PSCI support and Xen runtime
  suspend blockers, and log firmware support during initialization.
- Fold the arm64 resume trampoline into the CPU context save/restore patch
  and use asm-offsets-generated RESUME_CTX_* definitions for the assembly
  save/restore path.
- Tighten the GICv2/GICv3/ITS/IPMMU/SMMUv3 suspend/resume paths based on
  review feedback, including state-save/restore fixes and safer failure
  handling.
- Reorder the host suspend/resume phases so timer and GIC state are
  handled with local IRQs disabled and restored before console/IOMMU
  resume.

Changes in v8:
- Rebased to latest master and refreshed the series accordingly.
- Added a new GICv3 patch to tolerate retained redistributor LPI state
  across CPU_OFF/CPU_ON.
- GICv2 suspend now disables the CPU interface and distributor before
  saving state.
- GICv3 suspend/resume fixes the redistributor base used for LPI state.
- ITS and SMMUv3 suspend/resume paths were tightened, with safer
  restore/rollback handling and stricter fatal-error handling.
- System suspend now checks that all domains are already in
  SHUTDOWN_suspend before proceeding, and renames the hardware-domain
  suspend capability/helper for clearer semantics.
- Fixed alignment/cleanup issues in the low-level suspend/resume code.

Changes in v7:
- Timer helper renamed/clarified; virtual/hyper/phys handling documented.
- GICv2 uses one context block; restore saved CTLR; panic on alloc failure.
- GICv3/eSPI/ITS always suspend/resume; restore LPI/eSPI; rdist timeout.
- IPMMU suspend context allocated before PCI setup.
- System suspend: control domain drives host suspend.
- Dropped v6 IRQ descriptor restore patches; use setup_irq and re-register
  local IRQs on resume instead.

For earlier changelogs, please refer to the previous cover letters.

Mirela Simonovic (5):
  xen/arm: Add suspend and resume timer helpers
  xen/arm: gic-v2: Implement GIC suspend/resume functions
  xen/arm64: Save/restore CPU context across SYSTEM_SUSPEND
  xen/arm: Implement PSCI SYSTEM_SUSPEND call (host interface)
  xen/arm: Add host system suspend backend

Mykola Kvach (8):
  xen/arm: gic-v3: tolerate retained redistributor LPI state across
    CPU_OFF
  xen/percpu: defer freeing per-CPU areas after resume failures
  xen/arm: gic-v3: Implement GICv3 suspend/resume functions
  xen/arm: gic-v3: add ITS suspend/resume support
  xen/arm: tee: keep init_tee_secondary() for hotplug and resume
  xen/arm: ffa: fix notification SRI across CPU hotplug/suspend
  xen/arm: smmu-v3: add suspend/resume handlers
  xen/arm: Add vPSCI SYSTEM_SUSPEND policy

Oleksandr Tyshchenko (1):
  iommu/ipmmu-vmsa: Implement suspend/resume callbacks

 xen/arch/arm/Kconfig                     |   2 +
 xen/arch/arm/Makefile                    |   1 +
 xen/arch/arm/arm64/asm-offsets.c         |  21 +
 xen/arch/arm/arm64/head.S                | 122 ++++++
 xen/arch/arm/cpuerrata.c                 |   7 +-
 xen/arch/arm/gic-v2.c                    | 232 ++++++++++-
 xen/arch/arm/gic-v3-its.c                | 146 ++++++-
 xen/arch/arm/gic-v3-lpi.c                |  87 +++-
 xen/arch/arm/gic-v3.c                    | 484 ++++++++++++++++++++++-
 xen/arch/arm/gic.c                       |  35 ++
 xen/arch/arm/include/asm/arm64/sysregs.h |   5 +
 xen/arch/arm/include/asm/cpuerrata.h     |   1 +
 xen/arch/arm/include/asm/gic.h           |  16 +
 xen/arch/arm/include/asm/gic_v3_defs.h   |   3 +
 xen/arch/arm/include/asm/gic_v3_its.h    |  28 ++
 xen/arch/arm/include/asm/mmu/mm.h        |   2 +
 xen/arch/arm/include/asm/psci.h          |   4 +
 xen/arch/arm/include/asm/suspend.h       |  37 ++
 xen/arch/arm/include/asm/time.h          |   5 +
 xen/arch/arm/mmu/smpboot.c               |   2 +-
 xen/arch/arm/pci/pci-host-rcar4.c        |   4 +
 xen/arch/arm/psci.c                      |  39 +-
 xen/arch/arm/suspend.c                   | 203 ++++++++++
 xen/arch/arm/tee/ffa_notif.c             |  63 ++-
 xen/arch/arm/tee/tee.c                   |   2 +-
 xen/arch/arm/time.c                      |  44 ++-
 xen/arch/arm/vpsci.c                     | 125 +++++-
 xen/common/Kconfig                       |   3 +
 xen/common/domain.c                      |   7 +-
 xen/common/percpu.c                      |   9 +
 xen/drivers/char/serial.c                |  12 +
 xen/drivers/passthrough/arm/iommu.c      |   6 +
 xen/drivers/passthrough/arm/ipmmu-vmsa.c | 323 ++++++++++++++-
 xen/drivers/passthrough/arm/smmu-v3.c    | 317 ++++++++++++---
 xen/drivers/passthrough/arm/smmu-v3.h    |   1 +
 xen/include/xen/list.h                   |  14 +
 xen/include/xen/serial.h                 |   1 +
 xen/include/xen/suspend.h                |   2 +
 38 files changed, 2295 insertions(+), 120 deletions(-)
 create mode 100644 xen/arch/arm/suspend.c

-- 
2.53.0
Imported by review-companion Internal server error - Forgejo: Beyond coding. We Forge.
500

Internal server error
Forgejo version: 18.0.0-dev-238-4242acdd5a+gitea-1.22.0