Repository navigation
Conversation
When Xen is itself a Hyper-V guest and hands its PV dom0 the host's VMBus, dom0 must speak Hyper-V to the L0 host through Xen. Detect this "nested on Xen" case (xen_pv_domain() + "Microsoft Hv" CPUID) and bring up a minimal Hyper-V guest: GUEST_OS_ID, a hypercall page and the per-cpu input pages, skipping the APIC/timer/isolation setup a PV dom0 does not use. Add hv_nested_hostpfn()/hv_nested_hostpa() helpers translating dom0's pseudo-physical frames to the machine (L1-physical) frames the L0 host requires, and have hv_do_hypercall() emit machine addresses in this mode. Reserve VIRQ_HYPERV_VMBUS and add hyperv_setup_xen_vmbus_irq() to bind the VMBus interrupt to it. The helpers compile to no-ops without CONFIG_XEN_PV. Signed-off-by: Ariadne Conill <ariadne@ariadne.space>
…n Xen The VMBus GPADL page lists and the SynIC SIMP/SIEFP pages must hold the machine (L1-physical) frames the L0 Hyper-V host can access, not dom0's pseudo-physical frames. Route them through hv_nested_hostpfn()/ hv_nested_hostpa() when running nested on Xen; the translation is a no-op otherwise. Signed-off-by: Ariadne Conill <ariadne@ariadne.space>
…n Xen When nested on Xen the host's synthetic interrupt is taken by Xen and relayed to dom0 as VIRQ_HYPERV_VMBUS. Bind the VMBus ISR to that VIRQ instead of the Hyper-V synthetic vector in that case. Signed-off-by: Ariadne Conill <ariadne@ariadne.space>
Like an isolation VM, a Xen PV dom0 nested under Hyper-V must hand the host machine (bus) addresses for transmit page buffers. Extend the DMA-map/unmap path (which routes through swiotlb-xen) to also run in the nested-on-Xen case, so RNDIS control and data sends reference machine frames the host can read. Signed-off-by: Ariadne Conill <ariadne@ariadne.space>
A Xen PV dom0 nested under Hyper-V cannot back vPCI MSIs with x86_vector_domain. The host composes them against a vector on a virtual processor belonging to Xen, so Xen owns the vector and tells us which one; and PV never onlines the local APIC, so cpu0 is the only CPU in the vector matrix and acking through apic_ack_edge() reaches xen_apic_eoi(), which warns. Query that vector and supply a minimal parent irq domain for these interrupts, one in which masking and affinity belong to the host: Xen has already acked the vector by the time it relays the interrupt as VIRQ_HYPERV_VPCI, and affinity takes effect through the retarget hypercall the vPCI irq_chip above issues on unmask. As one vector serves every vPCI device, run each registered handler on every relayed interrupt, and mark the interrupts polled so that the spurious detector does not eventually disable a busy device whose handler usually has nothing to do. Multi-message MSI cannot work in this configuration - a contiguous block of vectors cannot come from the single relay vector - so drop it; MSI-X, which NVMe and MANA use, is unaffected. Config space likewise takes the trapped MMIO path in preference to HVCALL_MMIO_READ/WRITE, whose output page Xen would otherwise have to marshal, as arm64 already does. Signed-off-by: Ariadne Conill <ariadne@ariadne.space>
hyperv_setup_xen_vmbus_irq() had no counterpart, so the VIRQ_HYPERV_VMBUS
bindings it installs were never removed. Two paths leaked them:
- vmbus_bus_init() failing after the bridge was set up fell into the
err_alloc unwind, which tested vmbus_irq == -1 and so called
hv_remove_vmbus_handler() -- tearing down a native SINT handler that
was never installed, while the per-cpu VIRQs stayed bound.
- vmbus_exit() did the same on module unload.
Add hyperv_remove_xen_vmbus_irq(), which drops the cpuhp state and thereby
runs xen_hyperv_vmbus_cpu_down() on every cpu, and dispatch to it from both
sites before the vmbus_irq tests. Record the cpuhp state returned by
cpuhp_setup_state() so it can be removed, and clear the stored handler when
setup fails.
Signed-off-by: Ariadne Conill <ariadne@edera.dev>
Port of Teddy Astie's RFC v3 driver for the Xen PV-IOMMU interface. Under
Xen the IOMMU belongs to the hypervisor, so a guest cannot use anything
that needs one -- VFIO, or DMA protection of its own. The PV-IOMMU
hypercall lets it manage IOMMU contexts through Xen instead, and this
driver presents that as a normal iommu_ops.
https://patchwork.kernel.org/project/xen-devel/list/?series=906092
Changes needed against 6.18:
- pgsize_bitmap moved from struct iommu_ops to struct iommu_domain, so
the mask the hypervisor reports is now stashed and applied to each
domain as it is allocated, including the identity domain.
- iommu_alloc_page()/iommu_free_page() are gone; the page table code uses
iommu_alloc_pages_sz()/iommu_free_pages().
CONFIG_XEN_IOMMU still depends on XEN_DOM0, matching the interface as
posted: the hypercall names devices by machine BDF, which a guest does not
see.
Additional changes needed against mainline (7.3):
- attach_dev takes the old-domain parameter added by fd71498
("iommu: Pass in old domain to attach_dev callback functions"). The
driver ignores it; IOMMU_reattach_device handles the transition.
- io_pgtable_init_table no longer carries the AMD_IOMMU entries, so only
the XEN_IOMMU_GENERIC entry is added.
The driver itself was never Dom0-specific -- it only checks xen_domain() and sends the SBDF it sees, which is what the hypervisor now translates. The restriction lived entirely in Kconfig, so drop it to depends on XEN. Guests do need one adjustment. Dom0 sees the real topology, bridges included, and Xen tracks all of it. A guest only gets the endpoints vPCI assigned to it, sitting under an emulated bridge that has no device on the Xen side, so don't claim bridges there; attaching a context to one can only fail. A guest also has to call IOMMU_init before it has any contexts to allocate, which the driver already does unconditionally.
xen_iommu_identity_domain is statically allocated, and the core only fills in type and ops for domains it allocated itself. Left at zero the type is IOMMU_DOMAIN_BLOCKED, not identity, and the ops pointer is NULL, so anything the core routes to this domain either does the wrong thing or dereferences NULL. Name both explicitly. attach_dev is the same as for a paging domain -- the identity domain is just context 0. [ Adapted for mainline: the forward declaration carries the old-domain parameter attach_dev gained in fd71498 ("iommu: Pass in old domain to attach_dev callback functions"). ]
A PCI host bridge's memory windows are reserved in IOVA space so that an
IOVA cannot collide with an address the bridge would route to MMIO rather
than to memory. Under a paravirtual IOMMU that reasoning does not apply:
the IOVA space belongs to the hypervisor, is programmed by hypercall, and
is not the bridge's address space at all.
Worse, the windows a Xen guest sees are invented. pcifront hands out
iomem_resource itself as the root bus window, and a PVH guest has no host
bridge _CRS so Linux falls back to its catch-all default, clipped only by
the guest's physical address width:
pci_bus 0000:00: root bus resource [mem 0x00000000-0x3fffffffffff]
Reserving that swallows the whole aperture the hypervisor reported, and
every IOVA allocation fails -- even for a single page:
nvidia 0000:00:01.0: IOVA alloc failed: len=1 limit=7fffffffff shift=12
granule=4096 start_pfn=1 aperture=0-7fffffffff
NVRM: GPU0 osIovaMap: failed to map allocation (status = 0x59)
The device then never gets a DMA mapping and the driver cannot attach. This
is inert without an IOMMU, since there is no IOVA space to reserve, so it
only appears once a guest drives one.
Let the driver owning the IOVA space opt out. Kept to a private header
rather than a flag in struct iommu_ops: adding a field there changes the
CRC of every exported symbol whose type graph reaches it, which breaks
every already-built out-of-tree module.
The BDF a PV guest sees for a passed-through device is invented by pciback in the hardware domain and never reaches the hypervisor, so it cannot be used to name the device in a hypercall. PV-IOMMU needs to. pciback already publishes both halves of the mapping in its xenstore directory -- dev-N is the machine BDF, vdev-N the one this domain sees -- and pcifront already reads vdev-N when detaching, so the mapping is right there. Walk it and hand the machine BDF back. Use it in the PV-IOMMU driver for both reattach and the reserved-region query. With vPCI the SBDF a guest sees is one Xen assigned and can translate, so this only changes what a PV guest sends. Both end up naming a device the hypervisor can find.
A PV guest reaches its devices through xen-swiotlb, which hands them machine addresses. Attaching one to a translated IOMMU context as well leaves two translations disagreeing about every address, and the DMA layer rejects the combination outright: WARNING: CPU: 0 PID: 11 at kernel/dma/mapping.c:881 dma_supported+0x5e/0x80 dma_set_mask+0x24/0xe0 nv_set_dma_address_size+0x52/0x70 [nvidia] nv_pci_probe+0x28f/0x1090 [nvidia] dma_supported() warns and returns false when a device both uses iommu-dma and has dma_map_ops, which in a PV guest it always does. The driver's dma_set_mask() then fails and every mapping after it is wrong; the GPU here got as far as GspStatusQueueInit before returning NV_ERR_RESET_REQUIRED. Report IOMMU_DOMAIN_IDENTITY as the default domain type there, which is context 0 and what the hypervisor already advertises as identity. Normal DMA stays on xen-swiotlb and translated contexts can still be asked for explicitly, which is what a guest wanting one for VFIO would do. Guests whose devices come through vPCI are unaffected; they have no dma_map_ops of their own and use iommu-dma as before.
A PV guest does not own the MSI-X table of a passed-through device. Xen programs it when mapping the pirq and maps the table read-only, so the mask, message and unmask writes in pci_write_msg_msix() fault. vfio-pci refreshes the cached message unconditionally before enabling a vector, to undo a backdoor reset, which crashes a PV guest: BUG: unable to handle page fault for address: ffffc9004001d000 #PF: supervisor write access in kernel mode RIP: __pci_write_msi_msg+0x74/0x1f0 vfio_msi_set_vector_signal+0x2c9/0x320 Return early as the is_virtual case already does; nothing in a PV guest programs the table itself, the Xen MSI domain routes allocation through PHYSDEVOP_map_pirq and Xen does the write.
A PV domain took the single-page path unconditionally, one hypercall per 4K page, because pfn_to_gfn(x + 1) != pfn_to_gfn(x) + 1 there in general and the batched path assumed the whole request was contiguous. Every subop also carries an IOTLB flush on the hypervisor side, so a large mapping costs a flush per page: a 4G premapping is a million hypercalls and a million flushes, which no caller waits out. Walk the request and issue one subop per maximal run contiguous in both dfn and gfn, as the TODO describing this asked for. Behaviour is unchanged where pfn_to_gfn is the identity, the run then covering the whole request. map_single_pages remains as a module parameter.
An IOMMU does not sit behind itself, so the hypervisor has no device for it and reattach fails with -ENODEV. The IOMMU core does not allow a driver to fail the first domain attach, so claiming it takes down registration of the whole driver: WARNING at drivers/iommu/iommu.c:3037 iommu_setup_default_domain iommu_device_register xen_iommu_init xen-iommu: Unable to register Xen IOMMU device -19 Only reachable in the initial domain, which sees the real topology; a guest is never given the IOMMU. Skip PCI class 0806.
pci_disable_msix() clears msix_enabled in pci_msix_shutdown() before pci_free_msi_irqs() reaches the domain teardown, so by the time xen_pv_teardown_msi_irqs() runs the flag can no longer say which of the two the device was using. It always took the MSI branch. The backend therefore never saw XEN_PCI_OP_disable_msix and left MSI-X enabled on the real device, and a later guest asking to enable it got -EALREADY from xen_pcibk_enable_msix(). Seen as a device usable exactly once per assignment. Take the type from a descriptor instead. The descriptor lock is already held by the caller.
A map subop covers a run of frames contiguous in both dfn and gfn, and in a PV domain gfns are machine frames, so a scattered buffer breaks into runs of a few pages. Each subop then flushed, and against an emulated IOMMU a flush is a command and a completion wait that costs far more than the mapping does. Mapping several gigabytes took long enough for callers to give up. The core already has somewhere to put this: ask the hypervisor to skip the per-subop flush and issue one over the whole range from iotlb_sync_map(). Gated on IOMMUCAP_deferred_flush so an older hypervisor keeps flushing per subop rather than never flushing at all.
"iommu/xen: Allow the PV-IOMMU driver in a guest" relaxed the dependency
from XEN_DOM0 to XEN so a domU could use the driver, but XEN_DOM0 is
x86-only and dropping it also dropped the architecture constraint. The
driver does not build anywhere else:
- HYPERVISOR_iommu_op() is defined only in
arch/x86/include/asm/xen/hypercall.h.
- arm64 lays out guest handles differently, so assigning a pointer to
the xen_reserved_device_memory handle is a type error there.
An arm64 defconfig with XEN=y and XEN_IOMMU=y therefore fails to build.
Depend on X86 as well, which is what the hypercall interface requires
anyway.
Signed-off-by: Ariadne Conill <ariadne@edera.dev>
Xen domU instances do not boot on x86 with ACPI enabled, so the entire ACPI subsystem is ultimately disabled. This causes acpi_mps_check() to trigger a warning that the ACPI MPS table is not present, which then disables APIC support on domU, breaking the CPU topology detection for all vCPUs other than the boot vCPU. Fixes: 7c0edad ("x86/cpu/topology: Rework possible CPU management")
Xen domU cannot access the given MMIO address for security reasons, resulting in a failed hypercall in ioremap() due to permissions. Fixes: ab81310 ("x86/CPU/AMD: Print the reason for the last reset") Signed-off-by: Ariadne Conill <ariadne@ariadne.space> Cc: stable@vger.kernel.org Signed-off-by: Ariadne Conill <ariadne@ariadne.space>
On a Xen dom0 boot, this feature does not behave, and we end up
calculating:
num_roots = 1
num_nodes = 2
roots_per_node = 0
This causes a divide-by-zero in the modulus inside the loop.
This change adds a couple of guards for invalid states where we might
get a divide-by-zero.
Signed-off-by: Steven Noonan <steven@uplinklabs.net>
Signed-off-by: Ariadne Conill <ariadne@ariadne.space>
CC: Yazen Ghannam <yazen.ghannam@amd.com>
CC: x86@vger.kernel.org
CC: stable@vger.kernel.org
[ Reduced for mainline: 72bd92b ("x86/amd_node: Avoid divide by zero on
virtualized systems") already handles roots_per_node == 0 by clamping it to
1 rather than bailing, and deliberately suppresses the FW_BUG warning under
X86_FEATURE_HYPERVISOR -- so the num_roots < num_nodes early return is
dropped here to avoid regressing that. Upstream still does not guard
num_nodes == 0 before the division, so that check is kept. ]
We've been seeing /dev/xen/evtchn fds enter the ring_overflow=1 state under sustained event load (specifically: many ports bound to a single fd, all firing concurrently during a daemon-side metrics-collection storm). Once the flag latches, every read returns -EFBIG and the fd is dead for event delivery until reset. The per-port masking invariant (each bound port can occupy at most one ring slot, because lateeoi_ack_dynirq masks the channel at Xen level before evtchn_interrupt runs and userspace can't unmask until it reads + writes the port back) combined with the ring being sized to nr_evtchns means overflow should not be reachable. No "Interrupt for port N, but apparently not enabled" WARNs precede the EFBIG state on hosts where this happens, so the masking invariant is holding. One plausible candidate: the producer's READ_ONCE(ring_cons) in evtchn_interrupt has no acquire pairing with the consumer's WRITE_ONCE in evtchn_read -- they take ring_prod_lock and ring_cons_mutex respectively, neither of which synchronizes ring_cons between them. The smp_wmb/smp_rmb pair already present covers ring entry visibility against ring_prod, not ring_cons. A producer that fires right after the consumer drains can observe a stale ring_cons and compute (prod - stale_cons) >= ring_size when the ring actually has room. On x86-TSO the staleness window is bounded to store-buffer drain time but is non-zero; on weakly-ordered hardware it's wider. Add an observe-only diagnostic at the overflow point: before setting ring_overflow, issue an smp_mb() and re-read ring_cons. Emit a rate-limited pr_warn with prod, both cons values, both depths, ring_size, nr_evtchns, and the triggering port's enabled/unbinding flags, tagging the log with " SPURIOUS" when the post-barrier depth is below ring_size (i.e., the original overflow detection was based on a stale view). This does not change runtime behavior other than emitting a log line at the overflow boundary (which already required entering a path that wedges the fd, so the printk cost is irrelevant). If the diagnostic confirms the stale-cons hypothesis, the real fix is to switch ring_cons access to smp_load_acquire / smp_store_release pairing. Signed-off-by: Steven Noonan <steven@edera.dev>
Dom0 code that grant-maps foreign pages -- xenbus ring mappings, gntdev -- has no way to learn which host NUMA node backs a foreign frame, so nothing downstream (kthread and IRQ placement, placeholder page homing) can be made node-local to the guest being served. Every backend kthread for every queue piles onto whatever node it happens to land on, regardless of the guest's vNUMA layout. Add xen_mfn_to_node(), resolving one host MFN to a Linux node id via the new XENMEM_get_mfn_pxms hypercall, along with the hypercall's interface definition. It lives in grant-table.c because its callers are the grant-mapping paths that need to home placeholder pages. Three NUMA identifier namespaces are involved. Xen returns host PXM (firmware-supplied), matching what dom0's own SRAT already uses; pxm_to_node() converts to a Linux dom0 node id, which is what callers feed to Linux helpers like cpumask_of_node() and kthread_create_on_node(). Keeping the Xen-side ABI in PXM-space lets callers translate with one standard lookup instead of maintaining their own Xen-nid -> Linux-node table. The query is meaningful only in the hardware domain: Xen restricts XENMEM_get_mfn_pxms to it, and a guest has no view of host topology to localize against in any case. xen_mfn_to_node returns NUMA_NO_NODE immediately when !xen_initial_domain(), without issuing the hypercall. Older or non-Edera hypervisors lack the hypercall. The first call returns -ENOSYS, which latches a global "unsupported" flag; every subsequent call returns NUMA_NO_NODE without entering the hypervisor. -EPERM (XSM policy, a late hardware domain) latches the same way, so no boot pays more than one refused attempt. Callers seeing NUMA_NO_NODE fall back to NUMA-oblivious behaviour, making the kernel-side change safe to ship without a lockstep hypervisor update. Gated by CONFIG_XEN_BACKEND_NUMA_AFFINITY, which depends on XEN_BACKEND, NUMA, ACPI_NUMA, and XEN_UNPOPULATED_ALLOC and defaults to y. XEN_UNPOPULATED_ALLOC is a hard dependency rather than a soft one because the placement work the rest of this series performs is only meaningful when grant-map placeholders come from a per-node pool: with the generic balloon allocator, page_to_nid() reflects only where dom0's free RAM happened to be, and per-ring NUMA placement would commit to wrong nodes confidently rather than silently no-op. Disabling the option compiles the query out; the header supplies a NUMA_NO_NODE stub so callers need no ifdefs. Sampling policy is left to callers. The hypercall accepts a batch of MFNs, though the consumers in this series query only a ring's first frame. Signed-off-by: Steven Noonan <steven@edera.dev>
xen_alloc_unpopulated_pages today hands out ZONE_DEVICE placeholder
pages from a single global free list, with the underlying section
registered via memremap_pages(pgmap, NUMA_NO_NODE). page_to_nid()
of any such page reports NUMA_NO_NODE, so every consumer of
foreign-mapped grant pages -- xenbus rings, gntdev mmaps, xlate_mmu,
privcmd, xen-drm -- has no useful node information for kernel code
that consults page_to_nid (slab placement decisions, autonuma,
numastat, various softirq routing heuristics).
Partition the free list by Linux node id and register each
section-aligned IOMEM allocation against a specific node. Pages
drawn from page_list[N] then report page_to_nid() == N by
construction. Callers that know the host node of the foreign frame
they intend to grant-map can request from the matching pool; the
foreign frame's node and the placeholder page's nid then agree.
Two exported entry points:
xen_alloc_unpopulated_pages_node(nr, pages, node) -- new, takes an
explicit node preference. NUMA_NO_NODE and out-of-range values
are clamped to numa_node_id() at the boundary; internal code only
ever indexes into a valid slot.
xen_alloc_unpopulated_pages(nr, pages) -- existing API, now a
wrapper that passes numa_node_id(). Strictly an improvement over
NUMA_NO_NODE for callers with no node hint of their own: the
caller's local node is at least as good a guess as none.
xen_free_unpopulated_pages routes each page back to the pool of the
node it came from via page_to_nid(). No new arguments; existing
callers do not change.
A single mutex still covers all per-node lists. Allocation and free
are connect-time events on I/O backends and on map_grant_ref ioctls,
not contention-sensitive paths.
MAX_NUMNODES sizes the two arrays; on CONFIG_NUMA=n that is 1 and the
behaviour is byte-for-byte equivalent to the previous single-list
implementation. The consumers that pair this allocator with
xen_mfn_to_node to home placeholders on the foreign frame's node come
in the patches that follow.
Signed-off-by: Steven Noonan <steven@edera.dev>
gnttab_alloc_pages draws grant-mapping placeholder pages via xen_alloc_unpopulated_pages, which prefers the calling CPU's NUMA node. That is the right default for a caller with no better information, but a consumer that already knows the host node of the foreign frames it is about to map -- because it queried the node via xen_mfn_to_node after a first map -- has no way to ask for placeholders from the matching pool. Add gnttab_alloc_pages_node, taking an explicit node preference and passing it through to xen_alloc_unpopulated_pages_node, and reduce gnttab_alloc_pages to a wrapper passing NUMA_NO_NODE (which the unpopulated allocator clamps to the local node, preserving today's behaviour exactly). Frees are unchanged: gnttab_free_pages already routes each page back to its home pool via page_to_nid. No functional change for existing callers. The first user is gntdev, which will relocate a fresh grant mapping's placeholders onto the foreign frames' node the same way the xenbus ring path gains later in this series. Signed-off-by: Steven Noonan <steven@edera.dev>
xen_allocate_irq_dynamic() unconditionally calls irq_alloc_desc_from(0, -1), so every Xen evtchn IRQ descriptor is allocated with NUMA_NO_NODE. This means /proc/irq/N/node always reads -1 even when the caller (netback, blkback, netfront, blkfront) already knows the right node for the IRQ. irqbalance treats node=-1 as "no NUMA preference" and distributes the IRQ across all CPUs for load balance, ignoring affinity_hint. With irqbalance running, the per-queue NUMA placement we install via irq_set_affinity_and_hint() gets overwritten almost immediately. Add _on_node variants of the four bind helpers Xen front/back ends use: bind_evtchn_to_irq_lateeoi_on_node(evtchn, node) bind_evtchn_to_irqhandler_lateeoi_on_node(..., node) bind_interdomain_evtchn_to_irq_lateeoi_on_node(..., node) bind_interdomain_evtchn_to_irqhandler_lateeoi_on_node(..., node) Each passes the caller's node through to the internal chip helpers and on to a new xen_allocate_irq_dynamic_node(node). The existing public functions become thin wrappers passing NUMA_NO_NODE, so every caller that hasn't been updated keeps today's behaviour. After this change, /proc/irq/N/node reflects the node the caller asked for, and irqbalance respects affinity_hint as a NUMA-local subset rather than treating the IRQ as floating. Signed-off-by: Steven Noonan <steven@edera.dev>
Xenbus backends in PVH dom0 today have no visibility into which host NUMA node the foreign frame backing a grant-mapped ring lives on, so nothing can drive kthread or IRQ placement off of it, and every backend kthread for every queue piles onto one host node regardless of the guest's vNUMA layout. Make xenbus_map_ring_hvm place its placeholder pages on the foreign frame's node. Placeholders are first drawn from numa_node_id()'s pool, because the foreign MFN is not known before the grant_map. After grant_map succeeds, the first page's host node is resolved via xen_mfn_to_node and the placeholders are relocated: unmap, free, re-allocate on the target node, re-map. When the initial allocation already landed on the right node, the relocate is skipped. After this, page_to_nid() of every mapped ring page equals the host node of its foreign MFN by construction. Expose the result as xenbus_ring_host_node(dev, vaddr), returning the Linux node id of the node hosting a mapped ring, or NUMA_NO_NODE when the information is unavailable. Thanks to the relocation, it is a thin wrapper around vmalloc_to_page() + page_to_nid(): no hypercall, no lock, no per-map caching. Backends feed the value to cpumask_of_node() and kthread_create_on_node(). The foreign MFN is sourced from dev_bus_addr in the gnttab map result. Xen sets that field unconditionally for host_map operations (see grant_table.c in the hypervisor), so it is reliable in PVH dom0. Sampling only the first ring page's MFN is sufficient for the common case where a ring is contiguous on one host node; multi-node huge-page grants are a future concern. The relocate costs one extra grant unmap + map pair per backend connect (a rare event), paid only when the placeholder pool's node disagrees with the foreign frame's. A failed unmap during the relocate marks the pages leaked and fails the whole map, since at least one grant may still be live against a placeholder that can no longer be reached safely. PV dom0 is not a NUMA-affinity target: PV mappings install foreign MFNs directly in the PTEs and have no struct page in dom0's mem_map for the foreign frame, so xenbus_ring_host_node returns NUMA_NO_NODE there. On hypervisors without XENMEM_get_mfn_pxms the node resolves to NUMA_NO_NODE at map time and the relocate is skipped entirely. Backends seeing NUMA_NO_NODE fall back to today's NUMA-oblivious behaviour. Signed-off-by: Steven Noonan <steven@edera.dev>
Frontend drivers today route every ring allocation through xenbus_setup_ring, which calls alloc_pages_exact with no node preference. The result is that every PV ring -- across all queues of all multi-queue devices on a guest -- ends up on whichever node the xenbus watch handler happens to run on, typically a single fixed value at boot. Multi-queue devices on multi-vnode guests therefore defeat the dom0 backend's per-queue NUMA affinity work: all of dom0's backend kthreads cluster on one host node because all of the guest's rings live on one guest node, which maps to one host node. Add a node-aware variant. xenbus_setup_ring_node(dev, gfp, node, ...) takes a Linux node id and draws the ring pages from that node's buddy list. xenbus_setup_ring() is now a thin wrapper that passes NUMA_NO_NODE, preserving existing behaviour for every caller until they opt in. The same-ring locality property is preserved by construction. A single buddy allocation comes from a single node's free list, so all pages of one ring remain on one node regardless of which node was requested. The new variant only changes which node that is. alloc_pages_exact_nid is __meminit-restricted and not exported, so the body cannot just delegate to it. Use alloc_pages_node() (which is exported and runtime-safe) to get an order-N block on the target node, then split_page() it so every subpage carries an independent refcount, and free any tail pages beyond ring_size back to the allocator. Also add xenbus_node_for_queue(index), a helper that rotates over the set of nodes with online CPUs. Frontend callers feed it a per-queue or per-ring index to pick the node they pass to xenbus_setup_ring_node. The natural shape is cpumask_local_spread(i, NUMA_NO_NODE), but with a NUMA_NO_NODE node argument that falls back to a linear walk of cpu_online_mask (see sched_numa_find_nth_cpu) and collapses every queue onto the first node's CPUs. Going through num_node_state(N_CPU) and for_each_node_state(node, N_CPU) actually rotates. Living in xenbus_client.c rather than open-coded in each driver lets future scsiback / pvcalls-back style frontends pick it up for free. Signed-off-by: Steven Noonan <steven@edera.dev>
Today both xenvif kthreads (guest-rx and dealloc) and the per-queue event-channel IRQs run wherever the scheduler happened to place them, which in PVH dom0 is typically all on one CPU regardless of the guest's vNUMA layout. When a guest's per-queue ring lives on a different host node from the kthread, every packet pays cross-node interconnect cost to walk the ring and to grant-copy payload pages. Use xenbus_ring_host_node() to find the host node hosting the queue's tx ring, then: - Create the guest-rx and dealloc kthreads with kthread_create_on_node() so the task_struct (and kernel stack) is allocated on the target node from the start. A bare set_cpus_allowed_ptr() after kthread_run leaves the stack on the caller's node and continues to pay cross-node cost on every context switch. - Pin each kthread to the node's cpumask with set_cpus_allowed_ptr(). This is a hard pin; an operator can still override with taskset. No-op when no node info is available. - Bind each evtchn IRQ on the ring's node using bind_interdomain_evtchn_to_irqhandler_lateeoi_on_node() (and the non-handler variant for the ctrl ring), so the underlying desc is allocated with the right node attribute and irqbalance treats the IRQ as NUMA-local rather than floating. For the ctrl ring this means resolving host_node before the bind instead of after. - Steer the tx, rx, and ctrl IRQs with irq_set_affinity_and_hint(). This writes both the actual affinity and the affinity_hint, so a fresh boot without irqbalance routes IRQs correctly and irqbalance agrees if it is running. Operator writes to /proc/irq/N/smp_affinity still win. - Clear the hint with irq_update_affinity_hint(irq, NULL) right before each unbind_from_irqhandler() in xenvif_disconnect_queue and xenvif_disconnect_ctrl. free_irq() warns if the hint is still set at teardown (kernel/irq/manage.c:1865); destroying a domU would otherwise WARN in xenwatch context. When xenbus_ring_host_node() returns NUMA_NO_NODE (older Xen without the underlying hypercall, kernel built without CONFIG_XEN_BACKEND_NUMA_AFFINITY, or a PV mapping), every NUMA-aware step is skipped and the behaviour is byte-for-byte identical to the previous code path. Signed-off-by: Steven Noonan <steven@edera.dev>
The xenblkd kthread and per-ring event-channel IRQ today run wherever the scheduler happens to place them, which in PVH dom0 is typically all on one CPU regardless of the guest's vNUMA layout. When the guest's per-ring buffer lives on a different host node from the worker, every request pays cross-node interconnect cost to walk the ring and to grant-copy payload pages. The blkback flow is split across two call sites: xen_blkif_map() maps the ring and binds the event channel, while xen_update_blkif_status() spawns the xenblkd kthread later. Stash the ring's host node on the xen_blkif_ring at map time so the kthread creation site can pick it up without re-querying. At map time: - Call xenbus_ring_host_node() right after xenbus_map_ring_valloc() succeeds. Store the result in ring->host_node. - Bind the IRQ on the ring's node using bind_interdomain_evtchn_to_irqhandler_lateeoi_on_node() so the underlying desc is allocated with the right node attribute and irqbalance treats the IRQ as NUMA-local. - Steer the bound IRQ with irq_set_affinity_and_hint() so a fresh boot without irqbalance routes correctly and irqbalance agrees if it is running. Operator writes still win. At kthread creation time: - Use kthread_create_on_node() with ring->host_node so the task_struct (and kernel stack) is allocated on the target node from the start. A bare set_cpus_allowed_ptr() after kthread_run would leave the stack on the caller's node and pay cross-node cost on every context switch. - Hard-pin the kthread to the node's cpumask with set_cpus_allowed_ptr(). An operator can still override with taskset. At disconnect time: - Clear the hint with irq_update_affinity_hint(ring->irq, NULL) right before each unbind_from_irqhandler(). free_irq() warns if the hint is still set at teardown (kernel/irq/manage.c:1865); destroying a domU would otherwise WARN in xenwatch context. ring->host_node is initialised to NUMA_NO_NODE in xen_blkif_alloc_rings so any path that misses the map step (or runs on a hypervisor without XENMEM_get_mfn_pxms, or a kernel without CONFIG_XEN_BACKEND_NUMA_AFFINITY) falls back to today's NUMA-oblivious behaviour without further checks. Signed-off-by: Steven Noonan <steven@edera.dev>
Stack mapping entropy is currently hard-wired to 11 bits of entropy on 32-bit and 22 bits of entropy on 64-bit. The stack itself gains an extra 8 bits of entropy from lower bit randomization within 16 byte alignment constraints. The argument block could have all lower bits randomized but it currently only gets the mapping randomization. Rather than hard-wiring values this switches to using the mmap entropy configuration like the mmap base and executable base, resulting in a range of 8 to 16 bits on 32-bit and 28 to 32 bits on 64-bit depending on kernel configuration and overridable via the sysctl entries. It's worth noting that since these kernel configuration options default to the minimum supported entropy value, the entropy on 32-bit will drop from 11 to 8 bits for builds using the defaults. However, following the configuration seems like the right thing to do regardless. At the very least, changing the defaults for COMPAT (32-bit processes on 64-bit) should be considered due to the larger address space compared to real 32-bit. Signed-off-by: Daniel Micay <danielmicay@gmail.com> Signed-off-by: Ariadne Conill <ariadne@ariadne.space>
This code mostly inspired by linux-hardened patch set. Signed-off-by: Ariadne Conill <ariadne@ariadne.space>
Signed-off-by: Ariadne Conill <ariadne@ariadne.space>
Signed-off-by: Daniel Micay <danielmicay@gmail.com> Signed-off-by: Ariadne Conill <ariadne@ariadne.space>
Previously we would read randomize_va_space in openpax_setup_flags, and then again later after calling openpax_setup_flags. As randomize_va_space is a sysctl, its value can change at any time. Accordingly, snapshot randomize_va_space before calling openpax_set_flags and pass the snapshot value to openpax_set_flags. This resolves a minor race condition where one may be able to slightly weaken process ASLR by turning ASLR on or off during an exec call. Signed-off-by: Ariadne Conill <ariadne@ariadne.space>
Otherwise we have to add a bunch of #ifdefs when we otherwise would not need to. Fixes: a78588f ("init: add pax_softmode= kernel command-line option") Fixes: edera-dev#1 Signed-off-by: Ariadne Conill <ariadne@ariadne.space>
XENMEM_populate_physmap takes an extent order, but the only wrapper exposed here pins it to EXTENT_ORDER, the order that covers one native page and nothing more. Add a variant that takes an additional order, so a caller can ask Xen to back a run of pages with a single machine-contiguous allocation rather than one frame per page. For a PV domain Xen reports back only the base machine frame of each ordered extent, the rest of the extent following it. For a PVH domain it reports nothing, as the guest never handles machine frames. No functional change: xenmem_reservation_increase() becomes a wrapper at order 0 and the new variant has no callers yet. Backends: PV, PVH Direction: deflate (frames move from Xen to the guest) Signed-off-by: Alexander M. Merritt <alexander@edera.dev>
A PV domain's page tables hold machine frames, so moving a page between
the guest and Xen means rewriting its direct map PTE through Xen. The
per-page helpers in drivers/xen/mem-reservation.c make one
HYPERVISOR_update_va_mapping call per page.
Add exported helpers that do the same for a contiguous run in batches,
through HYPERVISOR_mmu_update as xen_remap_pfn() does:
- xen_zap_contig_pfns() unmaps a run and invalidates its p2m entries,
for pages about to be handed to Xen.
- xen_remap_contig_pfns() maps a run onto frames Xen has just
populated. It writes the p2m with __set_phys_to_machine(), which
never allocates, so the caller runs xen_prealloc_p2m_range() over
the run first, before the populate hypercall, while a failure can
still be backed out of.
Both return the first failure, whether a PTE they could not find or Xen
refused to update, or a p2m entry that could not be written, and leave
what to do about it to the caller. mmu_update reports a failure back to its caller, where a
multicall batch would only log it.
Neither flushes the TLB: the intended caller only maps over PTEs that
were already cleared, and flushes after unmapping itself.
PVH domains never reach these: their page tables hold guest frames,
which do not change when Xen moves the machine frames behind them.
No callers yet.
Backends: PV only
Direction: inflate and deflate
Signed-off-by: Alexander M. Merritt <alexander@edera.dev>
A PV domain's page tables hold machine frames, so moving a page between
the guest and Xen means updating its direct map PTE and p2m entry. The
existing helpers for that take parallel arrays of pages and frames, one
entry per page. That fits order-0 extents, where Xen reports a frame per
page, but not an ordered extent, where it reports only the base frame.
Add variants that take a base page, and for mapping a base frame, and
derive the rest of a contiguous run from them, on top of the batched arch
helpers:
- xenmem_reservation_va_mapping_reset_contig() unmaps a run before its
frames are handed to Xen.
- xenmem_reservation_p2m_prealloc() and
xenmem_reservation_va_mapping_update_contig() map a run in once Xen
has populated it. The mapping step cannot allocate, so the
preallocation is a separate call, made before asking Xen to
populate: it is the only step expected to fail, and failing after
the populate would strand the frames Xen handed over.
The two directions treat a failed update differently:
- A failed map is fatal, as in the per-page helper. Callers treat the
run as mapped once the call returns, and a page with no mapping
would fault wherever it was next used, far from the cause.
- A failed unmap only warns, unlike the per-page helper. The frame
stays mapped, and Xen will not hand it to another domain while it
is, so the frame leaks but nothing is corrupted.
All three are no-ops outside a PV domain, like the helpers they sit
beside.
No functional change: the new helpers have no callers yet.
Backends: PV only
Direction: inflate and deflate
Signed-off-by: Alexander M. Merritt <alexander@edera.dev>
The balloon inflates a page at a time: it takes whatever pages the guest
buddy allocator hands out and returns their frames to Xen one order-0
extent each. Those frames need not form an aligned run, so Xen's
free_heap_pages() cannot merge them back into a 2 MiB buddy.
Add a second unit of trade, a block of 1 << 9 Xen pages: 2 MiB, the
largest extent Xen grants a domain asking on its own behalf. Inflate now
takes a whole free block with one alloc_pages() and parks it on a block
list, so its frames go back to Xen as an aligned run. On PV the block is
unmapped with the batched contiguous-run helper; on PVH there is nothing
to unmap.
The extents submitted stay order-0. XENMEM_decrease_reservation
releases each frame individually whatever the extent order, so an
ordered extent would change nothing there while asking Xen to trust that
the block is machine-contiguous, a claim it cannot check for a PV
domain, where the frame number it is handed is already the machine
frame.
A block is only an option, never a requirement, in either direction:
- balloon_inflate() takes a block only when the guest has a whole free
2 MiB run, without reclaiming or compacting to make one, and
otherwise inflates a page at a time. A guest with plenty of free
memory can still have no such run, and that must not stop it
shrinking.
- Deflate still populates a page at a time and walks only the page
list, which cannot see blocks. When that list is empty it now breaks
a parked block onto it, so the guest can always grow back into
memory it gave up as a block.
The OOM notifier sized its deflate from the page list counters alone,
so a balloon holding only blocks would look empty to it and the OOM
killer would run while the balloon still held memory. It now counts
block pages too, and its deflate reaches them through the block-breaking
path above.
Set xen.balloon_superpages=0 to inflate a page at a time as before.
Backends: PV, PVH
Direction: inflate (frames move from the guest to Xen); deflate only
changes to consume blocks
Signed-off-by: Alexander M. Merritt <alexander@edera.dev>
Deflate still breaks a parked block into pages and asks Xen to populate
each with its own order-0 extent, which lets Xen back the block with any
512 frames it has.
Populate a parked block as a single order-9 extent instead, so that Xen
allocates one machine-contiguous 2 MiB run for it:
- On PVH Xen maps that run with a single 2 MiB EPT entry rather than
512 4 KiB ones.
- On PV there is no EPT, and the guest cannot use 2 MiB mappings. The
guest maps the run in itself, with the batched contiguous-run
helper, from the base machine frame Xen reports.
The p2m leaves of each block are preallocated while walking the block
list, before the populate hypercall: mapping the block afterwards cannot
allocate, and once Xen has populated an extent there is no way to hand
it back. Failing there only shortens the batch.
A block is only an option here too. balloon_deflate() falls back to the
page at a time path when a block makes no progress: Xen refuses an
order-9 extent when the host has no free 2 MiB run, or when the domain
is less than 2 MiB below its max_pages ceiling, and neither should stop
the guest growing by what it can still get. Credits smaller than a block
take the page at a time path as well.
Blocks are populated at most BALLOON_BLOCK_BATCH at a time, which bounds
the work done under balloon_mutex; the OOM notifier can only trylock it.
Backends: PV, PVH
Direction: deflate (frames move from Xen to the guest)
Signed-off-by: Alexander M. Merritt <alexander@edera.dev>
The OOM notifier deflates through increase_reservation(), which works a page at a time and reaches parked blocks only by breaking them up. Deflate through balloon_deflate() instead, so that an OOM deflate populates a whole block as a single order-9 extent where Xen can supply one, and still falls back to single pages where it cannot, including the last 2 MiB below the domain's max_pages ceiling. Backends: PV, PVH Direction: deflate (frames move from Xen to the guest) Signed-off-by: Alexander M. Merritt <alexander@edera.dev>
xen_init_time_common() points x86_platform.calibrate_tsc at xen_tsc_khz() but leaves calibrate_cpu native, so tsc_early_init() -> determine_cpu_tsc_frequencies() still calls native_calibrate_cpu_early() for cpu_khz. That tries CPUID 0x16, then MSR_PLATFORM_INFO, and then falls through to quick_pit_calibrate() -- which on a PVH guest waits out its full 50ms timeout on a PIT that is not there, prints "Fast TSC calibration failed", and arrives at a value the hypervisor had already been asked for a moment later. It is 48ms of every boot, and it is the largest single item in one: a 512 MiB PVH guest reached "Run /init as init process" at 0.330s, and 0.118s once this and the xenbus boot wait were out of the way. There is nothing for the calibration to find out here. Xen reports the TSC frequency, tsc_khz already comes from it, and cpu_khz differing from tsc_khz under a hypervisor is not information -- determine_cpu_tsc_frequencies() would overwrite cpu_khz with tsc_khz anyway once they disagree by more than 10%. Note that tsc_early_khz= does not help: it only short-circuits calibrate_tsc, and the 50ms is spent in calibrate_cpu. Set in xen_init_time_common() so PV and PVH/HVM guests both get it, which is where calibrate_tsc is already set for the same reason. Signed-off-by: Alex Zenla <alex@edera.dev>
boot_wait_for_devices() sleeps HZ/10 between checks, so on a CONFIG_HZ=100 kernel it waits a tenth of a second to notice something that has already happened. What it is waiting for is not slow: a toolstack has every backend sitting in XenbusStateInitWait before it unpauses the guest, so the wait covers the frontend half of the handshake, which finishes a few milliseconds after the late_initcall starts looking. Measured on a PVH guest, the last frontend connected at 0.232s and init started at 0.311s -- 79ms of sleeping past the event. Frontends move to XenbusStateConnected (and blkfront sets is_ready) from their otherend_changed callback, which runs from the frontend bus's backend_changed() watch. Bump a sequence count and wake a wait queue there, and have wait_for_devices() sleep on it instead of for a fixed interval. The sequence is sampled before each check, so a change that lands between the check and the sleep still wakes the waiter. The HZ/10 interruptible sleep stays as the timeout, so state changes that do not come through the watch are noticed no later than before, and the 30/270-second budgets and their reporting are unchanged. Signed-off-by: Alex Zenla <alex@edera.dev>
Port the nightly upstream rebase from the Xen tree to the kernel. Every night at 03:23 UTC, Claude rebases edera/mainline onto torvalds master, following .automation/skills/upstream-rebase/SKILL.md: it replays the series, resolves conflicts, fixes what the replay broke, builds, audits upstream fixes that meet the series, and writes a report. The model's own account of its result decides nothing. It runs with a read-only token; its branch leaves the job as a git bundle and is judged by scripts taken from the commit the run started from: - verify-upstream-rebase.sh (unchanged from Xen) proves the series sits on the upstream tip, is linear, kept every commit, and moved the tree by exactly the upstream delta; - kernel-build.sh builds x86_64 and arm64 vmlinux with the Hyper-V, Xen dom0/backend, PV-IOMMU, NUMA affinity and OpenPaX options built in, and fails if olddefconfig drops any of them. If both pass and the report asks for no decision, edera/mainline is force-pushed with a lease pinned to the tip the rebase started from, and the old tip is kept as a backup branch. Otherwise the result goes up as a pull request carrying the report, and approving it lands it (rebase-land.yml). A night with no result opens an issue. Every job clones blobless: the jobs need the kernel's commit graph but only the contents of the files the series and upstream touch.
The rebase runs from a scheduled or dispatched event and needs its own federation rule rather than the organization-wide one, so read it from vars.REBASE_FEDERATION_RULE_ID.
edera/6.18-lts carries its own rebase automation, but scheduled workflows run only from the default branch. Add a workflow here whose only job is to dispatch edera/6.18-lts's kernel-nightly.yml each night.
OPENPAX currently defaults to y, which means OpenPaX gets enabled for any configuration that picks up the new symbol, including ones that never asked for it. Default it to n so it has to be selected explicitly. While at it, make OPENPAX_SOFTMODE depend on OPENPAX. Soft mode does nothing without OpenPaX enabled, so there is no reason for it to be selectable (and default y) on its own. Signed-off-by: Alex Zenla <alex@edera.dev> (cherry picked from commit 514cd60)
edera/6.18-lts proposes a backport of each pull request merged into edera/mainline, driven by its own workflows. Add a workflow here whose only job is to dispatch that, skipping pull requests labelled no-backport and the nightly's rebase pull requests.
xen-pciback seizes a hidden device in an fs_initcall, but the Xen IOMMU registered from a device_initcall. Registering probes every PCI device that already exists, so a hidden device was found with a driver bound and the IOMMU core warned about a late probe at driver bind. Register from subsys_initcall_sync instead, so the IOMMU is in place before pciback binds anything. Signed-off-by: Alex Zenla <alex@edera.dev>
Signed-off-by: Alexander M. Merritt <alexander@edera.dev>
Signed-off-by: Alexander M. Merritt <alexander@edera.dev>
All of the extra memory Linux reserves at boot goes to the mapping pool, so dom0's balloon starts empty. Growing dom0 at runtime then means memory hotplug, which places the new range at the first free physical address: on PVH that is the host's own map, where nothing guarantees the range is not a device's MMIO window. Keep the part of the extra memory dom0 may grow into on the balloon instead. Xen's ceiling for dom0 (dom0_mem=...,max:) sizes that part, so a host booted without a max: keeps today's behaviour. Every region gives up the same fraction, which spreads the growth over the NUMA nodes on PVH and leaves each node's mapping pool its share, and the last memory section stays with the pool so that early grant mappings never fall back to holes in the physical map. Signed-off-by: Alexander M. Merritt <alexander@edera.dev>
A request that does not fit in its ring sleeps in io_wait_event_killable() until the backend frees space, but the event handler wakes the ring with wake_up_interruptible(), which only wakes TASK_INTERRUPTIBLE sleepers. A TASK_KILLABLE waiter therefore sleeps until it is killed, which hangs a guest that has enough large requests in flight to fill a ring. Wake the ring with wake_up() so killable waiters see the freed space. Fixes: 9523fea ("net/9p: Switch to wait_event_killable()") Signed-off-by: Alex Zenla <alex@edera.dev>
The frontend's resume handler only warned that suspend and resume are unsupported. After a restore its rings, grant references and event channels are gone, yet the mounted client kept writing requests into them, so every request on the mount hung. The next InitWait from the backend then allocated a second frontend instance for the same device, which no client used. On resume, collect any replies the backend left in the rings, free the rings, fail the requests that are still unanswered with -EIO, and publish Initialising so that a backend which still serves the old rings starts over. When the backend reaches InitWait, allocate and publish new rings for the existing frontend, keeping its clients. Requests that arrive while the frontend is not connected wait until it is, and a rw_semaphore keeps them off the rings while a resume replaces them. Signed-off-by: Alex Zenla <alex@edera.dev>
A forked, restored or imported domain resumes against a new backend whose fid table is empty, while the guest's 9P clients still hold the fids of the session they were suspended with. With the transport reconnected every request on those fids still fails, so even an idle mount comes back unusable. When the backend advertises feature-session-transfer, wait at suspend for the requests in flight to be answered, then ask the backend for its fid table with Tsession_save and keep the pages in guest memory, where they survive into whatever domain the guest resumes in. Once a restored frontend's new backend connects, replay the pages with Tsession_restore before any request can use the rings. The pages are opaque to the frontend; the backend validates every entry as it would a walk or open. Signed-off-by: Alex Zenla <alex@edera.dev>
The toolstack has to know whether a guest can carry its 9P session into a new backend before it exports the domain, because a guest that cannot comes back with every open file on its mounts unusable. Until now only the backend advertised the feature, and the frontend's support was invisible from dom0. Publish feature-session-transfer = 1 in the frontend's own xenstore directory, next to version and num-rings. Signed-off-by: Alex Zenla <alex@edera.dev>
arch/x86/xen/enlighten.c includes xen/interface/memory.h for XENMEM_maximum_reservation, but memory.h uses xen_ulong_t, domid_t and GUEST_HANDLE without including where they are defined, xen/interface/xen.h. Other configurations reach those through an earlier include. The zone-tiny configuration does not, and fails to build with CONFIG_WERROR=y: include/xen/interface/physdev.h:41:5: error: unknown type name 'xen_ulong_t' Include xen/interface/xen.h first, as arch/arm/xen/enlighten.c does. Signed-off-by: Alex Zenla <alex@edera.dev>
A userspace backend that serves guest block I/O has to move data between a file and the pages the guest granted it. Today that means mapping the grants with mmap(), then either copying through a bounce buffer or issuing O_DIRECT I/O on the mapping, which pins every page with GUP for each transfer and looks each page up in its own VMA. xen-blkback avoids both: it hands the struct pages of its grant mappings to the block layer directly. Give gntdev the same ability. Add GNTDEV_URING_CMD_READ and GNTDEV_URING_CMD_WRITE, issued as IORING_OP_URING_CMD on a gntdev fd from an SQE128 ring. The payload names a live mapping by its index, a list of (page, offset, len) segments, and a target file descriptor and position. The segments become a bio_vec array over the mapping's pages and go to the target file through vfs_iocb_iter_read/write(), so the transfer is direct if the target was opened O_DIRECT and buffered otherwise. Lifetime: the command holds a reference on the map and on each page for the whole transfer. gnttab_unmap_refs_async() already defers while a page has an extra reference, so a transfer cannot outlive its grant. A command takes its page references and then re-reads being_removed; the unmap side sets being_removed and then reads the page counts, and a full barrier on each side orders the two. A command that loses that race fails with -ENOENT. Completion can run in interrupt context, so the request is released and the CQE posted from task work, where dropping the last map reference may unmap grants. Checks on the guest-influenced input: the page index, offset and length of every segment are validated against the mapping and the page size, a read needs a writable mapping, and PV domains are refused because their pages carry foreign MFNs the block layer cannot use. Signed-off-by: Alex Zenla <alex@edera.dev>
kaniini
force-pushed
the
edera/mainline
branch
from
October 23, 2026 16:50
4e04bef to
a90d5db
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds two
IORING_OP_URING_CMDcommands to gntdev,GNTDEV_URING_CMD_READandGNTDEV_URING_CMD_WRITE. They move data between a file and pages of a live grant mapping without bouncing it and without GUP-pinning the pages for each transfer.The goal is zero-copy block I/O in the userspace protect-vbd backend (edera-dev/protect#2809). Today that backend copies each request through a staging buffer, because it assumed foreign grant pages cannot be direct I/O targets.
xen-blkbackalready does direct I/O on these pages by handing theirstruct pages to the block layer, so this gives gntdev the same path.How it works
IORING_SETUP_SQE128ring.struct gntdev_uring_io) names an existing mapping by its index, a user array of(page, offset, len)segments, a target fd and a file position.bio_vecarray overmap->pagesand go to the target file throughvfs_iocb_iter_read/write(). The transfer is direct when the target fd was openedO_DIRECT.Lifetime and races
gnttab_unmap_refs_async()already defers while a page has an extra reference.being_removed. The unmap side setsbeing_removedand then reads the page counts. A full barrier on each side orders the two: either the unmap sees our reference and waits, or the command sees the page marked as removed and fails with-ENOENT. The barrier on the unmap side is the one change to existing code (__unmap_grant_pages()).gntdev_privalive for the life of the command.Input checks
GNTDEV_URING_MAX_SEGS(256), and at least one.map->count, non-zero length, offset and length inside one 4 KiB page.GNTMAP_readonly).loff_t.-EOPNOTSUPP; their pages carry foreign MFNs the block layer cannot use.Not verified
CONFIG_IO_URING) before review.MEMORY_DEVICE_GENERIC) pages.xen-blkbacksuggests it does.Risks
xen-blkbackhas the same property.