watchthelight labs

System sleep and wake

Before this change the driver had no system sleep support. On a desktop the PCIe slot loses power in S3, so the GPU came back reset while NVRM still thought GSP-RM was running. On wake NVRMFB re-lit its head through NVKMS, the push to the core channel spun in nvEvoMakeRoom forever, and the Mac needed a hard reset ("Darkwake Exit Failure", Sleep Wake Failure report). The work below makes sleep and wake behave as on Linux, plus the steps macOS needs on top.

Tested on an RTX 5050 (GB207, FSP boot), MacPro7,1 SMBIOS, macOS 15.8.1: repeated sleep and wake with the desktop, windows and Metal apps intact.

What happens

Step Where What
System is about to sleep NVAccel A root-domain priority power interest opens the sleep window: page-offs IOGraphicsAccelerator2 does now keep their VRAM (see below).
Display goes off NVRMFB As before (DPMS).
NVRM setPowerState(0) NVRM Tell the framebuffers; nvKmsSuspend (heads off, while the GPU still serves the core channel); close the RM gate; stop user channels; save PCI config, Resizable BAR control and the root port window; rm_power_management(STANDBY).
— RM With PreserveVideoMemoryAllocations=1 RM copies every VRAM allocation out through its temporary-file interface. NVRM backs that "file" with pageable kernel memory (no disk I/O). GSP-RM is unloaded.
Slot powered again, NVRM setPowerState(1) NVRM Restore the root port window and the Resizable BAR size (the firmware resets BAR1 to 256 MB), check PMC_BOOT_0, rm_power_management(RESUME) (GSP boots again through FSP and VRAM is copied back), restart user channels, nvKmsResume, open the RM gate, tell the framebuffers.
NVRMFB setPowerState(1) NVRMFB Re-light the head once NVRM says the GPU is back, then do IOGraphics' own wake work (below).
CPU back on NVAccel The sleep window closes: page-offs give VRAM back again.

Why each part is needed

NVRM in the power tree. NVRM was not a power child of the GPU's IOPCIDevice, so nothing suspended or resumed RM. It now joins the tree under the PCI device. -nvrmnopm restores the old behaviour.

All of VRAM is preserved. By default RM saves only its own allocations. WindowServer's GPU channels (NVK through NVAccel) keep push buffers and semaphores in VRAM; without them the compositor waited on fences forever after wake. NVRM sets the RM registry key PreserveVideoMemoryAllocations=1. On Linux RM then writes VRAM to a file under NVreg_TemporaryFilePath; here os_open_temporary_file/os_write_file/os_read_file (os-xnu2.cpp) store it in 4 MB chunks of pageable kernel memory, so the VM can compress it on a machine short of RAM, with wired memory as the fallback. RAM keeps power in S3, and no disk is touched while the system goes to sleep. The RM itself is unmodified.

The RM gate. Linux freezes user space before suspend and also takes nv_system_pm_lock in every ioctl. macOS freezes nothing: on wake WindowServer and Metal apps called into RM while the GPU was still in standby, which corrupted GMMU state or handed errors to NVK. Every RM entry made for a client (escape ioctls, client close, NVAccel's GPU VA and surface-share calls, nvkms_call_rm) now waits at a gate that the PM thread closes for the suspend and opens after the resume. NVKMS has its own gate: ioctls wait (30 s at most), flips fail at once, timers wait.

PCI state. IOPCIFamily restores the config space it shadows, but not the Resizable BAR control register, and the firmware's S3 resume path resets BAR1 to 256 MB. NVRM writes back the BAR1 size, the BARs and the root port window it saved before standby.

NVRMFB and IOGraphics. NVRMFB overrides IOFramebuffer::setPowerState, so the family's own power work never runs, and IOFramebuffer::handleEvent() does nothing off the IOGraphics controller's power thread. At system sleep IOGraphics clears pagingState and sends kIOFBNotifyWillSleep to every client; after wake WindowServer's wake acknowledgement (_extEntry) slept until pagingState came back, and the accelerator stayed disabled. On a wake after a GPU standby NVRMFB now does what IOGraphics does (IOFBController::computeState and handleEvent(kIOFBNotifyDidWake) in IOGraphics-599): deliver DidWake with true and then false (the second re-enables the accelerators), deliver DidPowerOn, set pagingState and wake WindowServer on fServerConnect, under the framebuffer's work loop gate. It never pushes to NVKMS before NVRM has resumed the GPU (NVRMGpuReady, NVRMGpuPower).

NVAccel and VRAM across sleep. At sleep IOGraphicsAccelerator2 pages off its VRAM resources, and the wake's mode change pages them back on. If page-on got fresh VRAM, WindowServer kept drawing into the memory it had imported before sleep while the display showed the new one: a black desktop with only the cursor. NVAccel's sticky VRAM (debug.nvaccel_vm_sticky, on by default) keeps a surface's VRAM across page-off, and its next page-on takes the same VRAM back; RM has kept the contents through the sleep, so nothing is lost.

Switches and diagnostics

Name Kind Effect
-nvrmnopm boot-arg NVRM stays out of the power tree and keeps the stock RM settings: the behaviour before this change.
debug.nvaccel_vm_sticky sysctl (rw) 1 (default) a surface keeps its VRAM across page-off; 0 page-off gives it back and the desktop is black after wake.
debug.nvaccel_vm_parked, debug.nvaccel_vm_unparked sysctl Page-offs that kept their VRAM, and page-ons answered with it.
debug.nvrm_pm_waited sysctl RM calls that waited at the gate for a resume.
debug.nvrm_isr_count, debug.nvrm_isr_handled sysctl GPU interrupts taken and claimed by RM.
nvrm-pm NVRM registry property running, suspended or lost (resume failed).

Kernel log lines to look for (/usr/bin/log show --predicate 'process == "kernel"'): NVRM-xnu: sleep #, NVRM-xnu: wake #, VRAM save file closed: N MB held, NVRM-fb: system wake:.

Not covered

  • NMIntelFB (laptops whose panel is on the Intel GPU) has the same setPowerState/handleEvent pattern as NVRMFB and does not get the IOGraphics wake work yet.
  • Hibernation (hibernatemode 3/25) is untested. Desktops default to hibernatemode 0.
  • An open SSH session (ttyskeepawake 1) turns Apple menu → Sleep into a dark wake, not a sleep.
  • Some boards wake every minute through USB (pmset -g log: DarkWake ... due to XDCI ... XHCI). That is the firmware's USB wake GPE (0x6D), not the driver; the usual GPRW → XPRW rename with SSDT-GPRW stops it, and then only the power button wakes the Mac.