The second import-window case of agc_write_tracking makes one page
unreadable with VirtualProtect or mprotect and expects an import that
stays watched to throw before it copies its range. On Windows CheckRange
answers from its table of committed arena pages, which is kept current
by the registry's invalidator. The first case had checked the same
allocation, so the table still said the page was readable, the check
passed and the copy faulted on the page.
ProtectPage now reports the change through GuestAllocationsInvalidate,
as the registry does for a guest protection change and as
tests/BdaResources.cpp does after VirtualProtect.
A host import that leaves its range watched can resolve the write
protection of the imported pages without storing anything, as NVIDIA's
Linux driver does for every page. ImportWatched stamped those pages as a
possible CPU store (BlockMaybeWritten), and only a storage image tracked
by 64 KiB unit could clear the stamp, by comparing with a copy of the
guest bytes it was uploaded from. Since #3372 a fast-cleared upload
therefore reads its whole surface from guest memory wherever such an
import is possible, although nothing but that compare uses the copy.
ImportWatched now takes whether the import leaves its range watched. If
it does, the range is copied before the stores made so far are
collected, and after the import each page whose write protection was
resolved is compared with that copy. A page with other bytes is stamped
as a CPU store. A page with the same bytes keeps only the CPU-side stamp
the label rules read, so UnchangedSince holds for it for every consumer:
storage images tracked by layer, snapshots and caches as well.
If it does not (a host-pointer import on a device whose imports are
compared, a direct import on Windows), nothing is copied or compared:
the pages resolved in the window get the CPU-side stamp and the range
leaves the watch before the tracker is unlocked. Where host-pointer and
dma-buf imports differ in this, the dma-buf import that follows a failed
host-pointer import is a window of its own, so each window knows its
kind before it starts.
BlockMaybeWritten, the written-stamp table, WatchedImportsStampPages and
the unit compare in StorageTexture::Refresh are gone. The fast-clear
upload skips the guest-byte copy whenever its range is watched, as
before #3372.
On Windows the guest arena is reserved while libc loads, around whatever
the process has already mapped in its interval. About one start in forty
has nothing mapped there, so the arena has no host hole:
agc_driver_graphics then fails with "write-watched arena contained no
host exclusion to test" and agc_write_tracking prints that its host-hole
checks were skipped.
agc_write_tracking now starts itself suspended, reserves one allocation
granule in the middle of the child's arena interval before the child's
loader runs, and the child runs the host-hole checks on that range. The
range is a host hole on every start, so the checks no longer depend on
the address layout. The four queries agc_driver_graphics made on an
enumerated hole are among them and are removed there.
The memo stamp held TrackerGeneration, which every collect and driver store
advances, so it almost never matched. Driver stores into uncovered ranges
record their generation per 64 KiB block, and NewestStamp of the surface
replaces TrackerGeneration in the stamp.
On Windows agc_write_tracking has failed since #3594 with
write tracking test failed: an unwatched range reads as not stored over
from the check in CheckSharedBlock:
if (!Watched(outside, unwatched.size())) Require(StoredOver(outside, unwatched.size(), TrackerGeneration()), "an unwatched range reads as not stored over");
The host stack array lies in a host hole of the guest arena. #3594 made
Watched, collectWrites, UnchangedSince, UnchangedSinceAll and storeOwn
reject ranges overlapping such holes, but StoredOver, MarkWritten,
WrittenSince, ChangedBlocks and ImportWatched still accepted them through
WriteTracker::covers(), so StoredOver proved a range not stored over while
Watched called it unwatched (#4087).
The exclusion now lives in WriteTracker::covers() on Windows, which every
query already asks, and the per-function checks are removed. Host holes
are padded to the allocation granularity and the tracker blocks are 64 KiB
from the arena base, so no block is shared between a hole and arena
memory, and ranges outside the holes get the same answers as before.
A new case enumerates the arena for a host hole and checks every query
against it; on the previous code StoredOver, MarkWritten, StoreOwnBytes,
WrittenSince and ChangedBlocks fail it.
Fixes#4087
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Scan version stamps within contiguous leaves. On Linux, collect nearby
pages within queue epochs only after confirming full watched coverage.
Keep the requested range on Windows. Select the range before collection
and do not retry a failed walk.
Test sparse-leaf boundaries, write histories, partial unwatches and epochs.
Fedora 44 Release tests pass with default, disabled and shared caching.
Isolated upstream comparison reduces the saved collect replay by 11-12%.
Windows was not tested.
AI-assisted: yes.
StorageTexture::writeBackLayers keeps the CPU's bytes for an edge unit when StoredOver
reports that the CPU wrote it since the unit's generation. That check read one generation
per 64 KiB block, so a store to another allocation sharing the block counted as a store
over this surface and the whole block was kept, discarding GPU results written into it.
The driver path in the same check already compares range pieces; the CPU path now does
the same. Each collect records the generation of every page it found dirty, and StoredOver
tests only the pages its range overlaps. A block with no page record keeps the previous
whole-block answer.
Measured on an RTX 3080 Ti, main at df16c4c2:
- agc_write_tracking stores at base + Block + 8, outside the queried range but inside the
shared block. That case fails on main ("a CPU store outside the shared range counts for
it") and passes here.
- Full suite 408 of 408 (Linux, CachyOS, GCC, Release, BUILD_TESTING=ON).
This does not cover the other half of #865: the report's finding 2, where the stamps fall
on the surface's own pages and the bytes under them are unchanged, still keeps the block.
No title was run here, so the fix is offered for the report's own repro.
A store the driver makes with the CPU into watched guest memory (a
GuestMemory::Write, a changed-bytes write-back, a completion action's label
store, the copy HLE's CPU path) faults its pages dirty in the write watch and
stamps the range with MarkWritten. The faults stayed behind: collects are
memoized per packet, so the next walk over those pages came later and stamped
them again, as a CPU write made after every check in between.
PPSA21564 fast-clears its normal G-buffer every frame with a register DCC key
fill on the GPU. While that fill is still pending, the first draw into the
target materializes the clear with a CPU store of the clear texels and a
refresh of the resident image. A dozen draws later a walk found the store's
faults, the target read as changed, and the image was uploaded again from
guest memory, dropping the draws recorded in between. Lighting then sampled
cleared normals and the level's moon and rocks went unlit in about half the
frames.
The driver's stores now consume those faults themselves: a walk before the
store stamps earlier CPU writes as such, a walk after it takes the store's
own faults as a driver store, and the range is stamped as before.
On Windows the arena's range also holds host regions that were already in
use when it was reserved (heaps, thread stacks). They are not write-watched,
so a store into one is stamped as before, without a walk.
Write stamps are per 64 KiB block, so a write-back of one surface stamps the
whole block it shares with the surface packed right after it. A pending
image of that neighbour read the stamp as a store over its own bytes: its
write-back kept the guest bytes of the shared block in place of its results
and its next refresh re-uploaded them, stale. PPSA21564's half-resolution
fog texture (960x540 RGBA16F, linear, starting 0xb800 bytes before a block
end right after another 960x540 surface) lost its first 6 rows that way,
which the fog composite upsampled into an orange band across the top 12
rows of the scene in one of its cinematics.
The tracker now remembers the byte ranges of the driver's own stores
(MarkWritten) in the blocks they cover only in part, and StoredOver says
whether a store reached a given range since a generation (a CPU store still
counts for its whole block). The write-back keep rule and the refresh drop
rule ask it for the blocks a surface covers only in part.
umang
Dean
Dean Galvin
ThatsLeo
Scramblrrrr
screamm
TibixDev
boykopovar
ToyuAU
Dextroid17