Key 0x10 (GFX9_DCC_CLEAR_SINGLE in Mesa) marks a 256-byte block whose texels all equal its first texel. A CB DCC decompress pass over uniform comp-to-single keys now writes that texel over its block and stores uncompressed keys; a fast-clear eliminate over them throws.
RADV rounds a strided buffer descriptor's record count up, so stride * records
can run up to one stride past the end of the buffer's allocation. The guest
buffer region then no longer fits inside the allocation's host import and is
copied on the CPU, whole, for every draw that binds it.
When a strided descriptor ends less than one stride past the end of the
readable registered allocation that contains its base, the range is bound up
to the allocation's end. RetroArch's Dolphin core binds a 48 MiB stream buffer
with stride 20 (records 0x266667, 12 bytes past the allocation) on every draw.
With this change the buffer is bound in place from its import.
AI-assisted (Claude Code).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comments are reserved for technical debt (docs/dev/TechnicalDebt.md), so
the ones the previous commit added to the recorder, the snapshot keep and
the tests go.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HZHkb8kUwnHAuBBbHywvJf
A ShaderResources from the resource caches is the same object for every
draw or dispatch that uses it, and Recorder::ReusableDrawSnapshot hands
the next draw the same snapshot buffer, so passing their bytes with each
Keep counted one allocation once per use and submitted the batch before
it held the budget. Recorder::KeepBytes counts an owner's bytes once per
batch, and the dispatch, draw and snapshot keeps use it.
The recorded kept bytes test runs a dispatch and a draw that copy a
guest buffer and two draws that snapshot one on the device, and fails
when any of the three sites stops counting or a reused snapshot counts
twice. The BdaResources check pins the copied region it sets up.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HZHkb8kUwnHAuBBbHywvJf
The recorder bounds the bytes a batch keeps alive (KeptBytesBudget) and
waits for older batches past twice that. A recorded dispatch or draw kept
its ShaderResources, and a draw its snapshot buffers, with a size of 0, so
the host-visible copies of guest buffers they own never counted. A title
that binds a 124 MiB writable guest buffer in every draw then held about
ninety copies, 11 GB, until the GPU caught up, and aborted with
VK_ERROR_OUT_OF_HOST_MEMORY from vkAllocateMemory.
GuestBufferMemory reports the bytes of its copied regions, and the
dispatch and draw keeps and the draw snapshots pass their bytes to
Recorder::Keep, so the existing bound applies to them.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The registry calls an observer on the mapping thread, after the registry
lock is released, with the ranges a kernel mapping gave GPU access:
direct and flexible maps with it, and fixed remaps with it over a
registered range (a reservation). The mutation looks them up while it
still holds the lock and hands them over weakly, with the registry
generation it ended at, so the observer does not keep a range alive.
The driver imports those ranges on its current device, so an
address-based build finds the import instead of pinning the memory on
the queue that first uses it. A range that gains GPU access through
sceKernelMprotect, and AnyPS5's own libc heap, are still imported on
first use.
The import at map does not reconcile the other imports with the
registry, so its cost does not grow with the number of imports. It skips
a range that is gone, and any range when the imports were reconciled
with a newer registry since; the next lookup imports those as before.
It installs the pin waiter as the other import paths do, so unmapping a
range imported at map retires its import. The live import bytes are kept as a
running total instead of being summed over every import. Once the import
table belongs to the current device the import takes only the table's
lock; the first import on a device (which runs the write-watch probe)
and a device that was just replaced go through the GPU lock. Clearing
the observer waits for calls already running.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The mesh stage was compiled at the driver's default subgroup width, while the translated mesh program lays wave64 lanes out on, and ballots over, a 32-lane host subgroup. Intel's Windows driver (Arrow Lake, 32.0.101.8991) reports subgroupSize 32 but prefers 16 for mesh shaders, and every agc_driver_mesh draw drew no triangle there. The mesh stage now gets VkPipelineShaderStageRequiredSubgroupSizeCreateInfo{32} when the program's hostSubgroupSize is 32 and the device lists the mesh stage in requiredSubgroupSizeStages, as compute pipelines already do.
An upload of a storage image up to 1 MiB reads the surface's guest bytes
into a baseline (#1599), which a later write-back compares against for
blocks the CPU stamped. When a unit shadow, another image's pending
results or a recorded GPU store covers the surface, that read goes
through the flush hook: the shadow is published and the worker waits for
the GPU, although the upload itself runs on the GPU from the import.
The baseline is now taken only when nothing of that kind overlaps the
surface. Otherwise the image has none, and its write-back keeps the CPU's
bytes for stamped blocks, the rule before #1599. TechnicalDebt's entry
names the condition.
agc_storage_baseline_settled creates a storage image over a watched host
import with a recorded store pending over part of it and checks that the
upload did not submit the open batch; without the change the baseline
read syncs on that store. agc_unchanged_cpu_stamps (#1599) still passes.
PPSA14632 (LEGO Horizon Adventures) refreshes such images about 870
times per 10 s from queue 0 dispatches: the hook's recorded-store syncs
go from about 3400 waiting 4.0 s to about 630 waiting 0.3 s per 10 s,
and a direct dispatch's device call from 356 to 107 us.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A build's sampled image elements often repeat the element just before them
in plan order (the same eight descriptor words). Each still took its own
stage-A record (decode, collect, DCC key read, cache entry) and its own
stage-B lookup.
The lookups run in plan order and nothing runs between two consecutive ones,
so a repeated element's lookup would find the state the previous lookup just
left: the entry it validated or made, current against the same collect. Such
an element now binds the previous element's texture (when the depth-compare
flag matches too, which is part of the cache key), and stage A gives it a
record of the decoded descriptor only. An element repeating an earlier but
not the immediately preceding one still takes its own lookup, as does any
element after a storage image lookup.
A resource build decodes every image element's eight descriptor words,
describes the surface (a mip layout with a vector of levels) and finds the
sampled texture's cache entry by hashing its key, once in stage A and again
in stage B. Most elements repeat a small set of descriptors from build to
build.
Each worker now keeps a small direct-mapped memo by descriptor words of the
decoded resource, its guest size and view component mapping (pure functions
of the words) and of the cache entry the words last found. The entry is
reused only while the sampled texture cache has erased no entry since it
was found (an erase counter bumped by eraseTexture), so the list iterator is
still valid and still holds the same key; anything else takes the hash
lookup.
Buffer and DeviceBuffer created their VkBuffer at the pool's size class, so
a robust read past the requested size stayed in bounds and returned the
bytes an earlier user of the pooled memory left there instead of an
out-of-range result. Each VkBuffer now has the requested size, created with
the pool's usage, over memory of its class (or its slab slot); one taken
from the pool for a different size is recreated over the same memory at
the same offset. agc_buffer_pool follows a pooled buffer by its memory, so
its class and reuse cases hold with the exact-size buffers.
Every draw into a resident target refreshes the target's storage image
(StorageTexture::Refresh): a pending-registry scan for an alias,
FlushPending over the surface, the collect, the key proof, a
whole-surface UnchangedSince, four vectors and a generation update over
every 64 KiB unit, nearly always finding the image current.
A refresh that finds the image current now keeps a proof: the image's
generation and uploaded keys, its key proof's generation and the pending
registry serial. The next refresh is answered from it when the image's
own state is still that, the surface and its key range are unstamped
since those generations (CollectWrites over both, then one
UnchangedSinceAll), and no other image has results pending over the
surface: the registry serial is unchanged, or the registry is scanned and
holds none (a newly pending image, a flush in progress). The proof is
taken only under GuestMemory::GpuMutex, without a pending alias, with
stable keys and when no other pending image overlaps the surface; any
full refresh drops it. A CPU write, a driver or GPU store, a key store,
another image's pending results or a different image therefore always
refresh in full, and the image's own pending results (a draw into it)
keep the proof.
agc_driver_recorder_tests refreshProofTests: an unchanged image and one
holding its own pending results are proved; a CPU write, a driver store
into the surface, a key store, another image's pending results, a new
image of the surface and a fast clear of the keys each refresh in full.
A geometry (NGG) strip draw on the mesh path assigned every primitive the next index positions without looking for the restart index: a triangle spanning a restart fetched vertex 0xFFFF and was drawn to wherever that read landed, and the strip parity never reset, so after an odd-length strip every triangle's winding flipped and face culling dropped it.
For an indexed strip draw with restart enabled and the restart index present, the draw now records each position's strip start (one past the last restart at or before it) and passes the table after the mesh arguments, flagged in draw word 3. The mesh prologue reads the start of its primitive's last vertex: a primitive whose start lies past its first vertex spans a restart and is made degenerate, and the others take their parity from their strip's start (or the draw's first index). The restart vertex itself fetches vertex 0. The table is built from the index bytes the draw already reads (its copy, or the guest memory an in-place import reads), next to #1968's restart-aware highest-index scan.
The new Mesh.cpp test draws quads as four-index strips each followed by a restart (16- and 32-bit indices, and an indirect draw starting mid-buffer) with front faces culled: every quad must render whole and the gaps between them stay clear. Without the table the first check fails ("a triangle crossed the restart after quad 0").
AI-assisted (Claude Code).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#2096 keeps a copy of a storage image's guest bytes so that Refresh can
compare memory the write watch does not cover. The host-import upload
path takes that copy only when the range is unwatched, but the DCC
fast-clear path took it on every upload. Reading the whole surface goes
through the flush hook, so each 64 KiB block of a surface that a unit
shadow covers recorded a publish and waited for the GPU.
The clear path now resolves the surface's host import first (the import
can unwatch the range, as a compared host-pointer import does) and takes
the copy only when the range is then unwatched, like the import path.
Write-watched memory, such as direct memory behind a dma-buf import,
skips it; a range that is unwatched later re-uploads instead of
comparing, as before #2096.
agc_driver_dcc_keys_first_write checks that a fast-cleared upload keeps
the copy exactly when its range is unwatched at that point.
PPSA14632 (LEGO Horizon Adventures) on Linux, RX 9070 XT: the flush hook
waited ~60,000 times per 10 s for these publishes (6.4 s), now ~3,500,
and presents rise from ~24 to ~100 per 10 s.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A stale storage snapshot is revalidated by comparing every tracker block
stamped since it was taken. A title that binds buffers a dispatch writes
in place every frame (the whole declared range stamped, few pages really
changed) makes the graphics worker compare gigabytes per second.
Compares of 1 MiB or more are now split by block across a small pool of
helper threads (hardware threads / 8, from 1 to 4). The compare is memory
bound, so the worker's share roughly halves rather than dividing by the
thread count. A caller that finds the pool busy compares on its own
thread. Results and the patch-in-place rules are unchanged.
agc_driver_graphics_tests parallelCompareTests: per-span results on the
pool and serial paths, and two threads comparing at once.
A stale storage snapshot was compared whole with the guest bytes and, on
any difference, copied whole into a new buffer, although few of the
tracker blocks stamped since the copy usually hold different bytes.
Only the 64 KiB tracker blocks stamped since the snapshot's generation
are compared now. When some differ and the cache holds the only
reference (no recorded draw binds the buffer), those blocks are copied
into it in place; otherwise the range is copied again as before.
agc_driver_recorder_tests drawSnapshotPatchTests: changed bytes in
several blocks and a whole changed block patch the snapshot nothing else
holds, an unchanged one is reused, and one a recorded draw still holds is
copied again with its bytes untouched.
A written V# element bound in place in a host import makes every load a
PCIe round trip. A particle update that loads and stores a 10 MiB pool
(2048x64 threads over 131072 records of 80 bytes), above the 2 MiB written
window, took about 4.8 ms of GPU per frame.
An element the shader also reads (DescriptorBinding::bufferRead), written by
a direct dispatch whose threads times the V# stride cover at least half of
it, now takes device-local staging up to 16 MiB: that dispatch drops to
about 0.06 ms plus 0.9 ms of bulk copies. The coverage test keeps a kernel
that updates about 12k records of a 16 MiB pool in place, where staging all
of it would cost 1.3 ms of copies for a 0.08 ms kernel. Write-only
elements, indirect dispatches and raw (stride 0) buffers keep the 2 MiB
window.
A storage write-back copied the image into a linear buffer and retiled
that buffer; an upload detiled into a linear buffer and copied it into
the image. With storage image access without a format, the retile now
reads each texel from the image through an unsigned integer view of the
element size, and the detile writes it there, so the linear buffer, its
copy and one barrier per move go away. The texels move bit for bit as
before, through the same tiling equations and windows.
This covers the block-unit windows of write-backs and uploads and the
whole-surface direct upload, for 2D, array and cube images.
agc_driver_recorder_tests singlePassTests: a first upload, a write-back
of new texels in every mip and a second upload over the stored memory
leave the images and guest memory byte for byte equal to the copies
through a linear buffer, in plain and write-watched memory.
A recorded draw continued the open render pass only after a draw that
wrote no memory, so every draw with a storage write ended the pass and
paid two barriers. A draw now continues the open pass when its accesses
have no hazard (PassHazards) against those of the draws since the last
barrier. Address-based draws, draws through buffer device addresses and
draws with copied writes count as reading and writing any guest memory.
A draw of other attachments with no hazard ends the pass without a
barrier and begins its own without one, and a hazard break records the
begin barrier alone, towards every graphics stage, instead of the pass
end barrier plus the begin barrier. A pass with a depth attachment or
attachment proxies, or one followed by work after the pass, keeps both
barriers. APS5_FULL_BARRIERS keeps every barrier, as for the other merges.
PassHazards keeps the guest ranges, images and attachments the draws
since the last barrier read and wrote, and answers whether a new draw has
a RAW, WAW or WAR hazard against them. Address-based reads and unknown
writes count as touching any guest memory; attachments of the current
pass are ordered by rasterization and only conflict with other passes or
with sampled and storage uses. Unit tests in agc_pass_hazards.
A draw's storage snapshot was copied again whenever the write tracker
reported a store over its range, although most such stores leave the
bytes as they were: in one title screen about 10k of 14k stale lookups per
10 s held the same bytes (3.5 GiB of copies).
A stale storage snapshot is now compared with the guest bytes, after the
caller's collect of the range, before it is dropped. When they are equal
it takes that collect's generation and is reused; any store after the
collect is stamped newer and fails the next check, as for a fresh copy. A
registry change still drops it without reading, and vertex and index
snapshots are dropped as before.
materializeRegisterClear read the target's DCC keys with CurrentDccKeys on
every draw into a resident target with metadata: a full AllKeysEqual scan
of the key range (52 KiB for a 2432x1368 RGBA8 target), about 440k scans
per 10 s and 7.2% of the queue 0 worker's CPU time in one level.
ProvedCurrentDccKeys answers CurrentDccKeys from a proof kept with the
resident image (DccRangeProof: the key range, the keys and the write
generation collected before the scan), the rule ProvedClearKeys already
applies to a storage image's own keys: the proof holds while the key range
is unstamped since that generation (CollectWrites first, then
UnchangedSince), and a scan becomes the proof only when it read the bytes
(not the pending store memo) and no recorded work still writes them. Every
key writer stamps the range, so a store, a pending result or another key
range always scans again. The proof is used under GuestMemory::GpuMutex
only, and not with APS5_NO_KEY_FAST_PATH.
agc_driver_recorder_tests: an unchanged range is proved, and a key store
stamp, a CPU write, another range or address, and a pending key fill each
force a scan; the fill's result is proved once its batch finished.
Adrià
Chad
Scott Schuster
MarcZX100
MathiasDevelopes
raghav314
Joey Fisher
Abdulrahman Elkordy
Dean
krishnaraya
alfalfa-star
Dean Galvin
TideOPL
Adrià Franch