3D textures in SW_4KB_S or SW_64KB_S whose smaller levels reach the mip
tail were refused ("3D texture mip tails are not implemented"). The
tail now follows addrlib's GFX10 thick layout
(Gfx10Lib::ComputeSurfaceInfoMacroTiled):
- a level enters the tail once its size, rounded up, fits the thick
tail extent (the block with its height halved for 4 KiB blocks and
its width halved for 64 KiB blocks) and at most GetMaxNumMipsInTail
levels remain (5 for 4 KiB, 10 for 64 KiB);
- each slab starts with one tail block, then the other levels from the
smallest to level 0;
- tail level i sits in slot maxMipsInTail - 1 - i, at the element offset
addrlib derives from that slot's byte offset and the 256-byte thick
micro block, at every depth slice.
The tests check element addresses of five volumes (4 KiB and 64 KiB
blocks, 1 to 16 bytes per element, odd sizes, a chain that starts in
the tail) against Mesa addrlib's Addr2ComputeSurfaceAddrFromCoord for
a Navi10 configuration.
PPSA21564 clears each occlusion query slot with a 256-byte WRITE_DATA
before its begin dump, about 8,400 times per frame in one level. The
clear is larger than a deferred label, so it is recorded at once, and
Recorder::NoteLabel entered all 64 of its dwords in the pending-label
table: 64 map inserts, 64 batch dword entries and 64 group entries, then
64 lookups in CloseLabelGroup and 64 erases when the batch is reaped. A
replay of that pattern (21 clears per batch, 20 batches in flight) costs
1.6 us for the inserts, 0.7 us for the close and 1.3 us for the erases
per clear, about 31 ms of the graphics worker per frame there. The new
path costs 0.07 us per clear in the same replay.
The table serves WAIT_REG_MEM values and CPU pollers, which read 4 or 8
bytes. A store larger than 64 bytes (the bound the driver's own label
history already uses) now enters one range instead of its dwords:
- table entries inside the range are removed, as the store replaced
them before: a lookup there finds no value and falls back to the
pending-write path, where it used to find the store's value;
- the range leaves with its batch, like the dword entries;
- PendingLabelIn sees the range, and the write-evidence check, which
syncs a capture read over any pending label dword, now also syncs
over a pending range (Recorder::WideLabelIn).
The store is still a pending write of the open batch (the flush hook
syncs CPU readers), still starts the label flush deadline and is
recorded exactly as before. Stores of 64 bytes or less are unchanged.
Test: agc_driver_recorder_tests largeLabelTests (a 256-byte label enters
no dwords and removes the older entry under it but not one beside it,
its range, pending write and deadline, a later 4-byte label over it,
a 64-byte label entered in full, everything gone after the batch).
Every occlusion counter dump the recorder handles on the GPU ends the
batch's occlusion query, copies the pending results, folds them with a
compute dispatch, then resets and begins a new query. Each of those
commands was resolved through vkGetDeviceProcAddr, a loader name lookup,
on every call: seven lookups per dump on the graphics worker, and a title
dumps the counter twice per occlusion query. The mesh argument and timing
paths resolved their bind, push, dispatch and reset per call too.
They join the entry points the recorder already resolves in its
constructor: vkCmdBeginQuery, vkCmdEndQuery, vkCmdResetQueryPool,
vkCmdCopyQueryPoolResults, vkCmdBindPipeline, vkCmdPushConstants and
vkCmdDispatch. The same functions are called with the same arguments;
APS5_NO_PROC_TABLE=1 still resolves every call.
A GPU-direct storage write-back retiles the image's elements into an
uninitialised device scratch buffer and then copies whole ranges of it
into the import or the import's unit shadow. The retile writes only the
bytes elements occupy, so every other byte of the copied ranges was
stored from uninitialised scratch memory: the padding of the tile blocks
a surface covers only partly (its last block column and row), mip tail
blocks, the bytes between mips and the pitch padding of linear surfaces.
Another surface over the same memory, or the CPU, then read garbage
there instead of what the memory held. Both the block-window path and
the whole-layer path did this.
Before the retile, the bytes of the copied ranges that no element holds
are now copied into the scratch from where they are current (a fresh unit
shadow, else the import), so the write-back stores them unchanged. Whole
tile blocks every byte of which holds an element are left out
(CoveredMipBytes), so only partly covered blocks pay for the extra copy.
Tests: CoveredMipBytes against the XOR address equations of the four
64 KiB XOR modes at every element size; a device test that stores
cleared images over patterned memory (tiled with and without mip chains,
off a 64 KiB boundary, 4 KiB standard, linear; plain and watched memory)
and checks every byte that is not a texel kept its pattern.
Pixel shader color exports are compacted: export MRTi goes to the i-th
MRT slot with a non-zero CB_SHADER_MASK nibble and takes the i-th
SPI_SHADER_COL_FORMAT field. Draws still mapped export i to slot i and
read slot i's format field, so a shader with CB_SHADER_MASK 0xf000f and
SPI_SHADER_COL_FORMAT 0x44 (export 1 to slot 4) was refused with
"color export format 0".
Each color target now carries its export index next to its slot.
Attachment references, blend states, export mappings and the pipeline
and pass keys follow the export index; the color buffer, blend control
and target mask come from the slot. An unwritten export below the
highest written one stays an unused attachment.
flat_store_* and global_store_* decoded and translated, but a store whose
segment is not scratch was refused when the SPIR-V was emitted ("writable
FLAT/GLOBAL addresses require GPU ownership tracking").
Such a store now goes through the BDA page table like a store through a
GPU-selected V#: the write lookup needs Write permission, which only a
writable registered range imported in place has, and the written page is
noted for the host to mark GPU-written. Byte and short stores, and dword
stores to an address that is not dword aligned, are stored byte by byte.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On MinGW, thread_local storage is emutls: libgcc frees a thread's blocks
from a winpthread key destructor when the thread function returns, and
the CRT runs the thread_local destructors later, from the module's TLS
callback at thread detach. The destructors of the driver's non-trivial
thread_locals (the deferred label queue, packet profile, capture and
revalidation scratch vectors, build profile, memory counters, disk cache
buffers) therefore ran on freed storage. Under the debug heap the flip
test failed on every run at the queue worker's exit after the injected
flip failure; without it the use-after-free showed up as intermittent
heap corruption or a stack overflow on the Windows CI job.
ShaderRecompiler::ThreadOwned keeps such state behind a trivially
destructible thread_local pointer and deletes it from its own pthread key
destructor, which receives the objects directly instead of reading
emutls. The driver no longer registers any thread_local destructor.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Color targets with the SW_4KB_S swizzle (CB_COLOR_ATTRIB3 tile mode 5) are laid out in 4 KiB
blocks with the standard element order the texture detiler uses, and are resident through the
same storage images as 64 KiB targets. A CB_COLOR_VIEW selecting one slice (SLICE_START equal to
SLICE_MAX) renders into that slice: the target starts one slice size per slice after the surface,
the slice size being the surface's mip chain as a texture of the same tile mode lays it out. Views
of several slices are still rejected.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
addrlib sizes a non-tail level with ShiftCeil; the layout used the image's
rounded-down extent, so volumes with a non-power-of-two width or height (33x20x12
in the new test) got a different layout.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
3D textures with more than one mip level were refused. Each slab of
blockDepth slices of a thick (SW_4KB_S / SW_64KB_S) volume now holds the
whole mip chain, smallest level first and level 0 last, as 2D chains are
laid out. A level's depth halves per level, so the texture loops skip the
slices a level does not have. Thin 3D (D/R 64KB_X) chains use the 2D mip
layout per depth slice. Thick mip tails still throw.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Decode and translate v_add/mul/fma/min/max/ldexp_f64, v_trunc/ceil/floor/
rndne/fract_f64, v_frexp_mant/exp_i32_f64 and the f64 <-> f32/i32/u32
conversions. f64 values stay as their raw bits (U64) in the IR and are
bitcast to double in SPIR-V, which needs the Float64 capability: the
device enables shaderFloat64 when it also preserves signed zero, inf and
NaN for 64-bit floats, and the shader declares SignedZeroInfNanPreserve 64.
Semantics measured on RDNA2 hardware (8209 inputs per op): literals are
the high dword, the 1/(2*pi) inline constant is the exact double,
omod is ignored, clamp maps NaN to +0, min/max quiet a
signaling NaN and drop a quiet one, NaN results are quiet, and the
integer conversions saturate with NaN -> 0. Truncation, rounding and
ldexp are done on the bits so they do not depend on the driver: lavapipe
returned wrong results for f64 RoundEven and Trunc, and GLSL Ldexp is
undefined past +-1024. Fma is NoContraction so RADV keeps it fused.
Co-Authored-By: Claude <noreply@anthropic.com>
The descriptors of one surface share its storage image through the
surface key, which includes the level count. A view past MAX_MIP
counts the extra levels, so its storage image was a second image of
the same memory, and the two flushed and re-uploaded each other every
frame (PPSA21564 writes level 6 of each 1920x1080 bloom chain after
building levels 1 to 5 through MAX_MIP 5 views).
A surface once viewed past its allocated levels now keeps a single
image holding the extended chain, and its other descriptors look that
image up; the allocated-level image it replaces is evicted with its
results stored. The extra levels lie in the allocation (the decode
proved that), so levels 0 to MAX_MIP keep their layout and the views
of them read the same texels.
PPSA21564 ends each of its two 1920x1080 RGBA16F bloom downsample
chains (MAX_MIP 5) with a compute pass that stores level 6 through a
view of levels 6..6, every frame from the PS logo on. The descriptor
decode threw for such a view.
GFX10 uses MAX_MIP only to place the levels: a level's address comes
from the levels below it, and a level in the mip tail is a fixed slot of
the tail block counted from the first tail level. A level past MAX_MIP
is addressed the same way, so when the chain extended to the view's last
level keeps every allocated level, the tail and the surface size
unchanged, the view reaches that slot inside the surface's allocation.
addrlib (GFX10, Navi10 with 16 pipes) computes the same 0x1640000-byte
layout for 6, 7, 9 and 11 levels of this surface, with level 6 in the
tail at (0, 32). The view then decodes with the extended chain over the
same guest bytes. Views whose extra levels would move the allocated
ones, such as linear chains or chains without a tail, still throw.
The first draw into a target whose DCC keys hold the register clear code
materializes the clear. The GPU clear of the resident image refuses while a
recorded write to the keys is still pending, so whenever the title's own key
fill had not run yet the clear went the CPU way: the clear texels stored
into guest memory (16 MiB for PPSA21564's normal G-buffer) and the image
uploaded again. With the occlusion dumps no longer draining the device, the
fill is usually still pending at that draw, and that was every frame of
PPSA21564's level.
The draw now waits for the recorded writes of the key range alone, checks
the keys again, and clears on the GPU when they still hold the register
code, as it did when the fill had already landed.
Every ZPASS_DONE counter dump drained the device so the CPU could add up
the occlusion queries of the finished batches and store the running total.
PPSA21564 dumps around each occlusion-tested pass: about 1.5k drains per
10 s at its title screen, and in a level's early passes four drains a frame
that took 5.6 s of every 10 s of the graphics worker.
The running total now lives in a small buffer the GPU updates in batch
order. A batch still ends its occlusion query at submit; a query that saw
draws is remembered, and its pool is kept alive until it is folded. A dump
ends the open batch's query, copies the remembered results next to the
total, adds them with a one-thread compute shader that also stores the 16
DB counters into the dump's host import, and begins a fresh query for the
rest of the batch. The dump is ordered with the work around it like any
other GPU store, noted as a pending write and stamped. More than 64
remembered results are folded at a submit. Query pools are recycled.
A dump into memory the GPU cannot see, under a pending copied write-back
or over a label the CPU stores at a completion falls back to the drain,
which settles the remembered results on the CPU. Without buffer device
addresses the counting stays on the CPU.
A mesh program took its draw's index count from push constants, so an
indirect mesh-path (NGG) draw needed the CPU to read its record first,
waiting for the GPU to finish everything recorded before. In PPSA21564
that was about 170 waits per 10 s at the title, menu and level (about
1 s), and one per frame in the intro (2.2 s per 10 s).
The two free mesh push dwords now carry the device address of an argument
record (a VkDrawMeshTasksIndirectCommandEXT followed by the index count
and first index). When the address is set, the prologue takes the index
count and the first index from the record (the new MeshArgument IR value,
a load through a physical storage buffer pointer) and offsets its index
reads by the first index; a direct draw leaves the address 0 and draws as
before. Mesh modules declare the physical storage buffer capability for
it, and the driver's SPIR-V validation accepts the extension in mesh
modules.
An indexed mesh draw with one record that patches no user SGPR now stays
on the GPU. A one-thread compute pass (Graphics/shaders/MeshArguments.comp)
turns the record, read in place from its host import, into the draw's
argument record: the groups and instances of vkCmdDrawMeshTasksIndirectEXT
and the index count and first index the mesh program reads, with the CPU
path's rules (count clamped to the index buffer, nothing for an empty
record or past the device's group limits). The index buffer descriptor
covers the whole buffer and the program offsets by the first index. When
the records cannot be read in place, the CPU reads them and fills the
same argument record (ResolveMeshArguments).
A store the driver makes with the CPU into watched guest memory (a
GuestMemory::Write, a changed-bytes write-back, a completion action's label
store, the copy HLE's CPU path) faults its pages dirty in the write watch and
stamps the range with MarkWritten. The faults stayed behind: collects are
memoized per packet, so the next walk over those pages came later and stamped
them again, as a CPU write made after every check in between.
PPSA21564 fast-clears its normal G-buffer every frame with a register DCC key
fill on the GPU. While that fill is still pending, the first draw into the
target materializes the clear with a CPU store of the clear texels and a
refresh of the resident image. A dozen draws later a walk found the store's
faults, the target read as changed, and the image was uploaded again from
guest memory, dropping the draws recorded in between. Lighting then sampled
cleared normals and the level's moon and rocks went unlit in about half the
frames.
The driver's stores now consume those faults themselves: a walk before the
store stamps earlier CPU writes as such, a walk after it takes the store's
own faults as a driver store, and the range is stamped as before.
On Windows the arena's range also holds host regions that were already in
use when it was reserved (heaps, thread stacks). They are not write-watched,
so a store into one is stamped as before, without a walk.
PPSA21564's galaxy map renders a 1920x1080 surface as a color target whose
CB names DCC keys at one address and then writes it from a compute shader
whose T# names keys at another. Both key ranges read uncompressed every
frame, but the storage cache remade the surface's image whenever a
descriptor named metadata other than the image's, so the image was
flushed, destroyed, created and uploaded twice a frame (1231 constructions
in a 56 s scene), with the views and draw recipes built on it thrown away
each time.
A descriptor naming other keys now keeps the image when the move is
unobservable: the image's own keys and the named keys both read
uncompressed, and its content was neither uploaded nor filled under a
clear code. That is exactly the content an image made under the named
keys would hold once this one's pending results were stored. Any other
case (a fast clear of either key range, a register clear, an image
holding a clear) still remakes the image as before, so clears of the new
keys reach it and pending results are stored first.
Every proof that stands in for a lookup checks the same rule: the sampled
view hit, the fast revalidation of views and storage elements (storage
elements now record their descriptor's keys), the own-object refresh, and
the draw recipe's resident targets. Named-key scans keep per-image proofs,
so a repeated check costs no scan while the key bytes are unstamped.
agc_dcc_key_sharing drives KeysServeSurface, the rule ServesKeysAt uses,
through a model of the storage cache.
PA_SC_WINDOW_OFFSET is already required to be zero, so a generic,
window or viewport scissor whose WINDOW_OFFSET_DISABLE bit is clear
selects the same rectangle as one that sets it. Only the reserved
bits are still rejected.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018m6Ad8zkQKPKZmyUQK9vfE
A CB metadata pass (and a resident target's register-clear check) read the
DCC keys with CurrentDccKeys, which waited for the GPU whenever recorded
work wrote the key range. In the intro video of PPSA21564 about 800 passes
per 10 s waited about 1.8 ms each, and every one of those waits was behind
a key store whose result the driver already knew: its own uncompressed key
store, or a title fill of uniform keys recorded on the GPU.
Pending-write notes now carry a serial in recording order, and the key
memo remembers the note of the store it describes. When the newest
pending note over the keys is that store's own note, nothing recorded
after it writes the keys, so they read as the store's keys without a
wait. Any later writer, a range the store does not cover, or a unit
shadow over the keys waits as before.
Dean
Dean Galvin
DotDebian
Nicolas Abosch
Chad
Pacuka
Michon
SP4C3B4R-8
Adrià Franch