The resource materializer refused every atomic image descriptor whose
format was not 32_UINT. RDNA2 image atomics act on the raw dword and
the numeric format only affects addressing, so 32_SINT and 32_FLOAT
surfaces are now accepted with the Uint numeric class and run through
the existing integer atomics and float compare-exchange loops. Other
formats still throw, and the message now names the format.
DescriptorBinding gains imageAtomic, carried through the shader disk
cache (format version 4) and the resource content key. The driver
binds those elements through a cached R32_UINT view of R32_SINT and
R32_SFLOAT storage images.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
3D textures in SW_4KB_S or SW_64KB_S whose smaller levels reach the mip
tail were refused ("3D texture mip tails are not implemented"). The
tail now follows addrlib's GFX10 thick layout
(Gfx10Lib::ComputeSurfaceInfoMacroTiled):
- a level enters the tail once its size, rounded up, fits the thick
tail extent (the block with its height halved for 4 KiB blocks and
its width halved for 64 KiB blocks) and at most GetMaxNumMipsInTail
levels remain (5 for 4 KiB, 10 for 64 KiB);
- each slab starts with one tail block, then the other levels from the
smallest to level 0;
- tail level i sits in slot maxMipsInTail - 1 - i, at the element offset
addrlib derives from that slot's byte offset and the 256-byte thick
micro block, at every depth slice.
The tests check element addresses of five volumes (4 KiB and 64 KiB
blocks, 1 to 16 bytes per element, odd sizes, a chain that starts in
the tail) against Mesa addrlib's Addr2ComputeSurfaceAddrFromCoord for
a Navi10 configuration.
PPSA21564 clears each occlusion query slot with a 256-byte WRITE_DATA
before its begin dump, about 8,400 times per frame in one level. The
clear is larger than a deferred label, so it is recorded at once, and
Recorder::NoteLabel entered all 64 of its dwords in the pending-label
table: 64 map inserts, 64 batch dword entries and 64 group entries, then
64 lookups in CloseLabelGroup and 64 erases when the batch is reaped. A
replay of that pattern (21 clears per batch, 20 batches in flight) costs
1.6 us for the inserts, 0.7 us for the close and 1.3 us for the erases
per clear, about 31 ms of the graphics worker per frame there. The new
path costs 0.07 us per clear in the same replay.
The table serves WAIT_REG_MEM values and CPU pollers, which read 4 or 8
bytes. A store larger than 64 bytes (the bound the driver's own label
history already uses) now enters one range instead of its dwords:
- table entries inside the range are removed, as the store replaced
them before: a lookup there finds no value and falls back to the
pending-write path, where it used to find the store's value;
- the range leaves with its batch, like the dword entries;
- PendingLabelIn sees the range, and the write-evidence check, which
syncs a capture read over any pending label dword, now also syncs
over a pending range (Recorder::WideLabelIn).
The store is still a pending write of the open batch (the flush hook
syncs CPU readers), still starts the label flush deadline and is
recorded exactly as before. Stores of 64 bytes or less are unchanged.
Test: agc_driver_recorder_tests largeLabelTests (a 256-byte label enters
no dwords and removes the older entry under it but not one beside it,
its range, pending write and deadline, a later 4-byte label over it,
a 64-byte label entered in full, everything gone after the batch).
Every occlusion counter dump the recorder handles on the GPU ends the
batch's occlusion query, copies the pending results, folds them with a
compute dispatch, then resets and begins a new query. Each of those
commands was resolved through vkGetDeviceProcAddr, a loader name lookup,
on every call: seven lookups per dump on the graphics worker, and a title
dumps the counter twice per occlusion query. The mesh argument and timing
paths resolved their bind, push, dispatch and reset per call too.
They join the entry points the recorder already resolves in its
constructor: vkCmdBeginQuery, vkCmdEndQuery, vkCmdResetQueryPool,
vkCmdCopyQueryPoolResults, vkCmdBindPipeline, vkCmdPushConstants and
vkCmdDispatch. The same functions are called with the same arguments;
APS5_NO_PROC_TABLE=1 still resolves every call.
A GPU-direct storage write-back retiles the image's elements into an
uninitialised device scratch buffer and then copies whole ranges of it
into the import or the import's unit shadow. The retile writes only the
bytes elements occupy, so every other byte of the copied ranges was
stored from uninitialised scratch memory: the padding of the tile blocks
a surface covers only partly (its last block column and row), mip tail
blocks, the bytes between mips and the pitch padding of linear surfaces.
Another surface over the same memory, or the CPU, then read garbage
there instead of what the memory held. Both the block-window path and
the whole-layer path did this.
Before the retile, the bytes of the copied ranges that no element holds
are now copied into the scratch from where they are current (a fresh unit
shadow, else the import), so the write-back stores them unchanged. Whole
tile blocks every byte of which holds an element are left out
(CoveredMipBytes), so only partly covered blocks pay for the extra copy.
Tests: CoveredMipBytes against the XOR address equations of the four
64 KiB XOR modes at every element size; a device test that stores
cleared images over patterned memory (tiled with and without mip chains,
off a 64 KiB boundary, 4 KiB standard, linear; plain and watched memory)
and checks every byte that is not a texel kept its pattern.
Pixel shader color exports are compacted: export MRTi goes to the i-th
MRT slot with a non-zero CB_SHADER_MASK nibble and takes the i-th
SPI_SHADER_COL_FORMAT field. Draws still mapped export i to slot i and
read slot i's format field, so a shader with CB_SHADER_MASK 0xf000f and
SPI_SHADER_COL_FORMAT 0x44 (export 1 to slot 4) was refused with
"color export format 0".
Each color target now carries its export index next to its slot.
Attachment references, blend states, export mappings and the pipeline
and pass keys follow the export index; the color buffer, blend control
and target mask come from the slot. An unwritten export below the
highest written one stays an unused attachment.
Color targets with the SW_4KB_S swizzle (CB_COLOR_ATTRIB3 tile mode 5) are laid out in 4 KiB
blocks with the standard element order the texture detiler uses, and are resident through the
same storage images as 64 KiB targets. A CB_COLOR_VIEW selecting one slice (SLICE_START equal to
SLICE_MAX) renders into that slice: the target starts one slice size per slice after the surface,
the slice size being the surface's mip chain as a texture of the same tile mode lays it out. Views
of several slices are still rejected.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
3D textures with more than one mip level were refused. Each slab of
blockDepth slices of a thick (SW_4KB_S / SW_64KB_S) volume now holds the
whole mip chain, smallest level first and level 0 last, as 2D chains are
laid out. A level's depth halves per level, so the texture loops skip the
slices a level does not have. Thin 3D (D/R 64KB_X) chains use the 2D mip
layout per depth slice. Thick mip tails still throw.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The descriptors of one surface share its storage image through the
surface key, which includes the level count. A view past MAX_MIP
counts the extra levels, so its storage image was a second image of
the same memory, and the two flushed and re-uploaded each other every
frame (PPSA21564 writes level 6 of each 1920x1080 bloom chain after
building levels 1 to 5 through MAX_MIP 5 views).
A surface once viewed past its allocated levels now keeps a single
image holding the extended chain, and its other descriptors look that
image up; the allocated-level image it replaces is evicted with its
results stored. The extra levels lie in the allocation (the decode
proved that), so levels 0 to MAX_MIP keep their layout and the views
of them read the same texels.
PPSA21564 ends each of its two 1920x1080 RGBA16F bloom downsample
chains (MAX_MIP 5) with a compute pass that stores level 6 through a
view of levels 6..6, every frame from the PS logo on. The descriptor
decode threw for such a view.
GFX10 uses MAX_MIP only to place the levels: a level's address comes
from the levels below it, and a level in the mip tail is a fixed slot of
the tail block counted from the first tail level. A level past MAX_MIP
is addressed the same way, so when the chain extended to the view's last
level keeps every allocated level, the tail and the surface size
unchanged, the view reaches that slot inside the surface's allocation.
addrlib (GFX10, Navi10 with 16 pipes) computes the same 0x1640000-byte
layout for 6, 7, 9 and 11 levels of this surface, with level 6 in the
tail at (0, 32). The view then decodes with the extended chain over the
same guest bytes. Views whose extra levels would move the allocated
ones, such as linear chains or chains without a tail, still throw.
Every ZPASS_DONE counter dump drained the device so the CPU could add up
the occlusion queries of the finished batches and store the running total.
PPSA21564 dumps around each occlusion-tested pass: about 1.5k drains per
10 s at its title screen, and in a level's early passes four drains a frame
that took 5.6 s of every 10 s of the graphics worker.
The running total now lives in a small buffer the GPU updates in batch
order. A batch still ends its occlusion query at submit; a query that saw
draws is remembered, and its pool is kept alive until it is folded. A dump
ends the open batch's query, copies the remembered results next to the
total, adds them with a one-thread compute shader that also stores the 16
DB counters into the dump's host import, and begins a fresh query for the
rest of the batch. The dump is ordered with the work around it like any
other GPU store, noted as a pending write and stamped. More than 64
remembered results are folded at a submit. Query pools are recycled.
A dump into memory the GPU cannot see, under a pending copied write-back
or over a label the CPU stores at a completion falls back to the drain,
which settles the remembered results on the CPU. Without buffer device
addresses the counting stays on the CPU.
A mesh program took its draw's index count from push constants, so an
indirect mesh-path (NGG) draw needed the CPU to read its record first,
waiting for the GPU to finish everything recorded before. In PPSA21564
that was about 170 waits per 10 s at the title, menu and level (about
1 s), and one per frame in the intro (2.2 s per 10 s).
The two free mesh push dwords now carry the device address of an argument
record (a VkDrawMeshTasksIndirectCommandEXT followed by the index count
and first index). When the address is set, the prologue takes the index
count and the first index from the record (the new MeshArgument IR value,
a load through a physical storage buffer pointer) and offsets its index
reads by the first index; a direct draw leaves the address 0 and draws as
before. Mesh modules declare the physical storage buffer capability for
it, and the driver's SPIR-V validation accepts the extension in mesh
modules.
An indexed mesh draw with one record that patches no user SGPR now stays
on the GPU. A one-thread compute pass (Graphics/shaders/MeshArguments.comp)
turns the record, read in place from its host import, into the draw's
argument record: the groups and instances of vkCmdDrawMeshTasksIndirectEXT
and the index count and first index the mesh program reads, with the CPU
path's rules (count clamped to the index buffer, nothing for an empty
record or past the device's group limits). The index buffer descriptor
covers the whole buffer and the program offsets by the first index. When
the records cannot be read in place, the CPU reads them and fills the
same argument record (ResolveMeshArguments).
PPSA21564's galaxy map renders a 1920x1080 surface as a color target whose
CB names DCC keys at one address and then writes it from a compute shader
whose T# names keys at another. Both key ranges read uncompressed every
frame, but the storage cache remade the surface's image whenever a
descriptor named metadata other than the image's, so the image was
flushed, destroyed, created and uploaded twice a frame (1231 constructions
in a 56 s scene), with the views and draw recipes built on it thrown away
each time.
A descriptor naming other keys now keeps the image when the move is
unobservable: the image's own keys and the named keys both read
uncompressed, and its content was neither uploaded nor filled under a
clear code. That is exactly the content an image made under the named
keys would hold once this one's pending results were stored. Any other
case (a fast clear of either key range, a register clear, an image
holding a clear) still remakes the image as before, so clears of the new
keys reach it and pending results are stored first.
Every proof that stands in for a lookup checks the same rule: the sampled
view hit, the fast revalidation of views and storage elements (storage
elements now record their descriptor's keys), the own-object refresh, and
the draw recipe's resident targets. Named-key scans keep per-image proofs,
so a repeated check costs no scan while the key bytes are unstamped.
agc_dcc_key_sharing drives KeysServeSurface, the rule ServesKeysAt uses,
through a model of the storage cache.
A CB metadata pass (and a resident target's register-clear check) read the
DCC keys with CurrentDccKeys, which waited for the GPU whenever recorded
work wrote the key range. In the intro video of PPSA21564 about 800 passes
per 10 s waited about 1.8 ms each, and every one of those waits was behind
a key store whose result the driver already knew: its own uncompressed key
store, or a title fill of uniform keys recorded on the GPU.
Pending-write notes now carry a serial in recording order, and the key
memo remembers the note of the store it describes. When the newest
pending note over the keys is that store's own note, nothing recorded
after it writes the keys, so they read as the store's keys without a
wait. Any later writer, a range the store does not cover, or a unit
shadow over the keys waits as before.
DB_DEPTH_CONTROL's DEPTH_BOUNDS_ENABLE now runs the Vulkan depth bounds
test with DB_DEPTH_BOUNDS_MIN/MAX, and PA_SU_SC_MODE_CNTL's
POLY_OFFSET_FRONT/BACK_ENABLE enable depth bias from
PA_SU_POLY_OFFSET_FRONT/BACK_SCALE/OFFSET and _CLAMP. Both are dynamic
state, so draws that differ only in their values share a pipeline.
The hardware slope scale is in 1/16 units. Depth bias with units other
than the depth format's (PA_SU_POLY_OFFSET_DB_FMT_CNTL), or that differs
between front and back faces that are both drawn, is refused. The
registers join the draw key. Depth-conditional color writes are still
refused.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The [present] line, printed under APS5_PROFILE_DRAW, takes its GPU busy and
idle gaps from the whole-batch timestamps in each batch's completion record.
Those stamps were only written under APS5_PROFILE_GPU, which also times every
command. Under APS5_PROFILE_DRAW alone every batch finished with zero stamps,
so the line read "GPU busy 0.0 ms, idle gaps 0.0 ms" with no missing records.
Under APS5_PROFILE_DRAW each batch now carries just its two batch stamps in a
two-query pool, reused across batches like the command buffers.
APS5_PROFILE_GPU is unchanged, and nothing is written with profiling off.
agc_driver_recorder_tests (GPU, by hand) checks that three consecutive
batches' completion records carry ordered stamps exactly when batch stamps
are on, and none with profiling off.
The draw resource key holds each read-only buffer's V# base and size and every
ShaderData/FlattenedSrt word, so draws whose buffers move per frame (constant and
vertex rings) never hit a template. The key now leaves those out; a hit checks every
moved read-only buffer (accessible, no pending or shadowed writes over it) and
rebinds it from its new address with the template's push adjustment, and data words
that changed get a fresh data buffer with the template's patches. A template whose
buffers cannot be rebound counts as a miss and is built fresh.
APS5_NO_MOVED_BUFFER_TEMPLATES=1 keys the words as before.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A RELEASE_MEM with an interrupt select drained the device: the queue
worker submitted everything recorded and waited for the GPU before it
stored the label and delivered the interrupt. PPSA21564 issues about 300
of them per frame, so the worker and the GPU took turns.
The label is now recorded like any other (on the GPU behind the work, or
as a completion action behind pending write-backs), and the delivery
becomes a completion action of the batch holding the work before the
packet (Recorder::AfterRecordedWork, after the batch's earlier
completions). It is counted with the completion labels, so idle workers
and pollers reap it, and it starts the label flush deadline. With draws
or dispatches recorded since the last submit the batch is submitted at
once. A batch holding only label stores stays open and goes out with the
next work, at the label flush deadline, at a wait, at a flip or at the
end of the submission, like any other pending label. A label the GPU
cannot store, or one that does not decode, still drains.
The interrupt so never arrives before the work ahead of it completed and
its label landed; it can only arrive later, by at most the flush
deadline plus the work recorded into the same batch.
Every vertex attribute copied its own range of guest memory, so a draw
over an interleaved buffer copied it once per attribute (seven or eleven
times for PPSA21564's level meshes). Attributes with the same stride and
input rate whose first elements lie within one record now share one copy
of their union, each bound at its offset into it.
PPSA21564's final composite renders into 4K 10-bit scanout buffers
(COLOR_2_10_10_10 with the alternate swap, A2R10G10B10). No guest texture
format matched that color format, so the target stayed non-resident:
every frame's composite draw ran synchronously (about 11 ms, 143 per 10 s
at the title) and the flip read the frame back through guest memory.
The blue-low 10-bit color target now keeps its bits in the storage image
of its surface's red-low 2-10-10-10 format, as B8G8R8A8 targets map to
R8G8B8A8.
Presenting it: DisplayTexelFormat names a display buffer's texel order as
the guest-memory path reads it (R8G8B8A8 base: red low, else blue low;
the 10-bit variant: A2B10G10R10 or A2R10G10B10). ResidentPresentPath
blits a resident image only when it holds the display's 8-bit texel
format (an exact copy). A 10-bit image (whose blit would round where the
guest path drops the low two bits), the other 8-bit order or an sRGB type
is converted instead: GpuColorTransfer::DetileImage copies the image's
texels raw and runs the guest path's own conversion shader on them, so
both paths present the same pixels.
The image's write-back is unchanged: its results stay pending like any
resident target's and are stored when a reader of the memory flushes
them.
Tests: agc_driver_recorder_tests checks the format decision for both
8-bit and both 10-bit orders, and converts an image in each order,
comparing it with the CPU decode and the GPU detile of the same words in
guest memory.
ColorTargetLayout built its SW_64KB_R_X offset tables in every
constructor, and every draw constructs a layout per color target while
decoding and again while setting the target up. The tables depend only on
the element size; they are now built once per size under std::call_once
and the layout points into them.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Take scanned every retained slot of its tier for one with the same size,
usage and memory properties, so the slot bound stayed at 512. With 512
slots GTA V's in-game builds overflowed the small tier: each draw evicted
and recreated buffers, and the submission thread spent most of its time
in the driver's allocation and free calls. The slots are now kept in
lists per key, oldest first: Take pops the newest of its key, eviction
takes the oldest front. The bound is 4096 slots per tier, held to a sixth
of maxMemoryAllocationCount.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Every draw copied its index and vertex ranges from guest memory into
fresh buffers and scanned the indices, although the draws of a frame read
the same meshes again: about 10 GiB of vertex copies per 10 s at
PPSA21564's title.
CopyDrawInput now keeps the copy in the recorder's draw snapshot cache
and binds it again while the write tracker proves the range unchanged
since the copy was taken, with the same rules as the storage-buffer
snapshots: UnchangedSince on the collect made right before the copy and
an unchanged allocation registry. As GuestMemory::Read does, the flush
hook runs first, so pending GPU results over the range are stored and
stamped before the check. Unwatched memory is copied every time. An index
copy keeps its highest index; the device limit is still checked for every
draw.
The cache key includes the snapshot's use (storage, vertex, 16-bit or
32-bit index), so each use keeps its own buffer with its own usage. A
vertex copy runs from the buffer's start to its highest fetched element,
so draws over one shared vertex pool ask for many sizes: a vertex
snapshot also serves a shorter read at the same address while that
read's bytes are unchanged, and a new one replaces the shorter ones.
Vertex and index snapshots have their own least recently used pool
(1 GiB, 16384 entries), so storage copies, most of which change every
frame, do not push them out before their next use.
Tests: agc_driver_recorder_tests copies index and vertex ranges of
write-watched memory and requires a reuse (with the stored highest index)
while the bytes are unchanged, a new copy with the current bytes after a
CPU store, a MarkWritten store and a registry mutation, separate copies
per use, no reuse without a recorder, prefix reuse for vertex reads only,
and storage snapshots not evicting vertex snapshots.
A recorded draw binds a CPU copy of each read-only guest buffer it reads
in place, made for every draw. PPSA21564's title-stage draws bind the
same large light-list buffers written by compute passes (up to 16 MiB
each) hundreds of times per frame, so about 4 ms of every draw went into
copying about 30 MiB into freshly allocated host buffers.
The recorder now keeps the snapshots, least recently used first, under a
256 MiB budget and 1024 entries. A later draw of the same range binds the
kept, immutable copy while GuestMemory::UnchangedSince holds for the
write generation collected right before the copy and the guest allocation
registry generation is unchanged. A CPU store (stamped by a collect), a
driver or GPU store (MarkWritten) or a registry mutation makes the next
draw copy again. Unwatched memory has no generation and is copied for
every draw, as before.
A recency list (each entry holding its own position) gives the eviction
victim and moves a used entry to the back in O(1).
Tests: agc_driver_recorder_tests binds a write-watched, host-imported
range twice and requires the same snapshot, then a new copy with the new
bytes after a CPU store, a MarkWritten store and a registry mutation; and
checks the least recently used eviction order at the entry cap, the byte
budget and a CPU store dropping a kept snapshot.
An MRT slot below the highest written one that the draw does not write is a VK_ATTACHMENT_UNUSED
reference with a disabled blend; color targets keep their slot for export mappings, pipeline and
pass keys. DB_DEPTH_VIEW SLICE_START moves the depth and stencil planes by whole slices, so each
slice of a depth array is its own depth surface.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Draws with CB_COLOR_CONTROL MODE ELIMINATE_FAST_CLEAR (2) or DCC_DECOMPRESS
(6) were rejected as "only normal color rendering with copy ROP". On
hardware these full-target blits write the fast-clear value into the
fast-cleared pixels (or store the pixels uncompressed); the shader's output
never lands. In this driver's DCC model (surfaces stored uncompressed,
uniform fast-clear keys make every read return the clear value) that is:
- keys that are a clear code: the target's texels become that value and
its keys read uncompressed. Reads see the same values before and after.
- uncompressed keys, or a target without DCC: nothing changes.
- mixed keys (per-block metadata), a mipmapped DCC target (its keys are not
decoded) or a register clear of a texel over 64 bits: throw.
The texels go into the target's resident image (its refresh already
cleared it to a code's value; the register code is cleared on the GPU with
FillClear) and reach memory through its write-back, or into guest memory
on the CPU when there is no resident image. A resident image made under
other DCC keys is remade under the target's keys by the storage cache
before the pass reads it. DecodeColorMetadataPass
requires the draw to be only that pass: copy ROP, no depth or stencil
work, no multisampling, a viewport and scissor covering each target.
Indirect passes throw. The CB_COLOR decode moves into DecodeColorBuffer,
which also reads CB_COLOR_CLEAR_WORD0/1.
The register clear code (DCC key 0x20) had no value before: its keys were
read as the stored texels. A target under register keys now reads as its
CB_COLOR_CLEAR_WORD texel: a resident target is cleared to it before a draw
renders into it (then its keys read uncompressed), and a non-resident
target's copy starts from it.
Every decision that materializes a clear reads the keys through
CurrentDccKeys, which first waits for the batches up to the newest recorded
writer of the key range. The title's fast-clear fill of the metadata can
still be pending on the GPU when the next draw is decoded; reading the
stale bytes there made a later draw clear the target over the results of
the earlier ones.
The remaining CB modes (resolve, decompress, FMASK decompress) keep the
rejection, which now names the mode.
Celegans12
Dean
Dean Galvin
Nicolas Abosch
Pacuka
Michon
Alexey
SamG-Coder