Color targets with the SW_64KB_S swizzle (CB_COLOR_ATTRIB3 tile mode 9) use the standard element
order of the 4 KiB mode extended by the texture detiler's 64 KiB bits, and are resident like the
other tiled targets. The standard blocks are now wider than tall for 2- and 8-byte elements, as the
standard swizzle and the texture layout have them (they were transposed, aliasing texels).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A CMASK holds 4 KiB per 1024x512-pixel metablock, not one nibble per 8x8
tile of the target: 1920x1080 spans 0x6000 bytes, as titles clear it,
not 16200. The swizzled nibbles of the lower-right tiles lie past the
packed size, so the fast-clear check missed them and the expanded store
left them cleared.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Draws into a color target with CB_COLOR_INFO FAST_CLEAR (bit 13) were
rejected. The CMASK keeps one nibble per 8x8 tile: 0 means the tile is
fast-cleared and reads as CB_COLOR_CLEAR_WORD, 0xF means expanded (the
texels hold the color). The texture units and display scanout do not read
it, so a title runs a fast-clear eliminate pass before sampling or
flipping such a target.
DecodeColorBuffer now takes the CMASK address (CB_COLOR0_CMASK and
CB_COLOR0_CMASK_BASE_EXT) and the bytes one nibble per 8x8 tile spans.
Before a draw renders into the target, and in a fast-clear eliminate
pass, a CMASK whose nibbles are all 0 is materialized: the target is
filled with the clear texel (FillClear on its resident image, else
guest memory on the CPU) and the CMASK bytes are stored as expanded,
through the same GPU-ordered store as the DCC "uncompressed" keys
(0xFF either way). The result equals the draw followed by the eliminate
pass. An all-expanded CMASK changes nothing; a mixed one throws, as do
fast clears of DCC, mipmapped, 3D and array targets.
PPSA14632 (LEGO Horizon Adventures) fast-clears its two 3840x2160
display buffers through CMASK, so its final composite draw was dropped
on every frame and only black was presented.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A storage image uploaded under fast-clear DCC keys left the clear code
in guest memory until its write-back, so a title kernel that read the
keys after a draw or dispatch into the image saw the clear code.
PPSA26344's DCC retile kernel copies such keys into its display
buffer's metadata every frame, and the flip presented the clear color.
MarkDirty now stores uncompressed keys the first time an image uploaded
under a clear code is written, unless the title filled the keys with a
clear code since, and the image's uploaded keys become uncompressed.
While they are, Refresh and the fast revalidation read the keys as the
recorded work writing them leaves them: from the bytes pending uniform
fills and driver key stores note (NotePendingFill) when those already
make the keys read uncompressed (bytes that differ, 0xff, or a byte the
format can't use as a clear code), otherwise after waiting for that work.
The conventions check rejects changed code comments without a
TechnicalDebt change; the header line only listed the new modes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Textures in SW_256B_D, SW_4KB_D, SW_64KB_D, SW_64KB_S_T, SW_64KB_D_T,
SW_4KB_S_X and SW_4KB_D_X (tile modes 2, 6, 10, 17, 18, 21 and 22)
threw while their T# was decoded. They are detiled with address
equations generated from AMD addrlib's GFX10 swizzle patterns for the
same 16-pipe non-RB+ configuration as the 64 KiB XOR table, and share
the block shape and mip tail placement of the standard mode of their
block size. The pipe/bank XOR base check of the XOR and T modes now
uses the block size.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The import write-watch probe imports private anonymous scratch pages
through VK_EXT_external_memory_host. On amdgpu that path resolves the
pages' write protection (the userptr is revalidated at every submission),
so the probe decides that imported ranges are compared instead of
watched, and every import is unwatched. Shared direct memory is never
imported that way: its imports go through udmabuf, which takes the
memfd's pages without touching the process page tables. Unwatching them
anyway left every surface in direct memory without write stamps: each
storage image refresh re-uploaded the whole surface ("untracked"), and
draw snapshots of imported ranges could never be reused. On PPSA14632
that was over 800 GiB of uploads in 90 s.
- HostImport records whether it is a dma-buf import, and the import
decides watch or unwatch after the import by its kind (ImportWatched
already absorbs the pages the import itself touches).
- A second probe, ProbeDmaBufImportWriteProtection, imports a sealed
shared memfd scratch range through udmabuf: dma-buf imports stay
watched when a GPU read of it marks no page written and a CPU store
after that is still collected. Otherwise, or when the probe fails,
they are compared like host-pointer imports.
- Both probes also report whether a CPU store after the GPU read was
collected (ImportProbe::writtenByCpu).
- APS5_WRITE_WATCH_IMPORTS=watch or unwatch overrides both kinds.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On Linux, direct memory is a memfd mapped MAP_SHARED. amdgpu accepts only
private anonymous memory for a host pointer import (libdrm sets
AMDGPU_GEM_USERPTR_ANONONLY), so every import of direct memory failed with
VK_ERROR_INVALID_EXTERNAL_HANDLE and its ranges were copied.
When a host pointer import fails, the range's memfd slice is turned into a
dma-buf through /dev/udmabuf and imported with VK_EXT_external_memory_dma_buf.
The direct memory backing is sealed against shrinking and growing, as
udmabuf requires. libkernel reports the backing of a guest range through
GuestArenaSharedBacking. Without /dev/udmabuf, or past its size limit, the
range is copied as before. APS5_TRACE_HOST_IMPORT now prints the mappings
behind a failed import on Linux too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A sampled texture upload keeps its staging, tiled and linear buffers
until the batch that recorded it completes. A draw whose image table
uploads thousands of textures records them all into one open batch,
so in PPSA26344 6.6 GiB of device buffers were live when the next
texture allocation failed with VK_ERROR_OUT_OF_DEVICE_MEMORY.
Recorder::Keep now takes the bytes an object holds, and the sampled
upload and the storage upload and write-back scratch paths pass theirs.
Before each sampled or storage texture lookup, BoundKeptBytes submits
the open batch once it keeps 512 MiB. While the batches in flight and
those a sync or reap of this hold already finished keep more than twice
that, it destroys the finished ones' objects, then waits for the oldest
in flight and releases its objects at once instead of at the unlock.
agc_recorder_kept_bytes checks these bounds on a mock device.
A guest range mapped again at the same base and size (a fixed remap into
a registered range) kept its import, which still shows the pages pinned
or aliased before the remap. The GPU then read stale memory. Each import
now remembers its registry range (weakly, since a held range is a lease
the remap waits for) and is retired when the registry has a different
range at its base.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The validator takes the geometryShader and sampleRateShading enables from the
context (set where #295 enables them), so a device without them refuses the
draw instead of creating a module that needs features it never enabled.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A depth-only pass can leave a stale pixel program address in
SPI_SHADER_PGM_LO/HI_PS while never setting the pixel registers it does
not need. The draw then failed with "missing register at DWORD 0x1b3"
(SPI_PS_INPUT_ENA), although the hardware never launches that program:
with no color written (CB_TARGET_MASK & CB_SHADER_MASK == 0), no depth,
stencil or mask export (SPI_SHADER_Z_FORMAT 0 and those enables of
DB_SHADER_CONTROL clear), and neither KILL_ENABLE, EXEC_ON_HIER_FAIL,
EXEC_ON_NOOP nor EXEC_IF_OVERLAPPED set, the pixel shader has no effect
and is not executed.
PixelProgramUnset becomes PixelProgramSkipped: true for a zero program
address as before, and for such an inert pixel stage. Both use the
driver's null pixel program; the precheck and the decode agree.
PPSA14632 (LEGO Horizon Adventures) issues an HTILE expand this way
(DB_RENDER_CONTROL DEPTH/STENCIL_COMPRESS_DISABLE, CB disabled, depth
test and writes off) once per run.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A non-indexed mesh draw (DRAW_INDEX_AUTO, or a non-indexed indirect
record) gave the mesh program's hidden index buffer V# (user words 4..7)
a 4-byte range at the program's own code address, so the descriptor had
a guest address although the prologue never loads through it (the load
is predicated on the draw's index size). That range is bound from live
guest memory. When the program also reads guest memory by address, the
address build captures the program's code as an immutable snapshot, and
the two overlapped: the draw failed with "writable guest memory
overlaps an immutable snapshot", or, when the live bytes at the code
address had changed since the capture, "guest snapshot differs from the
writable memory it overlaps".
MeshIndexBufferDescriptor now returns a null V# (base and size 0, same
format word) for a non-indexed draw, which ShaderResources binds to its
empty placeholder buffer, and takes no unread address any more.
PPSA14632 (LEGO Horizon Adventures) hit this on its layered mesh-path
draws.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Storage images of a SINT format threw "storage image descriptor uses
an unsupported format"; only write-only 32_SINT was accepted.
The recompiler now writes 16_SINT, 8_8_SINT, 32_SINT, 16_16_SINT,
8_8_8_8_SINT, 32_32_SINT, 16_16_16_16_SINT and 32_32_32_32_SINT
storage images with the Uint numeric class. 8- and 16-bit components
are saturated (EmitFormatStoreComponent) and 32-bit ones stored
unchanged, as measured on RDNA2 (gfx1035); image_store_pck stores
32-bit components unchanged too. The driver binds these surfaces
through a cached view of the same-size UINT format.
8_SINT, 32_32_32_SINT, the packed SINT formats (11_11_10, 10_11_11,
2_10_10_10, 10_10_10_2), D16 stores, packed stores of 8- and 16-bit
components and atomics other than 32-bit ones on 32_SINT and 64-bit
ones on 32_32_SINT still throw. New test: agc_driver_image_store_sint,
with DST_SEL X,Y,0,1 on the two-channel formats, as with X,Y,Z,W the
hardware writes channel 1 from the W data slot (TechnicalDebt).
NVIDIA 610.62 cannot sample R8G8_SRGB, and lavapipe can sample neither
it nor R8_SRGB, so an 8_8 sRGB T# made an image the device does not
support: creating it crashes the NVIDIA driver and lavapipe loads zeros.
The device's format support is read once. An 8 or 8_8 sRGB texture the
device cannot sample is created and viewed as R8_UNORM or R8G8_UNORM,
and SpirvTarget::srgbDecodeFormats makes the materializer mark the
image, so image loads map every channel through a 256-entry table of
the sRGB EOTF rounded to nearest even at bf16 precision, which is what
RDNA2 (gfx1035) returns; both channels of 8_8 are sRGB, and DST_SEL 0
and 1 map to themselves. Samples and gathers of such an image throw,
and its bindless table entries are unmapped like converted formats.
APS5_SRGB_SHADER_DECODE=1 takes this path on every device. Shader disk
cache entries and keys carry the mark: cache format version 7.
A pixel or vertex shader with a 64-bit image atomic declares Int64Atomics,
Int64ImageEXT and SPV_EXT_shader_image_int64, which graphics shader
validation rejected ("SPIR-V requires unsupported device capability"),
so the draw was dropped. Compute dispatches are not validated this way.
The device records whether it enabled shaderImageInt64Atomics
(imageInt64Atomics in the device state and the graphics Context), and
ValidateShaders accepts the two capabilities and the extension when it
did.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
image_atomic_* with a two-dword data mask (four for cmpswap) is the
64-bit form of an integer image atomic. The decoder rejected it with
"invalid MIMG data mask". PPSA14632 (Unreal Engine) uses
image_atomic_umax this way in a compute shader during its first frames.
Swap, add, sub, signed and unsigned min/max, and, or, xor and cmpswap
now decode with 64-bit data and translate to new ImageAtomic*64 IR
opcodes. An image updated by them gets its own descriptor binding
range, declared as an R64ui image of 64-bit integers, and the module
uses Int64ImageEXT from SPV_EXT_shader_image_int64. The driver enables
shaderImageInt64Atomics when the device offers it and binds an
R64_UINT view of the 32_32 UINT, SINT or FLOAT surface. Without the
feature, recompiling such a shader throws.
64-bit inc, dec and the float atomics still throw, as does any other
access in the same shader to an image it updates with 64-bit atomics.
The new agc_driver_image_atomics64 test runs every operation on 32_32
UINT, SINT and FLOAT surfaces and checks the returned and stored values
against edge cases across the dword boundary.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An S# with FILTER_MODE (word0 bits 29-30) set threw in
DecodeSamplerResource. Modes 1 and 2 now map to Vulkan's MIN and MAX
sampler reduction through VK_EXT_sampler_filter_minmax, which the device
enables when it reports both filterMinmax properties; mode 3 throws.
Min/max still throws with anisotropy, a linear mip filter (lavapipe and
NVIDIA disagree there), depth comparison, color-texture compare
emulation, and, in PointFilteredSamplerWord (S# words 0 and 2), when it
filters between texels or mips of an image the recompiler point-samples.
A texture sampled through a linear min/max sampler needs
SAMPLED_IMAGE_FILTER_MINMAX on its view format; the image-sampler pairs
travel in DescriptorBinding::imageSamplers (shader disk cache format 5).
Adds the agc_driver_image_sample_reduction GPU test and extends the
sampler, emulated color compare and disk cache tests.
CB_COLOR_INFO format 1 with number type 6 (8_SRGB) threw at decode.
It now decodes as R8_SRGB with component swap 0; the other swaps and
8_8_SRGB still throw.
A device that cannot attach and blend R8_SRGB (NVIDIA, lavapipe)
renders a resident 8_SRGB target through an RGBA8_SRGB view of an
RGBA8 copy of its R8 image: the image is blitted in before the pass
and back when the pass ends, and the write mask keeps R only, so the
encode, decode and blend are the device's sRGB attachment rules.
APS5_SRGB_ATTACHMENT_PROXY=1 forces the copy. Linear, mipmapped and
3D 8_SRGB targets throw on such a device.
PPSA26344 renders its G-buffer slot 5 and a multiply-blend pass into
8_SRGB targets.
GFX10 image format 30 (10_11_11_UNORM: X and Y are unorm11 in bits
0-21, Z is unorm10 in bits 22-31, per Mesa's gfx10 tables) has no
Vulkan format, so its sampled and storage descriptors threw at decode.
Format 30 now takes the R32_UINT route of 10_11_11_UINT: the driver
binds the raw dwords as R32_UINT and the shader extracts each field and
converts it with the buffer format helpers (NormalizeFormatComponent on
loads, EmitFormatStoreComponent on stores); DST_SEL 1 reads 1.0f.
Samples, gathers, LOD queries, 16-bit data and loads or stores whose
DST_SEL names the missing W channel throw.
Converted textures are now viewed with an identity component mapping,
since the shader applies DST_SEL itself: a 10_11_11_UINT view whose X
selector was not X lost the packed dword. DCC clear codes 0001 and 1110
of a converted texture throw instead of filling one R32_UINT channel.
3D textures in the 64 KiB Z and S XOR swizzles threw. As in AMD
addrlib's GFX10 rules, a Z_X volume is thin: each depth slice is laid
out like an array slice with the 2D Z_X equation, as the R_X and D_X
volumes already were. An S_X volume is thick: it takes the standard
64 KiB thick block and mip tail with its own S3 XOR equation, generated
from addrlib for the same configuration as the other equations.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A T# of type 12 (1D array) threw while decoding. It now decodes to a
1D array dimension, checked like a 2D array with a height of one, and
holds depth + 1 layers laid out as a 2D array's. Its Vulkan image is
1D with that many layers, viewed as VK_IMAGE_VIEW_TYPE_1D_ARRAY for
sampling and storage, and the recompiler declares a 1D array shape for
arrayed 1D image resources instead of failing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An S# with FORCE_UNNORMALIZED threw in DecodeSamplerResource. It is now
bound through a Vulkan sampler with unnormalizedCoordinates when every
use stays where the S# and the host select the same texels.
ShaderInfoCollector records the live uses of each sampler after Apply
and dead-code elimination, and the emitter shares its explicit-LOD
predicate. Populate flags a direct S# with the bit that only explicit-LOD
image_sample without offset, comparison, derivatives or _a uses, paired
only with direct 1D or 2D images without a format conversion; anything
else throws. The driver checks the S# filters, clamps, TRUNC_COORD and
MC_COORD_TRUNC, creates the sampler with nearest mips and a zero LOD
range, and requires a single-level, single-layer 1D or 2D view of mip 0.
The shader disk cache format becomes 5.
sceVideoOutRegisterBuffers2 threw "DCC presentation is not implemented"
for category 1 (DCC-compressed) buffers, which PPSA26344 registers. Such
a buffer now needs DCC metadata and the tiled layout, and dcc_control may
only use the CB_DCC_CONTROL block layout bits 0x10026c.
At a flip the keys are read behind their recorded writers, one byte per
256 bytes as in the driver's DCC model. Uncompressed keys present the
stored texels as before; a uniform clear code presents its color with a
GPU clear of the scaler's source image; a pending storage image is used
when it serves the keys. Mixed or unreadable keys, clear codes the
display format has no DCC encoding for, register-clear keys over a
pending image and a clear color wider than 32 bits throw. Buffers
without DCC take the old path.
The resource materializer refused every atomic image descriptor whose
format was not 32_UINT. RDNA2 image atomics act on the raw dword and
the numeric format only affects addressing, so 32_SINT and 32_FLOAT
surfaces are now accepted with the Uint numeric class and run through
the existing integer atomics and float compare-exchange loops. Other
formats still throw, and the message now names the format.
DescriptorBinding gains imageAtomic, carried through the shader disk
cache (format version 4) and the resource content key. The driver
binds those elements through a cached R32_UINT view of R32_SINT and
R32_SFLOAT storage images.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
3D textures in SW_4KB_S or SW_64KB_S whose smaller levels reach the mip
tail were refused ("3D texture mip tails are not implemented"). The
tail now follows addrlib's GFX10 thick layout
(Gfx10Lib::ComputeSurfaceInfoMacroTiled):
- a level enters the tail once its size, rounded up, fits the thick
tail extent (the block with its height halved for 4 KiB blocks and
its width halved for 64 KiB blocks) and at most GetMaxNumMipsInTail
levels remain (5 for 4 KiB, 10 for 64 KiB);
- each slab starts with one tail block, then the other levels from the
smallest to level 0;
- tail level i sits in slot maxMipsInTail - 1 - i, at the element offset
addrlib derives from that slot's byte offset and the 256-byte thick
micro block, at every depth slice.
The tests check element addresses of five volumes (4 KiB and 64 KiB
blocks, 1 to 16 bytes per element, odd sizes, a chain that starts in
the tail) against Mesa addrlib's Addr2ComputeSurfaceAddrFromCoord for
a Navi10 configuration.
PPSA21564 clears each occlusion query slot with a 256-byte WRITE_DATA
before its begin dump, about 8,400 times per frame in one level. The
clear is larger than a deferred label, so it is recorded at once, and
Recorder::NoteLabel entered all 64 of its dwords in the pending-label
table: 64 map inserts, 64 batch dword entries and 64 group entries, then
64 lookups in CloseLabelGroup and 64 erases when the batch is reaped. A
replay of that pattern (21 clears per batch, 20 batches in flight) costs
1.6 us for the inserts, 0.7 us for the close and 1.3 us for the erases
per clear, about 31 ms of the graphics worker per frame there. The new
path costs 0.07 us per clear in the same replay.
The table serves WAIT_REG_MEM values and CPU pollers, which read 4 or 8
bytes. A store larger than 64 bytes (the bound the driver's own label
history already uses) now enters one range instead of its dwords:
- table entries inside the range are removed, as the store replaced
them before: a lookup there finds no value and falls back to the
pending-write path, where it used to find the store's value;
- the range leaves with its batch, like the dword entries;
- PendingLabelIn sees the range, and the write-evidence check, which
syncs a capture read over any pending label dword, now also syncs
over a pending range (Recorder::WideLabelIn).
The store is still a pending write of the open batch (the flush hook
syncs CPU readers), still starts the label flush deadline and is
recorded exactly as before. Stores of 64 bytes or less are unchanged.
Test: agc_driver_recorder_tests largeLabelTests (a 256-byte label enters
no dwords and removes the older entry under it but not one beside it,
its range, pending write and deadline, a later 4-byte label over it,
a 64-byte label entered in full, everything gone after the batch).
Every occlusion counter dump the recorder handles on the GPU ends the
batch's occlusion query, copies the pending results, folds them with a
compute dispatch, then resets and begins a new query. Each of those
commands was resolved through vkGetDeviceProcAddr, a loader name lookup,
on every call: seven lookups per dump on the graphics worker, and a title
dumps the counter twice per occlusion query. The mesh argument and timing
paths resolved their bind, push, dispatch and reset per call too.
They join the entry points the recorder already resolves in its
constructor: vkCmdBeginQuery, vkCmdEndQuery, vkCmdResetQueryPool,
vkCmdCopyQueryPoolResults, vkCmdBindPipeline, vkCmdPushConstants and
vkCmdDispatch. The same functions are called with the same arguments;
APS5_NO_PROC_TABLE=1 still resolves every call.
A GPU-direct storage write-back retiles the image's elements into an
uninitialised device scratch buffer and then copies whole ranges of it
into the import or the import's unit shadow. The retile writes only the
bytes elements occupy, so every other byte of the copied ranges was
stored from uninitialised scratch memory: the padding of the tile blocks
a surface covers only partly (its last block column and row), mip tail
blocks, the bytes between mips and the pitch padding of linear surfaces.
Another surface over the same memory, or the CPU, then read garbage
there instead of what the memory held. Both the block-window path and
the whole-layer path did this.
Before the retile, the bytes of the copied ranges that no element holds
are now copied into the scratch from where they are current (a fresh unit
shadow, else the import), so the write-back stores them unchanged. Whole
tile blocks every byte of which holds an element are left out
(CoveredMipBytes), so only partly covered blocks pay for the extra copy.
Tests: CoveredMipBytes against the XOR address equations of the four
64 KiB XOR modes at every element size; a device test that stores
cleared images over patterned memory (tiled with and without mip chains,
off a 64 KiB boundary, 4 KiB standard, linear; plain and watched memory)
and checks every byte that is not a texel kept its pattern.
Pixel shader color exports are compacted: export MRTi goes to the i-th
MRT slot with a non-zero CB_SHADER_MASK nibble and takes the i-th
SPI_SHADER_COL_FORMAT field. Draws still mapped export i to slot i and
read slot i's format field, so a shader with CB_SHADER_MASK 0xf000f and
SPI_SHADER_COL_FORMAT 0x44 (export 1 to slot 4) was refused with
"color export format 0".
Each color target now carries its export index next to its slot.
Attachment references, blend states, export mappings and the pipeline
and pass keys follow the export index; the color buffer, blend control
and target mask come from the slot. An unwritten export below the
highest written one stays an unused attachment.
Adrià
Michon
Chad
alfalfa-star
Dean
Celegans12
Claude
Mehran
Dean Galvin