M0 is unsigned, so an index at or past the window reads v0 and drops its
write, v_swaprel_b32 writes neither side when either is out of range, and
the window array starts zeroed like unwritten SSA registers; the window
load counts as a lane source so a wave64 program's second half keeps its
own registers. agc_driver_relative_indexing runs in wave32 and wave64 and
pins the rows measured on gfx1035 in the review of #3856, and the
TechnicalDebt entry states the measured behaviour, the window's end at the
last named register and the sizes.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`v_movrels_b32`, `v_movreld_b32`, `v_movrelsd_b32`, `v_movrelsd_2_b32`
and `v_swaprel_b32` lowered to a compare-and-select chain over every
register from the base to the program's last allocated register, so a
program addressing 154 registers 104 times turned 3176 ISA words into
649,515 SPIR-V words that Intel's compiler did not finish. The registers
from the lowest relative base up now live in a function-scope array per
invocation (`LoadVectorWindow`/`StoreVectorWindow`), so an indexed access
is one load or store; plain accesses to those registers go through the
same array, DPP-modified relative writes keep their `DppUpdateU32` merge,
an index past the window reads 0 and drops its write, and programs
without relative addressing are unchanged. The same program recompiles
to 392,333 words with 4,992 selects instead of 32,383.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
RADV (Mesa 26.2.4, GFX12) updates the user SGPR count and draw-id flag
it uses for a mesh dispatch only when the bound mesh shader has the
grid-size/draw-id slot. A mesh shader reading neither NumWorkgroups nor
DrawIndex keeps the previous vertex pipeline's values: the dispatch
opens a 3-register SET_SH_REG, writes one value, and the CP parses the
following DISPATCH_MESH_DIRECT as register data. The gfx ring then
times out and the GPU is reset with VRAM lost.
Kena: Bridge of Spirits (PPSA01802) hits this on its layered volume
draws, which interleave with vertex draws in one command buffer.
Confirmed with RADV_DEBUG=hang (the dispatch header shows up as
SPI_SHADER_USER_DATA_GS_4) and the CP header history in the amdgpu
devcoredump.
Every mesh shader now loads NumWorkgroups.x and sets zero vertices
when it is 0, which no dispatch produces, so RADV gives it the
grid-size slot and writes as many values as it announces.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
DecodeShaderStages accepts LS, HS and an NGG passthrough ES, reads the
3-bit VGT_TF_PARAM fields, refuses its other bits but DISTRIBUTION_MODE,
and needs HOS levels that keep [1, 63]; 0x286-0x287 join the draw key.
Local and domain programs are prepared with their stage group: its
headers are merged over one set of defaults, and a local program left
unbound by the hull header is prepared at its first draw.
The local program ends at its jump into the hull half.
TessellationLowering lowers LDS and ring addresses that are affine in
the control point and relative index to tessellation interfaces with
strides from the code, and refuses accesses outside the patch, per-patch
writes from other control points and a modified primitive export. Hull
lanes read their own saved exec bit, also of host-subgroup-masked
copies; other masks are refused, gl_InvocationID loads as a scalar and
factors clamp to 63.
A pixel shader whose stage info asks for alpha-to-coverage declares SampleMask, starts with every sample covered and, at its MRT0 export, clears the mask when the alpha is below one half or NaN, running with late fragment tests. The shader keeps running, as a zero coverage mask does on the GPU. The flag is part of the static shader identity and of the request serialization, now version 16.
A geometry (NGG) strip draw on the mesh path assigned every primitive the next index positions without looking for the restart index: a triangle spanning a restart fetched vertex 0xFFFF and was drawn to wherever that read landed, and the strip parity never reset, so after an odd-length strip every triangle's winding flipped and face culling dropped it.
For an indexed strip draw with restart enabled and the restart index present, the draw now records each position's strip start (one past the last restart at or before it) and passes the table after the mesh arguments, flagged in draw word 3. The mesh prologue reads the start of its primitive's last vertex: a primitive whose start lies past its first vertex spans a restart and is made degenerate, and the others take their parity from their strip's start (or the draw's first index). The restart vertex itself fetches vertex 0. The table is built from the index bytes the draw already reads (its copy, or the guest memory an in-place import reads), next to #1968's restart-aware highest-index scan.
The new Mesh.cpp test draws quads as four-index strips each followed by a restart (16- and 32-bit indices, and an indirect draw starting mid-buffer) with front faces culled: every quad must render whole and the gaps between them stay clear. Without the table the first check fails ("a triangle crossed the restart after quad 0").
AI-assisted (Claude Code).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Route oversized compute LDS through a per-workgroup storage buffer.
Retain native shared memory for programs that fit the device limit.
Carry the spill stride through prepared shader results and dispatches.
Preserve the stride in the full-result codec and its round-trip fixture.
Keep the Linux RecompileResult size guard aligned with the added field.
Exercise interleaved workgroups and the device-limit boundary.
This topic contains only the LDS change and its reviewer-requested repair.
The base includes upstream's separately merged NaN result correction.
CB_COLOR_INFO format 6 (COLOR_10_11_11) with NUMBER_UNORM threw in
DecodeColorFormat: Vulkan has only the B10G11R11 float format. Such a
target now renders into an R32_UINT attachment in the layout of
format-30 images (unorm11 R and G in bits 0-21, unorm10 B in bits
22-31), so later image loads of the surface read the same words.
The target format is draw state, so the packing is a specialization of
the registered pixel shader: every float MRT export also computes the
packed word (EmitFormatStoreComponent) behind specialization constant
ExportPackingBase + target, and the specialized module turns a packed
target's output into a uint4 that ValidateShaders expects. The new
request field targetExportPacking is in the module key, the result memo,
the draw key and the serialized request (version 14). Blending, partial
RGB writes, UNORM16/SNORM16/UINT16 exports, DCC and ROUND_MODE throw.
Two different vertex programs that compute the same position give
bit-identical depth on the GPU, which a depth pre-pass followed by an
equal-or-greater redraw relies on. Without Invariant, a Vulkan driver may
compile the same position arithmetic differently in each shader (for
example contracting into an FMA in only one), so the redraw fails the
depth test.
Decorate the gl_PerVertex Position member and the mesh shader's Position
output Invariant, and check both in a new recompiler test.
AI-assisted: yes
v_dot2_f32_f16 rounded a.x * b.x + a.y * b.y + c once, computed with f64
TwoSums and round-to-odd. GeForce GPUs run f64 at 1/32 or 1/64 of the
f32 rate, which made it cost about 44 v_fma_f32 on an RTX 4060.
The f16 products are exact in f32, and the correctly rounded sum of
three f32 values only needs f32: two TwoSums, a round-to-odd sum of
their errors and a final add (Boldo and Melquiond, "Emulation of FMA and
Correctly Rounded Sums: Proved Algorithms Using Rounding to Odd",
IEEE TC 57(4), 2008), with round-to-odd emulated by stepping an even
rounded sum one ulp toward its error. With finite inputs, c zero or at
least 2^-100 and a result between 2^-100 and 2^127, every value involved
is zero or a multiple of 2^-123, so no driver flush can reach it. Other
inputs, and zero results, take the f64 lowering in a branch.
New test agc_driver_f32_dot2_rounding checks random, cancelling and tie
cases against an exact fixed-point sum, and the NaN, infinity and zero
rules.
EmitClassifyF32Bits found a NaN from the exponent and mantissa fields
separately. The magnitude settles it: a NaN is above 0x7f800000 and a
zero is 0.
min and max ended with two NaN selects. An ordered compare is false when
either operand is a NaN, so the ordered pick already returns rhs for a
NaN lhs; only the select that returns lhs for a NaN rhs, also when both
are NaNs, is needed.
med3 is the minimum of the three when any operand is a NaN and the
median otherwise. The median is used only without NaNs, so it is now
built from compares and the zero-sign rule alone, and the minimum reuses
the operands' NaN tests instead of classifying intermediate values.
v_sin_f32 and v_cos_f32 take their input in cycles, and the SPIR-V
backend reduced it with GLSL Fract before scaling by 2*pi. For x in
(-0.5, 0), fract(x) = 1 + x is rounded to a multiple of 2^-24, which
drops the low bits of x, and for |x| <= 2^-25 it rounds to 1.0, which
RADV's v_fract_f32 clamps to 0x3f7fffff. On RADV, v_sin_f32(-2^-30)
returned 0xb4c90fde (-3.7e-7) where RDNA3 returns 0xb1c90fd5 (-5.9e-9).
The cycle is now x - roundEven(x). It lies in [-0.5, 0.5] and is exact
for every finite x: it is x when roundEven(x) is 0, and Sterbenz's
lemma applies otherwise. Finite inputs of magnitude 2^23 or more still
reduce to +0, so the select that forced them to 0 is removed. Infinite
and NaN inputs are unaffected: the translation selects their results
on the input bits.
Measured on RDNA3 (gfx1101) with the hardware oracle, with the same
results in all 16 combinations of IEEE, DX10_CLAMP and the f32 and f16
denormal modes: over 544 inputs, v_sin_f32(x - v_rndne_f32(x)) returns
the bits of v_sin_f32(x) for 519 and is at most 11 ulp off, while
v_sin_f32(v_fract_f32(x)) matches 403 and is more than 16 ulp off for
80, all of them in (-0.5, 0). For v_cos_f32 the counts are 523 (at
most 7 ulp off) and 484 (6 more than 16 ulp off). On RADV the
recompiled shader returns the bits of the v_fract_f32 sequence before
this change and of the v_rndne_f32 sequence after it, for all 544.
v_sin_f16 and v_cos_f16 share the reduction. On RADV, with f16
denormals kept, v_sin_f16 now returns the RDNA3 bits for all 65536
inputs, where it was 1 ulp off for 6 small negative ones, and
v_cos_f16 matches for all of them before and after.
Decode the three VOP3 16-bit interpolation instructions (attribute, channel
and high-half select in the src0 field, the I/J VGPR in src1, P0 or the first
step's result in src2) and translate them with a new
GetInterpolationParameterF16 IR value. It reads the selected 16-bit half of
each vertex's parameter through VK_KHR_fragment_shader_barycentric, so P10 and
P20 are vertex differences rounded to f16; both steps are a multiply and an
add, as in the f32 forms.
An input in FP16_INTERP_MODE whose low half is defaulted is still declared at
its slot when its high half is read, and the high half takes its default from
USE_DEFAULT_ATTR1 and DEFAULT_VAL_ATTR1. Without barycentrics, on an input
without FP16_INTERP_MODE or on a pass-through input the instructions throw.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On GFX10.3 to GFX11.7, ACO lowers a subgroup-scope OpReadClockKHR to
s_getreg_b32 of SHADER_CYCLES, 20 bits zero-extended to 64, so s_memtime
wrapped every 2^20 cycles and went backwards. Detect those families from
the RADV device name and read the device clock there instead.
Fixes#985
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
DecodeState and the draw precheck rejected DB_SHADER_CONTROL bit 16,
PRIMITIVE_ORDERED_PIXEL_SHADER (POPS), which runs the ordered section
of overlapping pixels' shaders in primitive order. The bit now reaches
the recompiler as a pixel stage field, and such a pixel shader declares
PixelInterlockOrderedEXT and runs its whole body between
OpBeginInvocationInterlockEXT and OpEndInvocationInterlockEXT; an
invocation that a BDA read fault kills ends the interlock first. The
POPS collision SGPRs read 0, so a title skips its POPS wait and runs
its per-collision loop once. An image_load still reads a sampled
binding, which misses an earlier overlapping primitive's image_store in
the same draw. The device enables VK_EXT_fragment_shader_interlock when
it has fragmentShaderPixelInterlock; without it the recompile throws,
as does a nonzero POPS_OVERLAP_NUM_SAMPLES. The field is part of the
recompile cache key, the draw key and the serialized request (version 10).
On the AMD Windows driver (Radeon RDNA2 iGPU, 32.0.21030.2001)
UnpackHalf2x16 of a byte-selected half converts the unmasked source
register: an SDWA BYTE_0 source of 0xfff0fff0 reads as f16 0xfff0
instead of 0x00f0, so agc_driver_sdwa_float_selectors and
agc_driver_sdwa_ldexp_f16 fail there. EmitF16BitsToF32 now decodes
sign, exponent and mantissa with integer operations, scaling
subnormals by 2^-24, which is exact for every half and leaves the
driver nothing to fold into the conversion.
ds_add/sub/rsub/inc/dec_u64, ds_min/max_i64/u64, ds_and/or/xor/mskor_b64,
ds_cmpst_b64/f64, ds_min/max_f64, their _rtn forms and ds_wrxchg_rtn_b64
threw in the decoder.
LDS is a 32-bit workgroup array, so the two dwords are read and written
under a lock dword placed after the guest LDS, which is cleared with a
workgroup barrier at entry. The lock is acquired and released in the same
loop iteration, so lanes of one subgroup cannot deadlock on it. Stages
whose LDS is a function variable update it directly. 64-bit GDS atomics
throw.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A mesh program took its draw's index count from push constants, so an
indirect mesh-path (NGG) draw needed the CPU to read its record first,
waiting for the GPU to finish everything recorded before. In PPSA21564
that was about 170 waits per 10 s at the title, menu and level (about
1 s), and one per frame in the intro (2.2 s per 10 s).
The two free mesh push dwords now carry the device address of an argument
record (a VkDrawMeshTasksIndirectCommandEXT followed by the index count
and first index). When the address is set, the prologue takes the index
count and the first index from the record (the new MeshArgument IR value,
a load through a physical storage buffer pointer) and offsets its index
reads by the first index; a direct draw leaves the address 0 and draws as
before. Mesh modules declare the physical storage buffer capability for
it, and the driver's SPIR-V validation accepts the extension in mesh
modules.
An indexed mesh draw with one record that patches no user SGPR now stays
on the GPU. A one-thread compute pass (Graphics/shaders/MeshArguments.comp)
turns the record, read in place from its host import, into the draw's
argument record: the groups and instances of vkCmdDrawMeshTasksIndirectEXT
and the index count and first index the mesh program reads, with the CPU
path's rules (count clamped to the index buffer, nothing for an empty
record or past the device's group limits). The index buffer descriptor
covers the whole buffer and the program offsets by the first index. When
the records cannot be read in place, the CPU reads them and fills the
same argument record (ResolveMeshArguments).
EmitSetAttribute read state.inputInfo.vertex->clipSpace for every
position export. A pixel shader has no vertex input info, so a pixel
program that exports a position dereferenced a null pointer and crashed
the process; three words (0xf80008cf, 0, s_endpgm) are enough.
EmitSetAttribute now throws "vertex input info is missing for a position
export", as EmitAuxPositionExport already does for auxiliary positions.
Found by fuzzing the recompiler with mutated pixel shaders.
agc_pixel_position_export recompiles that program and expects the error;
it segfaults without the change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5WJS4ZdQynAJesCZKH5hb
A pixel wave starts with EXEC holding its covered pixels. The other lanes of
a partly covered quad run as helpers outside EXEC until the shader asks for
whole quads with s_wqm. Pixel shaders started with every invocation in EXEC,
Vulkan's helper invocations included. Those take part in subgroup
operations, so they counted in ballots (v_mbcnt, s_bcnt1 of EXEC,
v_readfirstlane), and a wave's single ds_append or atomic could be performed
by one of them: an atomic in a helper invocation does nothing and returns an
undefined value.
The entry prologue now starts a pixel shader's EXEC as !gl_HelperInvocation,
through a new StageInputKind::HelperInvocation builtin input. s_wqm still
adds the helpers of every quad with a covered pixel, so derivative code keeps
them.
The agc_driver_pixel_helper_lanes execution test draws triangles that leave
partly covered quads across the target with a pixel shader that stores each
covered pixel's v_mbcnt rank in EXEC, the s_bcnt1 count of EXEC and the
v_readfirstlane of a pixel id. Every wave's covered pixels must be lanes
0..n-1 of an n-lane EXEC whose first lane is one of them, in wave64 and
wave32. It skips on a subgroup narrower than 32, such as lavapipe's.
Both SMEM opcodes read a 64-bit counter into an SGPR pair. They now
lower to OpReadClockKHR: s_memtime with subgroup scope (the shader
clock) and s_memrealtime with device scope (the constant real-time
clock). The device enables VK_KHR_shader_clock when it has both clocks
and the backend refuses the opcodes with a clear reason otherwise.
Co-Authored-By: Claude <noreply@anthropic.com>
ds_rsub(_rtn)_u32, ds_inc_u32, ds_dec_u32, ds_mskor(_rtn)_b32,
ds_cmpst(_rtn)_b32/f32, ds_add(_rtn)_f32, ds_min/max_rtn_f32,
ds_wrap_rtn_b32 and ds_nop threw in the decoder. The new ones run as a
compare-exchange loop over the old value, with the semantics measured on an
RDNA2 GPU.
ds_min_f32 and ds_max_f32 take one data operand on gfx10, but compared the
memory with a second register (the unused data1 field, usually v0) and
stored data0 when that compare held. They now keep the minimum/maximum of
memory and data like the hardware: a NaN loses to a number, -0 < +0. Both
IR ops return the old value, so the _rtn forms share them.
Co-Authored-By: Claude <noreply@anthropic.com>
SPI_PS_INPUT_CNTL_n.OFFSET (bits 0-5) names the parameter slot pixel
input n reads, so several inputs may read one slot and an input may read
a slot the vertex stage never exports. With OFFSET bit 5 set and
FLAT_SHADE clear the input is the DEFAULT_VAL constant instead (bits 8-9:
0000, 0001, 1110, 1111). With OFFSET bit 5 and FLAT_SHADE both set the
slot in the low bits is passed through with its three vertices unchanged:
v_interp_mov P0, P10 and P20 read vertices 0, 1 and 2 rather than P0,
P1-P0 and P2-P0. The PS5 compiler writes such inputs as 0x422 and 0x423.
The old mapping masked OFFSET to five bits, so a defaulted input landed
on a real slot, and gave the second of two inputs reading one slot either
the same Location (two variables, "duplicate shader interface location")
or, when one was flat, a free Location the vertex stage never wrote
("graphics interfaces disagree at location N").
Now one input variable is declared per slot actually read, at Location =
slot, and inputs reading that slot share it. A slot read both flat and
interpolated (or through the perspective and linear pairs) is read per
vertex: flat readers take the provoking vertex, the others interpolate
with the barycentrics of their own pair. Pass-through inputs are read per
vertex at their slot without subtracting vertex 0, and reading one with
v_interp_p1/p2 is refused. Defaulted inputs read their DEFAULT_VAL
constant and declare nothing; inputs the program never reads are no
longer declared.
The geometry path's merged ES/GS programs (VGT_SHADER_STAGES_EN.GS_EN)
were compiled as mesh-stage programs whose input metadata was never
filled in, so every such draw failed with "mesh shader translation
configuration is not supported".
Recompiler:
- BuildShaderStageInputInfo takes the draw's MeshConfiguration; the mesh
input helpers (primitive size, step and counts) are implemented.
- The prologue emulates one NGG subgroup per workgroup: s2 GS_TG_INFO,
s3 the merged wave info per wave, the GS vertex offsets (ES thread
index times VGT_ESGS_RING_ITEMSIZE, odd strip triangles swapped), the
primitive id, and the ES vertex and instance ids. An indexed draw's
vertex id is read from the index buffer through a raw V# the driver
puts in hidden user words 4..7.
- s_sendmsg GS_ALLOC_REQ becomes MeshAllocate. The epilogue zeroes the
allocation first, clamps it to the output limits, culls primitives no
thread exported and decodes the 9-bit vertex indices.
- The draw parameters live in the last six push dwords. A BDA fault in
a mesh program no longer stops the invocation (as for other programs
with barriers, the fault is recorded and the read returns zero), so
every invocation reaches the subgroup barriers and the epilogue.
- The cache keys cover the mesh configuration; requests serialize its
new esgsItemSize field (version 4).
Driver:
- DecodeShaderStages reads VGT_ESGS_RING_ITEMSIZE (now in the draw key)
and sizes the subgroup for its ES vertices, GS primitives and outputs.
- Stages compile below the mesh draw parameters, which every mesh draw
pushes; the mesh stage always takes part in the push range.
- The index-buffer V# is set per draw (per record on the CPU-resolved
indirect path); the index-buffer alias check spares read-only
elements.
- The workgroup and LocalSize checks count two guest lanes per
invocation on 32-wide subgroups; mesh draws may carry a first vertex
or instance.
agc_driver_mesh_tests (EXCLUDE_FROM_ALL, needs VK_EXT_mesh_shader)
recompiles a hand-assembled merged ES/GS program in the PS5 compiler's
layout and draws indexed and non-indexed triangle lists in one- and
two-wave subgroups and a triangle strip across odd-starting subgroups,
reading the pixels back. agc_shader_replay prints a request's draw and
mesh configuration.
Only the lines this branch added lose their comments; the commits before
keep them, with the reasoning behind each change, for whoever digs into
that code later.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A dispatch sized in threads (USE_THREAD_DIMENSIONS) launches a partial last
workgroup on RDNA2, but the host rounds it up to whole groups, so the extra
threads ran: past the end of a node list they read past it and wrote over
live records.
Such dispatches now compile a separate variant (ShaderComputeStageInfo::
partialThreads, keyed only on whether it is set) whose entry EXEC excludes
threads at or past the requested size. The size reaches the shader as three
shader data dwords after the memory offsets (StageInputKind::
DispatchThreadLimit), is mixed into the dispatch cache key and the result
memo hash, and serialized in recompile requests (version 3). Whole-group
dispatches keep their variant.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- MIMG word0 bit 12 (UNORM) is accepted on every op without a sampler
(compute image stores set it, as the ISA asks); sample and gather still
reject it, and rejections name the instruction words.
- The image_sample_*_a opcodes keep the plain address layout, as in Kyty.
- The _cl LOD clamp becomes the MinLod image operand (or clamps an
explicit level) on devices with shaderResourceMinLod; a gather with a
clamp fails loudly.
- v_add_co_ci_u32, v_sub_co_ci_u32 and v_subrev_co_ci_u32 in VOP3 form
take their carry-in from the SGPR pair of their third source, not VCC:
a glyph upload's reciprocal division corrected its quotient on the
wrong lanes and UI text came out stippled.
- Pixel exports follow each MRT slot's component mapping and write only
the components of their export format (32_R, 32_GR, 32_AR).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Adrià
Adrià Franch
Dean Galvin
fma-rc
Dean
Mouad-Scriptz-0
alfalfa-star
Celegans12
Gustavo Ariel Gamarra Rojas
Karan Vora
Scott Schuster
Gjon Gojcevic
Liav Mordouch
Daan Dehing
boykopovar
nisbenz
Suryansh Prajapati
Chad
Alberto Ruiz
STL1te