Decode the three VOP3 16-bit interpolation instructions (attribute, channel
and high-half select in the src0 field, the I/J VGPR in src1, P0 or the first
step's result in src2) and translate them with a new
GetInterpolationParameterF16 IR value. It reads the selected 16-bit half of
each vertex's parameter through VK_KHR_fragment_shader_barycentric, so P10 and
P20 are vertex differences rounded to f16; both steps are a multiply and an
add, as in the f32 forms.
An input in FP16_INTERP_MODE whose low half is defaulted is still declared at
its slot when its high half is read, and the high half takes its default from
USE_DEFAULT_ATTR1 and DEFAULT_VAL_ATTR1. Without barycentrics, on an input
without FP16_INTERP_MODE or on a pass-through input the instructions throw.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
image_atomic_* with a two-dword data mask (four for cmpswap) is the
64-bit form of an integer image atomic. The decoder rejected it with
"invalid MIMG data mask". PPSA14632 (Unreal Engine) uses
image_atomic_umax this way in a compute shader during its first frames.
Swap, add, sub, signed and unsigned min/max, and, or, xor and cmpswap
now decode with 64-bit data and translate to new ImageAtomic*64 IR
opcodes. An image updated by them gets its own descriptor binding
range, declared as an R64ui image of 64-bit integers, and the module
uses Int64ImageEXT from SPV_EXT_shader_image_int64. The driver enables
shaderImageInt64Atomics when the device offers it and binds an
R64_UINT view of the 32_32 UINT, SINT or FLOAT surface. Without the
feature, recompiling such a shader throws.
64-bit inc, dec and the float atomics still throw, as does any other
access in the same shader to an image it updates with 64-bit atomics.
The new agc_driver_image_atomics64 test runs every operation on 32_32
UINT, SINT and FLOAT surfaces and checks the returned and stored values
against edge cases across the dword boundary.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
GLOBAL opcode 0x34 subtracts the data register from a global dword and
clamps the result to zero, returning the previous value. It only exists
in the global segment, so the flat form throws in the decoder, and the
form without glc throws like buffer_atomic_csub does. The clamped
subtract is now a shared helper used by both opcodes.
Single rounding of the whole sum, f32 accumulator denormals flushed, NaN
picked on the bits, clamp ignored and the v_dot2c accumulator read from
the lane itself under DPP, all as measured on hardware.
Co-Authored-By: Claude <noreply@anthropic.com>
Without fetch-inactive or bound_ctrl, a v_permlane16_b32/v_permlanex16_b32
lane whose source lane is inactive keeps its old value on hardware;
AnyPS5 wrote 0 there. bound_ctrl alone now writes 0.
Co-Authored-By: Claude <noreply@anthropic.com>
v_pack_b32_f16 applied neg/abs to bit 31 of the source dword instead of the
selected half. v_cvt_pknorm_{i16,u16}_f32 and v_ldexp_f32 used GLSL
PackSnorm2x16/PackUnorm2x16/Ldexp, which leave NaN, rounding ties and
denormals to the driver. Compute them on the bits as measured on RDNA2.
Co-Authored-By: Claude <noreply@anthropic.com>
These make up the precise f64 division macro compilers emit for a / b.
Like the f32 ops, the rules were measured on an RDNA2 GPU (41152 rows,
model matches all of them):
- div_scale decides from the exponent fields only: 2^128 or 2^-128
scaling and VCC for differences >= 768 or <= -768, a denormal or huge
(>= 2045) denominator and a numerator exponent <= 53; a NaN source is
passed through unchanged and a zero input gives 0xfff8000000000000.
- div_fmas scales by 2^128 when the exponent of S2 is >= 1024, else
2^-128, before the single rounding, so it reuses the exact fma
emulation with the scale folded into its final rescale.
- div_fixup follows the ISA: NaN sources are quieted, a NaN quotient
with finite inputs becomes +-inf, exponent differences below -1075
give a signed zero.
Co-Authored-By: Claude <noreply@anthropic.com>
Decode and translate the remaining unary f64 transcendentals and
v_trig_preop_f64.
On RDNA2 hardware v_rcp_f64, v_rsq_f64 and v_sqrt_f64 are approximations
(relative error up to ~2^-24.3 over 8209 inputs; sqrt(1.0) is not even
exact), so they are emitted as exact double operations and the test
compares against the hardware value with a 2^-23 tolerance. Special
cases are exact and selected on bits: NaN inputs are quieted (RADV keeps
signaling NaN on FDiv), negative inputs give the default NaN
0xfff8000000000000, rsq(-0) = -inf, sqrt(-0) = -0.
v_trig_preop_f64 matches the hardware bit for bit on 24627 results plus
a sweep of the table tail: the 2/PI table is 1184 bits (bits below b17
read as zero), shift = 53 * S1[4:0] + max(exp - 1077, 0), +128 scale for
exp >= 1968, omod ignored. The 53-bit window is read from a private u32
table and scaled in two exact steps so only the final multiply rounds.
Stacked on the f64 infrastructure of #446.
Co-Authored-By: Claude <noreply@anthropic.com>
MUBUF opcode 0x34 subtracts the data register from a buffer dword and
clamps the result to zero, returning the previous value. The form without
glc throws: no GFX10 target is known to provide it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ds_add/sub/rsub/inc/dec_u64, ds_min/max_i64/u64, ds_and/or/xor/mskor_b64,
ds_cmpst_b64/f64, ds_min/max_f64, their _rtn forms and ds_wrxchg_rtn_b64
threw in the decoder.
LDS is a 32-bit workgroup array, so the two dwords are read and written
under a lock dword placed after the guest LDS, which is cleared with a
workgroup barrier at entry. The lock is acquired and released in the same
loop iteration, so lanes of one subgroup cannot deadlock on it. Stages
whose LDS is a function variable update it directly. 64-bit GDS atomics
throw.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The three MIMG float atomics threw as unknown opcodes. They now run as a
compare-and-swap loop on the 32-bit texel with the bit-level rules the
float buffer atomics already use (NaN, signed zero and sNaN quieting as
measured on RDNA2), so no float atomic feature is needed from the driver.
Co-Authored-By: Claude <noreply@anthropic.com>
buffer_atomic_fcmpswap, fcmpswap_x2, fmin_x2, fmax_x2, inc_x2 and
dec_x2 are compare-and-swap loops on the dword or qword. fcmpswap
compares as floats (-0 equals +0, NaN equals nothing) and fmin/fmax are
IEEE minNum/maxNum with -0 < +0, a quieted result for a signalling NaN
and the other operand for a quiet one, all with integer ops so the
driver's float controls do not matter. The existing 32-bit fmin/fmax
kept the old value whenever a NaN or two zeros were involved; it now
uses the same rule.
Co-Authored-By: Claude <noreply@anthropic.com>
buffer_atomic_inc, buffer_atomic_dec, buffer_atomic_cmpswap_x2 and
buffer_atomic_{add,sub,smin,umin,smax,umax,and,xor}_x2 threw in the decoder.
The 64-bit ones use native 64-bit SPIR-V atomics like the existing
swap_x2/or_x2; inc and dec wrap at the data operand through the existing
compare-exchange helpers.
Co-Authored-By: Claude <noreply@anthropic.com>
Decode and translate v_add/mul/fma/min/max/ldexp_f64, v_trunc/ceil/floor/
rndne/fract_f64, v_frexp_mant/exp_i32_f64 and the f64 <-> f32/i32/u32
conversions. f64 values stay as their raw bits (U64) in the IR and are
bitcast to double in SPIR-V, which needs the Float64 capability: the
device enables shaderFloat64 when it also preserves signed zero, inf and
NaN for 64-bit floats, and the shader declares SignedZeroInfNanPreserve 64.
Semantics measured on RDNA2 hardware (8209 inputs per op): literals are
the high dword, the 1/(2*pi) inline constant is the exact double,
omod is ignored, clamp maps NaN to +0, min/max quiet a
signaling NaN and drop a quiet one, NaN results are quiet, and the
integer conversions saturate with NaN -> 0. Truncation, rounding and
ldexp are done on the bits so they do not depend on the driver: lavapipe
returned wrong results for f64 RoundEven and Trunc, and GLSL Ldexp is
undefined past +-1024. Fma is NoContraction so RADV keeps it fused.
Co-Authored-By: Claude <noreply@anthropic.com>
ds_permute_b32 is the forward (scatter) counterpart of ds_bpermute_b32:
every active lane sends its data to lane ((addr + offset) >> 2) & 31 of
its wave. As measured on hardware, the highest sending lane wins, an
active lane nobody writes to gets 0 and inactive lanes neither send nor
receive. It lowers to one shuffle of the data, address and EXEC bit per
source lane, using the same wave base as ds_bpermute_b32.
Co-Authored-By: Claude <noreply@anthropic.com>
A mesh program took its draw's index count from push constants, so an
indirect mesh-path (NGG) draw needed the CPU to read its record first,
waiting for the GPU to finish everything recorded before. In PPSA21564
that was about 170 waits per 10 s at the title, menu and level (about
1 s), and one per frame in the intro (2.2 s per 10 s).
The two free mesh push dwords now carry the device address of an argument
record (a VkDrawMeshTasksIndirectCommandEXT followed by the index count
and first index). When the address is set, the prologue takes the index
count and the first index from the record (the new MeshArgument IR value,
a load through a physical storage buffer pointer) and offsets its index
reads by the first index; a direct draw leaves the address 0 and draws as
before. Mesh modules declare the physical storage buffer capability for
it, and the driver's SPIR-V validation accepts the extension in mesh
modules.
An indexed mesh draw with one record that patches no user SGPR now stays
on the GPU. A one-thread compute pass (Graphics/shaders/MeshArguments.comp)
turns the record, read in place from its host import, into the draw's
argument record: the groups and instances of vkCmdDrawMeshTasksIndirectEXT
and the index count and first index the mesh program reads, with the CPU
path's rules (count clamped to the index buffer, nothing for an empty
record or past the device's group limits). The index buffer descriptor
covers the whole buffer and the program offsets by the first index. When
the records cannot be read in place, the CPU reads them and fills the
same argument record (ResolveMeshArguments).
Decode MIMG image_atomic_cmpswap, sub, smin, smax, inc and dec and
recompile them on the image texel pointer. sub, smin and smax map to
the SPIR-V atomic of the same name. cmpswap takes the new value from
VDATA[0] and the comparand from VDATA[1] (dmask 0x3) and returns the
previous value in VDATA[0]. inc and dec use the compare-exchange loop
of ds_inc/ds_dec, whose replacement helpers are now shared.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Both SMEM opcodes read a 64-bit counter into an SGPR pair. They now
lower to OpReadClockKHR: s_memtime with subgroup scope (the shader
clock) and s_memrealtime with device scope (the constant real-time
clock). The device enables VK_KHR_shader_clock when it has both clocks
and the backend refuses the opcodes with a clear reason otherwise.
Co-Authored-By: Claude <noreply@anthropic.com>
ds_rsub(_rtn)_u32, ds_inc_u32, ds_dec_u32, ds_mskor(_rtn)_b32,
ds_cmpst(_rtn)_b32/f32, ds_add(_rtn)_f32, ds_min/max_rtn_f32,
ds_wrap_rtn_b32 and ds_nop threw in the decoder. The new ones run as a
compare-exchange loop over the old value, with the semantics measured on an
RDNA2 GPU.
ds_min_f32 and ds_max_f32 take one data operand on gfx10, but compared the
memory with a second register (the unused data1 field, usually v0) and
stored data0 when that compare held. They now keep the minimum/maximum of
memory and data like the hardware: a NaN loses to a number, -0 < +0. Both
IR ops return the old value, so the _rtn forms share them.
Co-Authored-By: Claude <noreply@anthropic.com>
SPIR-V float ops without NoContraction let the driver fuse or split them:
RADV contracted a guest v_mul_f32 + v_add_f32 into one fma, which made the
f32 division macro one ulp off. f32 add, sub, mul and fma now carry
NoContraction.
v_mad_f32, v_mac_f32, v_madmk_f32 and v_madak_f32 round the product before
the add, unlike their v_fma*/v_fmac* counterparts, but all of them were
translated to FPFma32. They now use a new FPMad32, emitted as a separate
NoContraction multiply and add.
Co-Authored-By: Claude <noreply@anthropic.com>
Every ray query returned a miss (a stub). The node test is now the GFX10
(RT IP 1.1) one AMD's GPURT software fallback defines (IntersectCommon.hlsl)
with the TriangleNode1_0 and BoxNode1_0 layouts, as SharpEmu ports it: the T#
(base, size, box grow and sort, triangle return mode) and the node pointer
are run-time values, triangle nodes return t over its denominator with the
barycentric numerators or the triangle id, float16/float32 box nodes the hit
children (clipped slabs, NaN misses, optionally sorted by entry time); an
inactive lane, a null BVH, an out-of-range node or another type returns four
invalid dwords.
The decoder addresses 11 dwords (8 with A16, the directions as halves), the
translation emits ImageBvhIntersectRay with the T#, MakeImageAddress and exec,
and the program takes the BDA path: one probe covers the node and direct loads
read it (a node the table does not map as one range records an Unmapped
fault); EmitBdaDwordReads can read every dword for the per-byte fallback.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Dispatch cache with per-key value variants and data-only hits (flat-SRT leaves the recompiler
proves pure are refreshed into the resource template instead of re-captured), dispatch and draw
recipes replaying a proven prepare, register-keyed draw entries, template data refresh, unit
shadows in device-local slabs, 64 KiB block-unit storage tracking with windowed detile, DCC
clear-key proofs, per-class GPU timing, a CPU copy path with batch read tracking, present slot
ring, rate-limited reaps and lazy completion counting, plan-failure and poisoned source memos,
bindless image tables (runtime-indexed descriptor arrays from a CPU-read table) so the
per-frame compute program that failed its plan now compiles. Worker affinity opt-in, per-draw
logs behind APS5_TRACE_DRAWS, wait-timeout reports include the recorder's view of the label.
Fixes a use-after-free behind BDA-miss dispatch snapshots and the CPU-write stamp of
UnchangedSinceCollected. Intro-video stage: 154 -> ~500 presents per 10 s at 4K.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0177gthrWEBBfBu5xJDXgBfo
Daan Dehing
alfalfa-star
Dean
Nahuel Genchi
Adrià Franch
DotDebian
Chad
Dean Galvin
Alexey
STL1te
devilismyfriend