v_pk_add_f16 and v_pk_mul_f16 computed each half in f32 and rounded it
back to f16, then fixed the NaN, FP16_OVFL and clamp results on the
bits. On a device with native f16 that rounds to nearest even and keeps
f16 denormals (the Float16, RoundingModeRTE and DenormPreserve
capabilities that #1252 already checks), the arithmetic is now one
f16vec2 OpFAdd or OpFMul decorated NoContraction, and the same NaN,
FP16_OVFL and clamp rules run on its result bits.
The results don't change. f32 has at least 2 * 11 + 2 significand bits,
so rounding an f16 sum or product to f32 and then to f16 gives the
correctly rounded f16 result. Vulkan requires f16 OpFAdd and OpFMul to
be correctly rounded, the module keeps f16 denormals, and any NaN they
produce is replaced by the NaN the existing rules select.
v_pk_fma_f16 and v_pk_fmac_f16 keep the f32 path: Vulkan gives GLSL Fma
only the precision of OpFMul followed by OpFAdd, so a native f16 Fma
may round twice.
New test agc_driver_packed_float16_native compiles one shader with
packed add, mul, fma and fmac (op_sel, op_sel_hi, neg_lo, neg_hi and
clamp variants) with and without Float16, checks that only add and mul
use f16vec2 math, and compares every result bit for bit on random and
special inputs (zeros, denormals, infinities, NaNs, values near the f16
maximum) in nine float modes.
`v_movrels_b32`, `v_movreld_b32`, `v_movrelsd_b32`, `v_movrelsd_2_b32`
and `v_swaprel_b32` lowered to a compare-and-select chain over every
register from the base to the program's last allocated register, so a
program addressing 154 registers 104 times turned 3176 ISA words into
649,515 SPIR-V words that Intel's compiler did not finish. The registers
from the lowest relative base up now live in a function-scope array per
invocation (`LoadVectorWindow`/`StoreVectorWindow`), so an indexed access
is one load or store; plain accesses to those registers go through the
same array, DPP-modified relative writes keep their `DppUpdateU32` merge,
an index past the window reads 0 and drops its write, and programs
without relative addressing are unchanged. The same program recompiles
to 392,333 words with 4,992 selects instead of 32,383.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
image_sample_c_lz on gfx1035 returns 0 for 8-bit sRGB, so the unorm clamp was the wrong result. Bilinear cube comparison without DISABLE_CUBE_WRAP keeps throwing because that filter blends across faces.
Co-authored-by: Cursor <cursoragent@cursor.com>
DecodeShaderStages accepts LS, HS and an NGG passthrough ES, reads the
3-bit VGT_TF_PARAM fields, refuses its other bits but DISTRIBUTION_MODE,
and needs HOS levels that keep [1, 63]; 0x286-0x287 join the draw key.
Local and domain programs are prepared with their stage group: its
headers are merged over one set of defaults, and a local program left
unbound by the hull header is prepared at its first draw.
The local program ends at its jump into the hull half.
TessellationLowering lowers LDS and ring addresses that are affine in
the control point and relative index to tessellation interfaces with
strides from the code, and refuses accesses outside the patch, per-patch
writes from other control points and a modified primitive export. Hull
lanes read their own saved exec bit, also of host-subgroup-masked
copies; other masks are refused, gl_InvocationID loads as a scalar and
factors clamp to 63.
Multi-level views, and views starting above mip 0, are bound as their base
level only when every use of the image is image_sample_lz through an
unnormalized S# without minimum LOD or LOD bias, the case measured exact on
RDNA2. Selected-LOD, implicit-LOD and offset uses keep throwing.
The recompiler records selected-LOD uses (SamplerUseNonZeroLod) and proves
the case per image (imageUnnormalizedBaseLevel); the shader disk cache
format goes to 27.
Co-Authored-By: Claude <noreply@anthropic.com>
Explicit-LOD offset samples through FORCE_UNNORMALIZED samplers now add
sign-extended texel offsets to f32 coordinates before clamping, using
OpFAdd with NoContraction and a sampler specialization selector.
Without native nonconstant-offset support, retain only runtime image
modes valid for unnormalized sampling and reject normalized samplers.
Preserve constant-swizzle handling, indirect-image rejection and main's
flat-volume and flat-line compatibility rules when selecting those modes.
Serialize image compatibility and folding flags and the sampler folding
flag in cache format 34, with roundtrip, preparation and execution coverage.
Retain main's request format 15 and runtime ABI 11.
A geometry (NGG) strip draw on the mesh path assigned every primitive the next index positions without looking for the restart index: a triangle spanning a restart fetched vertex 0xFFFF and was drawn to wherever that read landed, and the strip parity never reset, so after an odd-length strip every triangle's winding flipped and face culling dropped it.
For an indexed strip draw with restart enabled and the restart index present, the draw now records each position's strip start (one past the last restart at or before it) and passes the table after the mesh arguments, flagged in draw word 3. The mesh prologue reads the start of its primitive's last vertex: a primitive whose start lies past its first vertex spans a restart and is made degenerate, and the others take their parity from their strip's start (or the draw's first index). The restart vertex itself fetches vertex 0. The table is built from the index bytes the draw already reads (its copy, or the guest memory an in-place import reads), next to #1968's restart-aware highest-index scan.
The new Mesh.cpp test draws quads as four-index strips each followed by a restart (16- and 32-bit indices, and an indirect draw starting mid-buffer) with front faces culled: every quad must render whole and the gaps between them stay clear. Without the table the first check fails ("a triangle crossed the restart after quad 0").
AI-assisted (Claude Code).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Route oversized compute LDS through a per-workgroup storage buffer.
Retain native shared memory for programs that fit the device limit.
Carry the spill stride through prepared shader results and dispatches.
Preserve the stride in the full-result codec and its round-trip fixture.
Keep the Linux RecompileResult size guard aligned with the added field.
Exercise interleaved workgroups and the device-limit boundary.
This topic contains only the LDS change and its reviewer-requested repair.
The base includes upstream's separately merged NaN result correction.
ds_min/max/cmpst_f32 and ds_min/max/cmpst_f64 with their _rtn forms follow the
rule measured on RDNA2 (gfx1036): both operands are flushed to a signed zero
when the type's denormal field flushes, a signalling-NaN data operand is stored
quieted, else a signalling-NaN old value, else a quiet NaN loses, else the
ordered compare decides, and the flushed value is written. The shared f32
path used the host's float compares, so its NaN and denormal results were the
driver's; the shared f64 path lacked the signalling-NaN rules the buffer path
has. Both now use AtomicFloatBits. The translation marks the shared float
atomics with the mode's flush in MemoryInfo.
The new agc_driver_lds_float_atomic_modes pins 32 operand pairs per type in
the titles' mode, with both fields kept and with both flushed. The 64-bit
test's model gets the signalling-NaN rule and one lane kind now feeds the
signalling NaN of its table into min/max.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012GLfKH7Sqe51wX254k1XsV
A compute shader in Astro Bot samples an image as 2D while the bound
T# is 1D, and the recompiler refused it. Offer a 1D runtime mode for
sampled images used only by image_sample_lz and mip-less image_load,
which reads the texel the first coordinate selects, as measured on
RDNA2 in #2237.
#2279 made agc_driver_f32_arithmetic_nan fail on NVIDIA (Windows):
v_fmamk_f32 returns a quiet NaN where the signalling one is expected
(lane 1 no float mode v_fmamk_f32 is 0x7fc12345, expected 0x7f812345),
5/5 runs on main and 5/5 passing with #2279 reverted. CI's Windows job
skips the AGC execution tests, so it did not show there.
v_fma_legacy was rewritten as a split multiply and add after #2279; it
keeps that and gets back the invalid-product argument it passed before.
Fixes#2581
A global_store_dwordx2/x3/x4 was split into one StoreAddressU32 per
dword, each with its own write-pointer lookup and its own 4-byte store.
Across a wave those are strided partial writes, which are very slow to
host memory: RADV's clear_htile_mask kernel (one dwordx4 load and store
per thread over 640 KiB) took ~9 ms. Multi-dword stores are now one
StoreAddressU32xN; the emitter probes the whole span once and, when it
is mapped writable and 4-byte aligned, stores it as one vector (16- or
8-byte aligned) or as consecutive dwords, noting each written page.
Anything else takes the per-dword path as before, so faults and
range-spanning stores behave as they did. APS5_BDA_DWORD_STORES=1
restores per-dword stores.
AI-assisted (Claude Code).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A 3D image access bound to a 2D T# matched no runtime image mode and
threw "image descriptor is incompatible with the static runtime image
interface" (Sonic Origins, #2023). CouchRiv measured RDNA2 (gfx1036)
with the hardware oracle: image_sample_lz with dim 3D ignores r and
reads the (u,v) texel, and image_load reads the (x,y) texel when z is
0 and returns zero otherwise. Add a 2D mode for 3D accesses whose only
uses are those two forms, and make the load return zero for z != 0.
Every other access keeps throwing.
Fixes#2023
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Without VK_KHR_fragment_shader_barycentric the recompiler dropped
v_interp_p1_f32 and made v_interp_p2_f32 read the attribute Vulkan
interpolated, but never wrote the I/J VGPRs: a pixel shader that read
them for anything else got 0, and v_interp_mov_f32 of P10/P20 (or P0 of
an interpolated input) needed PerVertexKHR and failed pipeline
validation.
The fallback translation now loads the I/J VGPRs and marks v_interp_p1
and v_interp_p2 with witnesses. After the front-end simplification
(SSA, folding, DCE, read-lane and masked-select elimination),
HostInterpolationChecker requires every p2 to read its J, and its p1's
I, from a barycentric of the parameter's kind, every p1 result to reach
only the matching p2 (through a write masked by the p2's EXEC, whose
inactive lanes keep the old register), no parameter read as raw
vertices and no slot shared by different interpolations. It then
lowers the witnesses to the host-interpolated attribute as before,
reruns DCE and masked-select elimination, and accepts the program only
when no barycentric read remains, so reusing the I/J VGPRs after
interpolating keeps host interpolation and the previous SPIR-V. Any other
program is translated with the barycentric path and marked as emulated:
its barycentrics become vec3 inputs (NoPerspective for the linear
weights) and its per-vertex parameters flat vec4[3] inputs, placed in
the free locations by LayoutBarycentricEmulation. The driver then
inserts a generated geometry stage between the vertex and fragment
shaders that copies the position and parameters, writes the one-hot
weights of each vertex and the three vertices' parameter values, so the
rasterizer interpolates the same weights the extension provides.
The stage needs geometryShader and a vertex shader before it;
tessellation, mesh and rect-list pipelines, other primitive types and
interfaces beyond 32 locations or the device's geometry limits throw.
CompiledShaderArtifact carries the emulation (shader disk cache format 21).
Hardware snaps window corners to 1/256 pixel before interpolating
(measured by yahya308 on an RTX 4070 Laptop and a Radeon 780M in #1547);
lavapipe snaps only for coverage (lp_rast.h FIXED_ORDER 8), so the test
accepts either reference per pixel at the original 1e-4 margin.
Sources:
- v_interp_p1_f32/v_interp_p2_f32/v_interp_mov_f32 semantics (P0,
P10, P20, I/J): AMD RDNA2 ISA, "Parameter Interpolation".
- BaryCoordKHR/BaryCoordNoPerspKHR weights and PerVertexKHR vertex order:
SPV_KHR_fragment_shader_barycentric, VK_KHR_fragment_shader_barycentric.
- Geometry shader triangle input order and provoking vertex: Vulkan
specification, "Primitive Topologies" and "Geometry Shading".
- Interpolation decorations decided by the fragment input: Vulkan
specification, "Interpolation Decorations".
- lavapipe interpolation: Mesa src/gallium/drivers/llvmpipe/lp_rast.h
(FIXED_ORDER 8), lp_setup_tri.c (subpixel_snap for coverage only),
lp_state_setup.c (init_args, plane setup from float positions).
When every dst_sel of a sampled view selects constant 0 or 1, a sample
or gather returns that constant whatever the texel, filter, LOD or
coordinates. Such placeholder views can have a dimension the static
runtime image interface has no mode for (a 3D view under a 2D sample,
a 1D view under a 2D gather), and a 3D one paired with an unnormalized
sampler fails the unnormalized proof.
A direct sampled image whose accesses are all non-compare samples or
gathers gets three constant runtime modes (float, uint, sint). The mode
is selected when all four dst_sel are 0 or 1, unless the view is FMASK,
depth bits, converted or sRGB-decoded. In it the emitter returns each
channel's constant (the dmask channel's, in all four texels, for a
gather) from the image's swizzle specialization constants, and the
image is left unbound. ProveUnnormalized skips the mode after the
run-time-selected check, so table roots still throw.
A program with one GLC or DLC buffer load or store decorated the whole
guest buffer array Coherent, so its other loads were coherent too. On
NVIDIA that keeps them out of the L1 cache: one GLC store made the
re-reads of a 4 KiB window about 2.7 times slower.
RDNA keeps those loads incoherent. A load without GLC or DLC can hit the
per-CU L0 and per-SA L1 caches, which are not coherent between
work-groups, and code that needs another work-group's data reads it
with GLC/DLC or invalidates the caches first (LLVM AMDGPU memory model,
GFX10: an acquiring load is followed by buffer_gl1_inv; buffer_gl0_inv).
The program metadata now records whether a program invalidates a cache
(buffer_gl0_inv, buffer_gl1_inv, s_gl1_inv, s_dcache_inv). Without one,
a program with GLC/DLC accesses gets a second array on the same binding,
buffers_coherent, decorated Coherent, and both arrays are Aliased as
buffers_u64 already is. Loads without GLC or DLC read buffers; GLC/DLC
loads, stores and atomics use buffers_coherent. A program that
invalidates a cache keeps the whole array Coherent.
#2083 chooses the NaN result of the f32 fma family on the bits: the first
NaN source, quieted in IEEE mode, with the addend skipped when the
product is infinity times zero, and 0xffc00000 otherwise. The selection
and the infinity-times-zero test ran for every fma, but a NaN source
always gives a NaN result, so for any other result they return it
unchanged.
FPNanResultFma32 carries the result and the three sources, with the IEEE
mode as a flag. The SPIR-V emitter makes the selection, including the
infinity-times-zero test, in a branch taken only for a NaN result, and
the SRT evaluator computes it exactly. Add, sub and mul keep the inline
selects: with two sources a branch costs more than it saves.
Capturing a shader threw "null or misaligned address" when the flat
SRT walk read through a pointer loaded from memory that is null or
unmapped, even when the program skips that s_load. Such a read now
yields zero and a record with its pc and address, as does a slot whose
address needs it. The registered artifact checks one flag word per
guarded slot at each ReadConst; the capture appends the flags and the
records to the flattened SRT, and a set flag raises an Unmapped BDA
fault through the fault buffer and stops the invocation as a BDA read
does. Nothing per capture enters a key: poisoned and clean captures
share one variant and specialization. Descriptors, branch conditions,
uniform fills, table bases and user-data pointers that need such a
value still throw, and poisoned results stay out of the dispatch and
draw caches. The shader disk cache format becomes 21.
Measured on RDNA2 in #697: v_interp_p1ll_f16 and v_interp_p1lv_f16 are a
fused multiply-add rounded once to f32, and v_interp_p2_f16 rounds the exact
P20 * J + S2 once to f16. The separate multiply and add matched about 85% and
4% of the measured rows.
Both steps now use FPInterpolateF32 and FPInterpolateF16: the f16 delta times
the f32 coordinate is exact in f64, the sum is rounded to odd and then once to
nearest even at the target precision, so no driver can split or reround it.
Infinite and NaN operands take the unbounded f64 result.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Decode the three VOP3 16-bit interpolation instructions (attribute, channel
and high-half select in the src0 field, the I/J VGPR in src1, P0 or the first
step's result in src2) and translate them with a new
GetInterpolationParameterF16 IR value. It reads the selected 16-bit half of
each vertex's parameter through VK_KHR_fragment_shader_barycentric, so P10 and
P20 are vertex differences rounded to f16; both steps are a multiply and an
add, as in the f32 forms.
An input in FP16_INTERP_MODE whose low half is defaulted is still declared at
its slot when its high half is read, and the high half takes its default from
USE_DEFAULT_ATTR1 and DEFAULT_VAL_ATTR1. Without barycentrics, on an input
without FP16_INTERP_MODE or on a pass-through input the instructions throw.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Adrià
Adrià Franch
Dean Galvin
Karan Vora
Mouad-Scriptz-0
ivica00
Celegans12
Chad
Scott Schuster
Jan Šlégl
Nahuel Genchi
Daan Dehing
russhil
Gjon Gojcevic
Petre
boykopovar