With the f32 denormal field of FLOAT_MODE set to keep, v_cmp_X_f32 and
v_cmpx_X_f32 read a denormal operand as zero: NVIDIA, RADV and lavapipe
all flush denormals in their float compares. Compare the operands as
sign-magnitude integers instead, with NaN handled apart. Compares with
flushed denormals are unchanged. This matches RDNA2 (gfx1036) on 256
operand pairs per opcode, with f32 denormals kept and flushed.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
v_pk_add_f16 and v_pk_mul_f16 computed each half in f32 and rounded it
back to f16, then fixed the NaN, FP16_OVFL and clamp results on the
bits. On a device with native f16 that rounds to nearest even and keeps
f16 denormals (the Float16, RoundingModeRTE and DenormPreserve
capabilities that #1252 already checks), the arithmetic is now one
f16vec2 OpFAdd or OpFMul decorated NoContraction, and the same NaN,
FP16_OVFL and clamp rules run on its result bits.
The results don't change. f32 has at least 2 * 11 + 2 significand bits,
so rounding an f16 sum or product to f32 and then to f16 gives the
correctly rounded f16 result. Vulkan requires f16 OpFAdd and OpFMul to
be correctly rounded, the module keeps f16 denormals, and any NaN they
produce is replaced by the NaN the existing rules select.
v_pk_fma_f16 and v_pk_fmac_f16 keep the f32 path: Vulkan gives GLSL Fma
only the precision of OpFMul followed by OpFAdd, so a native f16 Fma
may round twice.
New test agc_driver_packed_float16_native compiles one shader with
packed add, mul, fma and fmac (op_sel, op_sel_hi, neg_lo, neg_hi and
clamp variants) with and without Float16, checks that only add and mul
use f16vec2 math, and compares every result bit for bit on random and
special inputs (zeros, denormals, infinities, NaNs, values near the f16
maximum) in nine float modes.
M0 is unsigned, so an index at or past the window reads v0 and drops its
write, v_swaprel_b32 writes neither side when either is out of range, and
the window array starts zeroed like unwritten SSA registers; the window
load counts as a lane source so a wave64 program's second half keeps its
own registers. agc_driver_relative_indexing runs in wave32 and wave64 and
pins the rows measured on gfx1035 in the review of #3856, and the
TechnicalDebt entry states the measured behaviour, the window's end at the
last named register and the sizes.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The f16 unary ops, v_frexp_mant_f16, v_frexp_exp_i16_f16, the norm
conversions and v_cvt_f32_f16 read denormal halves as they are and
rounded denormal results as they are, whatever the denormal field of
FLOAT_MODE says. On RDNA2 a clear bit 0 flushes a denormal half input
to its signed zero and a clear bit 1 flushes a denormal half result.
One shared rule now drives all of them, and v_ldexp_f16 uses it for its
input too.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
`v_movrels_b32`, `v_movreld_b32`, `v_movrelsd_b32`, `v_movrelsd_2_b32`
and `v_swaprel_b32` lowered to a compare-and-select chain over every
register from the base to the program's last allocated register, so a
program addressing 154 registers 104 times turned 3176 ISA words into
649,515 SPIR-V words that Intel's compiler did not finish. The registers
from the lowest relative base up now live in a function-scope array per
invocation (`LoadVectorWindow`/`StoreVectorWindow`), so an indexed access
is one load or store; plain accesses to those registers go through the
same array, DPP-modified relative writes keep their `DppUpdateU32` merge,
an index past the window reads 0 and drops its write, and programs
without relative addressing are unchanged. The same program recompiles
to 392,333 words with 4,992 selects instead of 32,383.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The f16 forms of v_fma_mix threw for any FLOAT_MODE with an f16 rounding
other than to nearest even. ps5-opengl's shaders run with FLOAT_MODE
0xcc, f16 rounding toward zero with f16 denormals kept, so PPSA99005
stopped at its first draw (#2631).
The exact product-sum, already computed in f64 with round to odd, is now
rounded once in the mode's direction: scaled by the f16 step at its
exponent, truncated, floored or ceiled, and scaled back. A result beyond
65504 becomes infinity or 65504 as IEEE 754 gives for the direction. On
RDNA2 (gfx1036) -1.5 * 2^-24 + 2^-126 gives 0x8002 with round toward
-infinity, and this gives the same. Flushed f16 denormals still throw.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A wave64 vertex program runs at one guest lane per invocation, so on a
32-lane host subgroup its lanes 32-63 have no invocation. The vertex
entry still set s3 = 64 | 64 << 8, telling the program that its wave
holds 64 vertices and 64 primitives. An NGG culling program then keys a
full-wave OR on lane < s3's primitive count, lanes 32-63 hold real cull
results, and its v_readlane_b32 of lane 63 cannot be built (#2783).
The entry now sets s3 = 32 | 32 << 8 for a wave64 program on a 32-lane
host, the wave the host runs; the console launches waves with fewer
vertices and primitives. The host subgroup width reaches the vertex
input the way it reaches the compute and mesh inputs
(BuildShaderStageInputInfo from HostSubgroupSize), and the source,
prepared-shader and disk cache keys already carry it for every stage.
Programs that do not read s3 are unchanged; wave32 programs and other
host widths keep the full wave.
The keys are then 0 in lanes 32-63, so the read is sound. On a 32-lane
host, when the existing half-wave scan rules fail, the scan may also:
- run under s_mov_b64 exec, -1 (EXEC constant true);
- use bound_ctrl:1 row_shr steps when 0, the fill, is the identity (OR,
add, unsigned max), also into another register, whose old value never
shows since such a step writes every lane;
- take keys shown to be 0 in lanes 32-63: built from 0 through selects
under a lane id or v_mbcnt of -1 below at most 32, v_cmp_ne_u32 against
0, ORs, DPP and permlanes, in stages whose entry EXEC is every lane.
Lane 63 is then the identity and lane 31 the subgroup reduction of the
host's invocations. A shuffle from lane 31 is not enough: an RTX 5070
runs the vertex warps of a non-indexed triangle list with 30 of their 32
invocations. A v_readlane_b32 outside the host subgroup of another value
that is 0 in lanes 32-63 reads 0. Only reads that used to throw change,
besides lane 31 of such scans.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
DecodeShaderStages accepts LS, HS and an NGG passthrough ES, reads the
3-bit VGT_TF_PARAM fields, refuses its other bits but DISTRIBUTION_MODE,
and needs HOS levels that keep [1, 63]; 0x286-0x287 join the draw key.
Local and domain programs are prepared with their stage group: its
headers are merged over one set of defaults, and a local program left
unbound by the hull header is prepared at its first draw.
The local program ends at its jump into the hull half.
TessellationLowering lowers LDS and ring addresses that are affine in
the control point and relative index to tessellation interfaces with
strides from the code, and refuses accesses outside the patch, per-patch
writes from other control points and a modified primitive export. Hull
lanes read their own saved exec bit, also of host-subgroup-masked
copies; other masks are refused, gl_InvocationID loads as a scalar and
factors clamp to 63.
normF32 took the error of p = |x| * scale from fma(|x|, scale, -p).
Vulkan only guarantees GLSL Fma on f32 the precision of a multiply
followed by an add, and lavapipe in Mesa 25.2.8 splits it, so the error
was 0 there and products that round to k + 0.5 were resolved by
roundEven: v_cvt_pknorm_u16_f32 of 0x37c000c0 gave 0x00020002 instead of
0x00010001.
scale + 1 is a power of two, so A = |x| * (scale + 1) is exact and the
product is A - |x|. Since A >= |x|, p = A - |x| and the error
(A - p) - |x| are computed with correctly rounded subtractions only
(Fast2Sum), which every driver gives exactly.
v_min_f16, v_max_f16, v_min3_f16, v_max3_f16, v_med3_f16, v_pk_min_f16
and v_pk_max_f16 converted their sources to f32, ran the f32 min, max
or med3 with its NaN and signed-zero rules, and converted the result
back to f16 before applying the f16 NaN rules on the bits. The result
is always one of the sources, or for two zeros their sign bits
combined, so it is now chosen on the f16 bits: an integer comparison of
the bits with the sign folded into an ordering key, the same AND/OR
rule for two zeros, then the existing NaN, IEEE_MODE, denormal flush
and clamp rules.
The integer comparison doesn't depend on how the driver converts f16
denormals to f32. Comparing the f32 conversions instead fails on an
Intel UHD, whose UnpackHalf2x16 makes f16 denormals zero when the
module declares no f16 DenormPreserve, as these instructions no longer
convert to f16.
New test agc_driver_float16_min_max_bits checks the seven instructions
(VOP2 and VOP3 forms with abs, neg, op_sel and clamp; packed op_sel,
op_sel_hi and neg) on random and special inputs against a model of
those rules, in seven float modes including the f16 input denormal
flush. Current main gives the same results as the model on an RTX
4060, an Intel UHD and lavapipe.
v_ldexp_f16 ignored the FLOAT_MODE denorm16 flush bits on main. Flush a
denormal half input when bit 0 is clear. When bit 1 is clear, flush the
result when its exact value is below the minimum normal half, decided
on the f32 product before the half conversion rounds it: 0x07ff scaled
by 2^-1 is zero on RDNA2, not the minimum normal. NaN handling and
destination high-half preservation stay the same.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
With the f32 denormal field of FLOAT_MODE set to keep, v_ldexp_f32
flushed subnormal inputs and results to zero. Keep subnormal inputs and
round a result that lands in the subnormal range in the f32 round mode
of FLOAT_MODE. An overflow follows the round mode as well, with or
without denormals kept: the largest finite value instead of infinity
when the mode rounds that sign toward zero. This matches RDNA2
(gfx1036) on 2163 rows in each round mode. Denormal modes 1 and 2 now
throw.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
A pixel shader whose stage info asks for alpha-to-coverage declares SampleMask, starts with every sample covered and, at its MRT0 export, clears the mask when the alpha is below one half or NaN, running with late fragment tests. The shader keeps running, as a zero coverage mask does on the GPU. The flag is part of the static shader identity and of the request serialization, now version 16.
#2839 decides f32 product tininess after rounding, in all four f32
round modes, from the exponents and a UMulHi and IMul of the
significands, and builds the flushed value from them. To nearest even,
the mode of shaders without a known float mode and of the 0xc0 most
titles use, one multiply decides it: one factor times 2^64, which is
exact below 2^64, times the other factor rounds the exact product once
to 24 bits in the normal range, and 0 < |that| < 2^-62 is tininess
after rounding. A product that rounds up to 2^-126 is then not tiny and
keeps the emitted product, which is 2^-126 there, so nothing has to be
built from the significands.
The flush of v_mul_f32 v5, v6, v7 takes 12 IR instructions instead of
34, counted as the instructions the flush select needs beyond the
product. vector_constant_factor_tininess pins 12 to nearest and 36 for
the directed path of the same instruction, and checks that the
directed modes and v_mad_legacy_f32 (before rounding) keep #2839's
check. Its round-to-nearest branches are removed as unreachable.
The f32 denormals entry says which rounding the check inherits: the
multiply is a plain OpFMul like the product, with no f32 rounding mode
requested, so it relies on the driver rounding f32 to nearest even, as
the products do. F32ConstantFactorTininess names the row a check before
rounding flushes, 0x3f7ffffe * 0x00800001, which keeps the emitted
0x00800000.
Against #2839 to nearest even, a model of both checks agrees on
1,074,977,546 operand pairs (every 24-bit significand at both rounding
thresholds, random pairs, specials) with each other and with an exact
reference. On an RTX 5070 the outputs of #2839's own tininess and
mixed-source shaders are bit-identical with either lowering over
165,298 and 164,008 generated rows in all six float modes tested.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Enable the existing input-only f32 flush policy for division scaling.
RDNA2 measurements pin all three sources and signed zeros; keep the
scaling rules unchanged and retain their separate technical debt.
Co-Authored-By: Codex gpt-6.1-sol <noreply@openai.com>
v_cvt_f32_i32 and v_cvt_f32_u32 used the host Convert which always
rounds to nearest even. When FLOAT_MODE bits 1:0 are directed, build
the f32 bits from the integer magnitude so discarded low bits follow
round toward +inf, -inf or zero, as measured on RDNA2 (gfx1036).
Co-Authored-By: Composer <noreply@cursor.com>
Directed FLOAT_MODE modes for v_cvt_f16_f32, v_cvt_f16_i16,
v_cvt_f16_u16 and v_ldexp_f16 were ignored, so results stayed
round-to-nearest-even. Hardware uses FLOAT_MODE bits 3:2 (round16);
the directed path runs only when f16 denormals are preserved.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Dean Galvin
Dean
jweidenh
Alex
Adrià
Karan Vora
fma-rc
Mouad-Scriptz-0
AlxcBTW
MarcZX100
Achraf Reyani
Celegans12
Adrià Franch
Gustavo Ariel Gamarra Rojas
Claude