VOP2 0x06 measured the same as VOP3 0x140 on gfx1010 (RX 5700 XT) in
wave32 and wave64, IEEE 0 and 1 and f32 denormal modes 0 to 3: the
product is rounded before the add, f32 denormal sources, product and
result are flushed in every mode, and a zero factor gives a +0 product.
It now shares the v_mad_legacy_f32 translation with the accumulator as
the addend, and f32DenormalFlushFor returns 3 for it as well. A new
execution test pins the measured results of its VOP2 and VOP3 forms.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With FP16_OVFL=1 the hardware writes an f16 result that would round to
infinity as the largest finite value of that sign, unless a source
operand is infinite. v_div_fixup_f16 keeps the infinity it writes for a
zero denominator or an infinite numerator. Measured on RDNA2 (gfx1036)
at IEEE=0 and IEEE=1 with f16 denormals kept and flushed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The f16 min/max/min3/max3/med3 now pick NaN results on the bits and
quiet a signaling NaN only with IEEE=1. With IEEE=0, v_min_f64 and
v_max_f64 return the other operand for a NaN, not a quieted NaN. This
completes #1826 for these forms, as measured on RDNA2 (gfx1036).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
With IEEE_MODE=1 the hardware quiets a signaling NaN that v_cubesc,
v_cubetc or v_cubema_f32 returns, keeping the sign the instruction gives
it; with IEEE=0 the bits pass through. Same rule as #1867, measured on
gfx1036 in both modes.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
#2279 made agc_driver_f32_arithmetic_nan fail on NVIDIA (Windows):
v_fmamk_f32 returns a quiet NaN where the signalling one is expected
(lane 1 no float mode v_fmamk_f32 is 0x7fc12345, expected 0x7f812345),
5/5 runs on main and 5/5 passing with #2279 reverted. CI's Windows job
skips the AGC execution tests, so it did not show there.
v_fma_legacy was rewritten as a split multiply and add after #2279; it
keeps that and gets back the invalid-product argument it passed before.
Fixes#2581
With f32 denormals flushed, as in the 0xc0 float mode titles use, the
hardware flushes all three v_mullit_f32 sources and its product like
v_mul_f32, so a positive denormal limit selects -FLT_MAX and a tiny
product gives zero. The recompiler kept denormals in every mode.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The f16 forms of v_fma_mix computed an f32 FMA and converted it to f16,
so a result that lands on an f16 tie after the f32 rounding was rounded
twice. RDNA2 rounds the exact product-sum once. They now use the exact
f64 path of v_interp_p2_f16 and flush f32 sources like v_fma_mix_f32.
They follow the f16 round and denormal fields on hardware, so another
f16 mode throws, as packed f16 does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
VOP3 0x140 is v_mad_legacy_f32 on gfx10.1. Measured on gfx1010 (RX 5700
XT) in wave32 and wave64, IEEE 0 and 1 and f32 denormal modes 0 to 3, it
rounds the product before the add, flushes f32 denormal sources, product
and result in every mode, and gives a +0 product for a zero factor.
The translation multiplies, applies the zero rule and flushTinyProduct,
adds, and keeps the nanResultF32 wrapper. f32DenormalFlushFor returns 3
for the instruction before reading FLOAT_MODE. A sum of two zeros takes
its sign on the bits, as lavapipe returned -0 for +0 + -0. The execution
test pins the measured results and compares NaN bits exactly. RDNA2
(gfx1031), where the encoding is v_fma_legacy_f32 and fused, is noted in
TechnicalDebt.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#2083 chooses the NaN result of the f32 fma family on the bits: the first
NaN source, quieted in IEEE mode, with the addend skipped when the
product is infinity times zero, and 0xffc00000 otherwise. The selection
and the infinity-times-zero test ran for every fma, but a NaN source
always gives a NaN result, so for any other result they return it
unchanged.
FPNanResultFma32 carries the result and the three sources, with the IEEE
mode as a flag. The SPIR-V emitter makes the selection, including the
infinity-times-zero test, in a branch taken only for a NaN result, and
the SRT evaluator computes it exactly. Add, sub and mul keep the inline
selects: with two sources a branch costs more than it saves.
v_fma_f16, v_fmac_f16, v_fmamk_f16, v_fmaak_f16, v_pk_fma_f16 and
v_pk_fmac_f16 rounded an f32 fused multiply-add to f16, rounding twice.
The product of two f16 values is exact in f32, so the sum is now rounded
to odd in f32 with an exact error term and then rounded to nearest even
in f16, which rounds the exact result once.
With f32 denormals flushed, as in the 0xc0 float mode titles use, the
hardware flushes the denominator and numerator of v_div_fixup_f32 but
keeps a denormal quotient and result. The recompiler kept all three,
so a denormal divisor gave the quotient back instead of an infinity.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
With IEEE=1, v_pk_min_f16 and v_pk_max_f16 return a quieted NaN for a
signaling NaN operand, and DX10_CLAMP=1 clamps it to +0. The clamped
form skipped the bit-level NaN selection and saturated the driver's
min/max, which returns the other operand, so it gave that operand.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
v_ldexp_f16 read src1 with readU32, so a float inline constant gave
the low 16 bits of its f32 pattern: 0 for 0.5, 1.0, 2.0, 4.0 and
their negatives, which returned src0 unchanged, and 0xf983 for
1/(2*pi), which clamped to -64 and returned a signed zero. The
hardware treats src1 as an f16 operand, as LLVM's
LDEXP_F16_VOPProfile in VOP2Instructions.td records, so it receives
the f16 encoding: 0.5 is 0x3800, an exponent of 14336. Measured on
gfx1101, ldexp(1.0, 0.5) is +inf and ldexp(1.0, -0.5) is +0.
Read VOP3 src1 through readF16SourceBits, which maps the float inline
constants to their f16 bits and reads other operands as before.
Registers, integer inline constants and literals already matched the
sign-extended low 16 bits the hardware uses. SDWA src1 keeps its old
read, as gfx11 has no SDWA to measure it on.
With VCC set and a src2 exponent field of 128 or more (scale 2^64),
v_div_fmas_f32 recomputed an fma result with exponent field 0 as an
fma of the smaller multiplicand times 2^64, the other multiplicand and
src2 * 2^64. For |src2| >= 2^64, src2 * 2^64 overflows to infinity, so
an fma that cancels exactly returned -inf or +inf. The hardware
returns +0: measured on gfx1101 for (1.0, FLT_MAX, -FLT_MAX) and
(1.0, -2^100, 2^100) in all 16 combinations of IEEE mode, f32 and
f16/f64 denormal mode and DX10_CLAMP.
Only exact cancellations reach that path. With |src2| >= 2, a nonzero
sum below 2^-126 needs a product above 1, which is a multiple of
2^-47, while src2 is a multiple of 2^-22, so a nonzero sum is at least
2^-47 and the fma result is normal. The scaled result is then the fma
result times 2^64, +0 for a cancellation. The recomputation is now
done only for the 2^-64 scale, when the fma result overflows.
clamp (and omod, ignored while f16 denormals are enabled) were refused or
dropped per op: v_{add,sub,subrev,mul,min,max}_f16, v_ldexp_f16,
v_frexp_mant_f16, v_cvt_norm_{i,u}16_f16, v_{max3,min3,med3,div_fixup}_f16.
They now share one clamp on the f16 bits (clampF16Bits).
Co-Authored-By: Claude <noreply@anthropic.com>
Single rounding of the whole sum, f32 accumulator denormals flushed, NaN
picked on the bits, clamp ignored and the v_dot2c accumulator read from
the lane itself under DPP, all as measured on hardware.
Co-Authored-By: Claude <noreply@anthropic.com>
- v_cndmask_b32: neg/abs on DPP; neg/abs applied on the bits, as the hardware does
- v_mul_legacy_f32: neg/abs on DPP and SDWA, clamp/omod on SDWA and VOP3
- v_ldexp_f16: neg/abs on src0 for VOP3, DPP and SDWA; NaN result chosen on the bits
Co-Authored-By: Claude <noreply@anthropic.com>
v_pack_b32_f16 applied neg/abs to bit 31 of the source dword instead of the
selected half. v_cvt_pknorm_{i16,u16}_f32 and v_ldexp_f32 used GLSL
PackSnorm2x16/PackUnorm2x16/Ldexp, which leave NaN, rounding ties and
denormals to the driver. Compute them on the bits as measured on RDNA2.
Co-Authored-By: Claude <noreply@anthropic.com>
v_ceil/floor/trunc/rndne/fract/sqrt/rsq/rcp/rcp_iflag/log/exp/sin/cos
(f32 and f16) and v_frexp_mant_f32 left NaN, invalid, zero and infinite
results to the driver: signalling NaNs came back unquieted, lavapipe lost
the NaN sign and payload, invalid results were +NaN instead of -NaN, f32
sin/cos at whole and half cycles returned tiny values instead of +0, and
v_fract_f16 of a tiny negative rounded up to 1.0.
Adds an execution test for the 27 opcodes with expected values from
hardware; approximate ops are compared with tolerances from the Vulkan
precision table.
Co-Authored-By: Claude <noreply@anthropic.com>
writeF16 dropped the clamp bit, so VOP3/SDWA f16 results with clamp
(v_rcp_f16, v_fma_f16, v_cvt_f16_f32, v_add_f16_sdwa, ...) were not
saturated. Clamp now limits to [0, 1] with NaN and -0 going to +0.
The f16 sqrt/rsq/log and sin/cos paths applied omod, which the
hardware ignores on f16 results in the default float mode, and
v_cvt_pkrtz_f16_f32 applied clamp and omod to its inputs, which the
hardware ignores as well.
Co-Authored-By: Claude <noreply@anthropic.com>
Marcus
ale
Adrià
Dean
Alex
Dean Galvin
Karan Vora
Jordan Hans
Adrià Franch
Liav Mordouch
Zheorgyan