Scalar f16 add/mul/sub ignore FLOAT_MODE denorm16 flush bits on main.
Flush inputs when bit 0 is clear and results when bit 1 is clear at
nearest-even. Multiply checks tininess after 11-bit rounding
(0x387ff000); add/sub use 0x38800000.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
The ternary forms share minMaxF16 with v_min_f16 and v_max_f16 and follow
the same rule on gfx1036: a subnormal half input becomes a signed zero when
bit 0 of the f16 denormal field is clear, at IEEE=0 and IEEE=1.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
v_div_fmas_f32 always quieted source NaNs and let a NaN addend
hide the default NaN from an invalid product. Reuse the existing
bit selection helpers, measured on gfx1036 in four float modes.
Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
RDNA2 min/max flush half inputs to signed zero when input
preservation is disabled, while output flushing alone keeps them.
Apply this to the measured binary forms and pin the hardware rows.
Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
With f32 denormals flushed, RDNA2 flushes a product only when its value
rounded to 24 bits in the current round mode is below 2^-126. The
tiny-product check tested the exact product, so products that round up
to 2^-126 became zero in v_mul_f32, v_mul_legacy_f32, v_mullit_f32 and
zero-addend f32 FMAs. The separately rounded v_mad_legacy_f32 and
v_mac_legacy_f32 keep the exact check, as they are not measured there.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
VOP2 0x06 measured the same as VOP3 0x140 on gfx1010 (RX 5700 XT) in
wave32 and wave64, IEEE 0 and 1 and f32 denormal modes 0 to 3: the
product is rounded before the add, f32 denormal sources, product and
result are flushed in every mode, and a zero factor gives a +0 product.
It now shares the v_mad_legacy_f32 translation with the accumulator as
the addend, and f32DenormalFlushFor returns 3 for it as well. A new
execution test pins the measured results of its VOP2 and VOP3 forms.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With FP16_OVFL=1 the hardware writes an f16 result that would round to
infinity as the largest finite value of that sign, unless a source
operand is infinite. v_div_fixup_f16 keeps the infinity it writes for a
zero denominator or an infinite numerator. Measured on RDNA2 (gfx1036)
at IEEE=0 and IEEE=1 with f16 denormals kept and flushed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The f16 min/max/min3/max3/med3 now pick NaN results on the bits and
quiet a signaling NaN only with IEEE=1. With IEEE=0, v_min_f64 and
v_max_f64 return the other operand for a NaN, not a quieted NaN. This
completes #1826 for these forms, as measured on RDNA2 (gfx1036).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
With IEEE_MODE=1 the hardware quiets a signaling NaN that v_cubesc,
v_cubetc or v_cubema_f32 returns, keeping the sign the instruction gives
it; with IEEE=0 the bits pass through. Same rule as #1867, measured on
gfx1036 in both modes.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
#2279 made agc_driver_f32_arithmetic_nan fail on NVIDIA (Windows):
v_fmamk_f32 returns a quiet NaN where the signalling one is expected
(lane 1 no float mode v_fmamk_f32 is 0x7fc12345, expected 0x7f812345),
5/5 runs on main and 5/5 passing with #2279 reverted. CI's Windows job
skips the AGC execution tests, so it did not show there.
v_fma_legacy was rewritten as a split multiply and add after #2279; it
keeps that and gets back the invalid-product argument it passed before.
Fixes#2581
With f32 denormals flushed, as in the 0xc0 float mode titles use, the
hardware flushes all three v_mullit_f32 sources and its product like
v_mul_f32, so a positive denormal limit selects -FLT_MAX and a tiny
product gives zero. The recompiler kept denormals in every mode.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The f16 forms of v_fma_mix computed an f32 FMA and converted it to f16,
so a result that lands on an f16 tie after the f32 rounding was rounded
twice. RDNA2 rounds the exact product-sum once. They now use the exact
f64 path of v_interp_p2_f16 and flush f32 sources like v_fma_mix_f32.
They follow the f16 round and denormal fields on hardware, so another
f16 mode throws, as packed f16 does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
VOP3 0x140 is v_mad_legacy_f32 on gfx10.1. Measured on gfx1010 (RX 5700
XT) in wave32 and wave64, IEEE 0 and 1 and f32 denormal modes 0 to 3, it
rounds the product before the add, flushes f32 denormal sources, product
and result in every mode, and gives a +0 product for a zero factor.
The translation multiplies, applies the zero rule and flushTinyProduct,
adds, and keeps the nanResultF32 wrapper. f32DenormalFlushFor returns 3
for the instruction before reading FLOAT_MODE. A sum of two zeros takes
its sign on the bits, as lavapipe returned -0 for +0 + -0. The execution
test pins the measured results and compares NaN bits exactly. RDNA2
(gfx1031), where the encoding is v_fma_legacy_f32 and fused, is noted in
TechnicalDebt.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#2083 chooses the NaN result of the f32 fma family on the bits: the first
NaN source, quieted in IEEE mode, with the addend skipped when the
product is infinity times zero, and 0xffc00000 otherwise. The selection
and the infinity-times-zero test ran for every fma, but a NaN source
always gives a NaN result, so for any other result they return it
unchanged.
FPNanResultFma32 carries the result and the three sources, with the IEEE
mode as a flag. The SPIR-V emitter makes the selection, including the
infinity-times-zero test, in a branch taken only for a NaN result, and
the SRT evaluator computes it exactly. Add, sub and mul keep the inline
selects: with two sources a branch costs more than it saves.
v_fma_f16, v_fmac_f16, v_fmamk_f16, v_fmaak_f16, v_pk_fma_f16 and
v_pk_fmac_f16 rounded an f32 fused multiply-add to f16, rounding twice.
The product of two f16 values is exact in f32, so the sum is now rounded
to odd in f32 with an exact error term and then rounded to nearest even
in f16, which rounds the exact result once.
With f32 denormals flushed, as in the 0xc0 float mode titles use, the
hardware flushes the denominator and numerator of v_div_fixup_f32 but
keeps a denormal quotient and result. The recompiler kept all three,
so a denormal divisor gave the quotient back instead of an infinity.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
With IEEE=1, v_pk_min_f16 and v_pk_max_f16 return a quieted NaN for a
signaling NaN operand, and DX10_CLAMP=1 clamps it to +0. The clamped
form skipped the bit-level NaN selection and saturated the driver's
min/max, which returns the other operand, so it gave that operand.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
v_ldexp_f16 read src1 with readU32, so a float inline constant gave
the low 16 bits of its f32 pattern: 0 for 0.5, 1.0, 2.0, 4.0 and
their negatives, which returned src0 unchanged, and 0xf983 for
1/(2*pi), which clamped to -64 and returned a signed zero. The
hardware treats src1 as an f16 operand, as LLVM's
LDEXP_F16_VOPProfile in VOP2Instructions.td records, so it receives
the f16 encoding: 0.5 is 0x3800, an exponent of 14336. Measured on
gfx1101, ldexp(1.0, 0.5) is +inf and ldexp(1.0, -0.5) is +0.
Read VOP3 src1 through readF16SourceBits, which maps the float inline
constants to their f16 bits and reads other operands as before.
Registers, integer inline constants and literals already matched the
sign-extended low 16 bits the hardware uses. SDWA src1 keeps its old
read, as gfx11 has no SDWA to measure it on.
With VCC set and a src2 exponent field of 128 or more (scale 2^64),
v_div_fmas_f32 recomputed an fma result with exponent field 0 as an
fma of the smaller multiplicand times 2^64, the other multiplicand and
src2 * 2^64. For |src2| >= 2^64, src2 * 2^64 overflows to infinity, so
an fma that cancels exactly returned -inf or +inf. The hardware
returns +0: measured on gfx1101 for (1.0, FLT_MAX, -FLT_MAX) and
(1.0, -2^100, 2^100) in all 16 combinations of IEEE mode, f32 and
f16/f64 denormal mode and DX10_CLAMP.
Only exact cancellations reach that path. With |src2| >= 2, a nonzero
sum below 2^-126 needs a product above 1, which is a multiple of
2^-47, while src2 is a multiple of 2^-22, so a nonzero sum is at least
2^-47 and the fma result is normal. The scaled result is then the fma
result times 2^64, +0 for a cancellation. The recomputation is now
done only for the 2^-64 scale, when the fma result overflows.
clamp (and omod, ignored while f16 denormals are enabled) were refused or
dropped per op: v_{add,sub,subrev,mul,min,max}_f16, v_ldexp_f16,
v_frexp_mant_f16, v_cvt_norm_{i,u}16_f16, v_{max3,min3,med3,div_fixup}_f16.
They now share one clamp on the f16 bits (clampF16Bits).
Co-Authored-By: Claude <noreply@anthropic.com>
Single rounding of the whole sum, f32 accumulator denormals flushed, NaN
picked on the bits, clamp ignored and the v_dot2c accumulator read from
the lane itself under DPP, all as measured on hardware.
Co-Authored-By: Claude <noreply@anthropic.com>
Adrià
Alex
Marcus
ale
Dean
Dean Galvin
Karan Vora
Jordan Hans
Adrià Franch
Liav Mordouch
Zheorgyan