A loop whose exits have no common post-dominator, such as a normal exit
to one s_endpgm tail beside an early-out to a kill tail, got no merge
block, and Structurize threw "loop at block N has no structured
merge/continue". With three or more exits, detectNaturalLoops threw
first, on a lookup of InvalidControlFlowId.
Such a loop now leaves through one new merge block that dispatches on
route variables. They are cleared before the header and set on each
exit edge to a target other than the latch's exit. Inner loops are
routed first; a routing that would change the natural loops throws.
A loop-exit tail branch whose in-body target is the continue block no
longer merges there, which spirv-val rejects ("Header block ... is
contained in the loop construct ..., but its merge block ... is not").
DecodeState, the draw precheck and DecodePixelStageInfo rejected every
nonzero CONSERVATIVE_Z_EXPORT (DB_SHADER_CONTROL bits 13-14).
EXPORT_LESS_THAN_Z (1) and EXPORT_GREATER_THAN_Z (2) promise that the
exported Z stays on one side of the interpolated Z. The value now
reaches the recompiler as a pixel stage field, and a pixel shader that
writes FragDepth declares DepthLess or DepthGreater next to
DepthReplacing, the SPIR-V modes RADV maps to these values. Vulkan lets
those modes only add a depth test of the interpolated Z before shading,
so a shader that keeps the promise renders the same. The reserved value
3 still throws.
The field is part of the recompile cache key, the draw key and the
serialized request (version 7), and ValidateShaders accepts both modes.
SOP1 opcode 33 with SDST null keeps no return address, so it is exactly
s_setpc_b64; the decoder now reads it that way, as KytyPS5 does. A call
with a real destination still throws "unsupported SOP1 opcode 33".
GraphBuilder resolves two more jump shapes. A long branch is
s_getpc_b64 and s_add_u32/s_addc_u32 or s_sub_u32/s_subb_u32
immediates, also through vcc, all in one block. A dword jump table
branches to the table base minus an entry that s_load_dword reads at an
index bounded by s_min_u32 and scaled by 4; the resolver follows
register writes back through straight-line code, and anything else is
still refused. The code table load now also handles one-dword entries.
agc_shader_control_flow checks the decode, the graphs and the refusals;
the new agc_driver_scalar_swappc runs all shapes on the GPU.
MIMG 0xe7 is the node test of image_bvh_intersect_ray with a 64-bit node
pointer: the pointer takes the first two address dwords (low, high), so
the instruction addresses 12 dwords, or 9 with A16.
The node index and address are computed from the whole pointer. A zero
base in the T# is a null BVH only for the 32-bit form: with 64-bit
pointers the T# may cover all of memory from base 0 and the pointer then
carries the node address, so only the T# size bounds the node.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ds_permute_b32 is the forward (scatter) counterpart of ds_bpermute_b32:
every active lane sends its data to lane ((addr + offset) >> 2) & 31 of
its wave. As measured on hardware, the highest sending lane wins, an
active lane nobody writes to gets 0 and inactive lanes neither send nor
receive. It lowers to one shuffle of the data, address and EXEC bit per
source lane, using the same wave base as ds_bpermute_b32.
Co-Authored-By: Claude <noreply@anthropic.com>
A selection whose one arm only returns (kill or s_endpgm) and whose other
arm branches straight to the merge of an enclosing selection had no
structured merge, so the draw was skipped. PPSA21564's BotEye G-buffer
pixel shaders (eight variants) have this shape.
When no other split applies, the edge from such a header to the shared
join is now split into an empty block, which becomes the header's merge.
image_sample_*_o takes its texel offset from an address VGPR. The emitter
accepted only an immediate there and failed with "requires a constant texel
offset for image sampling" for any other, including an offset the program
moves into the VGPR under EXEC.
An offset that resolves to a constant at its packed address position stays a
ConstOffset. Any other becomes the Offset image operand, which Vulkan allows
on sampling instructions only with VK_KHR_maintenance8's maintenance8 feature
and the ImageGatherExtended capability. The device enables maintenance8 when
available and reports it through SpirvTarget::nonConstantImageOffsets, which
the cache key, the request serialization (version 6) and the SPIRV-Tools
validator options follow. Without it the sample still fails.
agc_shader_replay --maintenance8 recompiles a request for such a device.
A conditional inside a loop with one target in the loop body and the
other outside it was taken for a break and emitted without a merge. When
the outside target is not the loop's merge but a tail of blocks that
reaches it, SPIR-V has no construct for that branch: spirv-val rejects
the module ("Selection must be structured") and the driver is free to
compile it into anything.
Such a branch now merges at its in-body target when the tail is private
to it (every tail block is dominated by the tail's first block, stays in
the loop construct and leaves only for the loop's merge), so the tail is
an "if (c) { tail; break; }" the loop already allows. Merges are given
to these after every other construct's, and one already taken leaves the
branch as before.
All 13 PPSA21564 programs whose V#s are read from GPU memory (the
deferred lighting and ray-traced passes) had two such branches in their
traversal loops. With this change none of the 74 dumped title programs
has one, and the 209 modules of a title-to-cinematic run validate apart
from three unrelated image-offset and entry-point errors.
Tests: agc_structurizer_tests structures a loop that exits through one
tail block and a loop with two exit tails, and requires every merge-less
conditional to target its loop's merge or continue.
SPI_PS_INPUT_ENA/ADDR with PERSP_CENTROID (bit 2) or LINEAR_CENTROID
(bit 6) was rejected; PPSA21564 binds 0x326 (perspective center and
centroid, linear center, POS_X/Y).
The PS input VGPR layout now follows the hardware rule: SPI_PS_INPUT_ADDR
places every input it names in SPI order (two VGPRs per I/J pair, three
for the pull model, one otherwise), loaded or not, and ENA & ADDR selects
the loaded ones. ShaderPixelStageInfo carries inputAddr (replacing
perspectiveCenterVgpr) plus perspectiveCentroid and linearCentroid; the
recompiler places each loaded input with PixelInputVgpr. This also fixes
the system inputs, which were always written from v0 (psSystemInputBase
was never set), so POS_X overwrote the center I in any draw with
barycentrics enabled.
Draws are rasterized single-sampled, where the centroid is the pixel
center: with fragment barycentrics the centroid pairs carry the center
barycentrics (BaryCoordKHR for perspective; the linear center and
centroid pairs now get BaryCoordNoPerspKHR).
NoPerspective was decided for every parameter by one flag (LINEAR_CENTER
loaded). v_interp_p2 now records whether it reads the J of a linear or a
perspective pair, and a parameter is NoPerspective when it is read
through a linear pair; parameters read through neither keep the old
rule. A parameter read through both kinds of pair throws. Only pixel
stages consult this.
The request serializer moves to version 5 (older requests read back with
their loaded inputs as the layout); the cache key and the driver's draw
key compare and hash the new fields. The unsupported-input error names
ENA and ADDR.
A selection whose region (the blocks reachable from its arms before its
merge) is also entered from outside the header cannot be expressed as a
SPIR-V selection construct, and the structurizer gave up with "semantic
block cloning is disabled". Pixel shaders hit this through a shared
discard tail (s_mov_b64 exec, 0 / exp null / s_endpgm) that several
selections branch to, compute shaders through short-circuit conditions
that jump into a shared body, and ray-traced compute passes through an
s_cbranch_execz that skips past a join into the middle of a selection.
Such a region is made single-entry in one of two ways, chosen per region
by its estimated translated size:
- Cloning (node splitting): the blocks the outside edges reach inside the
region are copied and the outside edges redirected to the copies. When
the header's merge is a shared linear return tail, giving the header a
private copy of that tail is tried first. Each block now carries an
estimate of the SPIR-V words it translates to. The per-instruction
weights come from a fit of the recompiler's per-block output (before
spirv-opt, phis excluded) over 8,300 blocks of captured shaders: VALU
60, buffer and flat 120, SMEM 180, image 220, LDS 400. SALU (8) and
exports (20) fit near zero and keep a small weight. Lane ops (120) and
image_bvh_intersect_ray (2000) were too rare there to fit and are
weighted for the wave and traversal expansions they need. A region is cloned only when it is at most 1024 estimated
words and at most an eighth of the program (at least 256), and while
all clones stay within a quarter of the program.
- Routing through a junction, for larger regions: every edge into the
entered part (from outside, and from the header's own arms) and every
edge from the arms to the merge is sent through a new empty junction
block. Each rerouted edge first passes empty blocks that assign
Boolean route variables (the IR's goto variables); the junction tests
them in order to reach the edge's original target. The entered part
then follows the junction and is emitted once. Branch conditions are
still evaluated only at the end of their original blocks: a route
variable records the edge a block already took, so a later SCC, VCC or
EXEC write cannot change the route, and in the two-lane wave64 layout
the source branch is combined across both halves as before while the
route variable holds the same constant in both. Routing is refused
when a rerouted edge is a back edge or leaves an indirect branch, when
the region wraps around to a dominating block, or when it would change
the natural loops or make the graph irreducible; cloning bounded by the
program's estimated size is the fallback, and otherwise it throws with
the header, the entered block and both reasons.
Selections whose arms only meet again after private return tails had no
post-dominator merge. When the existing rules find none, the merge is
now looked up with edges into private return tails ignored, or taken as
the other arm when one arm only returns; the candidate must still exit
the construct only through the merge. applyBlockOrder also remaps
indirect branch targets.
agc_shader_control_flow_tests (under ctest) covers hand-assembled
gfx1030 programs: the cloned shapes above, and routed ones (an
s_cbranch_execz into an expensive region with a loop, a region inside a
loop, nested entered regions, a shared body reached after SCC is
overwritten, early returns beside an entered region, an arm that only
returns). It checks that every selection construct exits only through
its merge, follows route variables to check that the structured graph
runs the same instruction ranges under the same branch conditions as
the original, bounds the cloned instructions, recompiles each program
for 64- and 32-lane subgroups, and keeps the routed execz program within
10% of the SPIR-V of the same program without the entering branch.
Given .req files it checks their graphs instead. agc_shader_replay --cfg
prints a request's graph, now with each block's estimate, before and
after structurization.
The geometry path's merged ES/GS programs (VGT_SHADER_STAGES_EN.GS_EN)
were compiled as mesh-stage programs whose input metadata was never
filled in, so every such draw failed with "mesh shader translation
configuration is not supported".
Recompiler:
- BuildShaderStageInputInfo takes the draw's MeshConfiguration; the mesh
input helpers (primitive size, step and counts) are implemented.
- The prologue emulates one NGG subgroup per workgroup: s2 GS_TG_INFO,
s3 the merged wave info per wave, the GS vertex offsets (ES thread
index times VGT_ESGS_RING_ITEMSIZE, odd strip triangles swapped), the
primitive id, and the ES vertex and instance ids. An indexed draw's
vertex id is read from the index buffer through a raw V# the driver
puts in hidden user words 4..7.
- s_sendmsg GS_ALLOC_REQ becomes MeshAllocate. The epilogue zeroes the
allocation first, clamps it to the output limits, culls primitives no
thread exported and decodes the 9-bit vertex indices.
- The draw parameters live in the last six push dwords. A BDA fault in
a mesh program no longer stops the invocation (as for other programs
with barriers, the fault is recorded and the read returns zero), so
every invocation reaches the subgroup barriers and the epilogue.
- The cache keys cover the mesh configuration; requests serialize its
new esgsItemSize field (version 4).
Driver:
- DecodeShaderStages reads VGT_ESGS_RING_ITEMSIZE (now in the draw key)
and sizes the subgroup for its ES vertices, GS primitives and outputs.
- Stages compile below the mesh draw parameters, which every mesh draw
pushes; the mesh stage always takes part in the push range.
- The index-buffer V# is set per draw (per record on the CPU-resolved
indirect path); the index-buffer alias check spares read-only
elements.
- The workgroup and LocalSize checks count two guest lanes per
invocation on 32-wide subgroups; mesh draws may carry a first vertex
or instance.
agc_driver_mesh_tests (EXCLUDE_FROM_ALL, needs VK_EXT_mesh_shader)
recompiles a hand-assembled merged ES/GS program in the PS5 compiler's
layout and draws indexed and non-indexed triangle lists in one- and
two-wave subgroups and a triangle strip across odd-starting subgroups,
reading the pixels back. agc_shader_replay prints a request's draw and
mesh configuration.
Only the lines this branch added lose their comments; the commits before
keep them, with the reasoning behind each change, for whoever digs into
that code later.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
findSelectionMerge took the nearest common post-dominator of a selection's
targets, computed across the loop's back edge. When one arm only continues
(reaching that block in a later iteration) and the other target is shared
with an enclosing selection, the shared merge was split off the header, the
post-dominator stayed where it was, and splitting repeated until the budget
threw.
Inside a loop, an arm that reaches the post-dominator only past the continue
or merge block now leaves the selection by continuing, and the merge is the
other arm's target; after one split that is the header's own synthetic merge.
agc_structurizer_tests builds the smallest such graph; it hit the budget
before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A dispatch sized in threads (USE_THREAD_DIMENSIONS) launches a partial last
workgroup on RDNA2, but the host rounds it up to whole groups, so the extra
threads ran: past the end of a node list they read past it and wrote over
live records.
Such dispatches now compile a separate variant (ShaderComputeStageInfo::
partialThreads, keyed only on whether it is set) whose entry EXEC excludes
threads at or past the requested size. The size reaches the shader as three
shader data dwords after the memory offsets (StageInputKind::
DispatchThreadLimit), is mixed into the dispatch cache key and the result
memo hash, and serialized in recompile requests (version 3). Whole-group
dispatches keep their variant.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Celegans12
Dean
Adrià Franch
DotDebian
Alexey
Dean Galvin
SamG-Coder
STL1te