A shader store through the BDA table records its 4 KiB page in the fault
buffer, and the driver marks those pages written after the use. The page
number was truncated to 32 bits, so a store above 16 TiB marked a page at
another address. And the driver scanned all 4096 slots after every use,
even when nothing was stored.
Slots now hold a 128 KiB chunk (its number plus one, 32 bits for any
address below 512 TiB) and a mask of its 4 KiB pages. A shader that
claims a slot appends it to a list, with a count. The driver reads the
count, decodes only the claimed slots, marks each run of pages once and
clears only those slots. A writable BDA range past 512 TiB is refused
when the table is built.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A global_store_dwordx2/x3/x4 was split into one StoreAddressU32 per
dword, each with its own write-pointer lookup and its own 4-byte store.
Across a wave those are strided partial writes, which are very slow to
host memory: RADV's clear_htile_mask kernel (one dwordx4 load and store
per thread over 640 KiB) took ~9 ms. Multi-dword stores are now one
StoreAddressU32xN; the emitter probes the whole span once and, when it
is mapped writable and 4-byte aligned, stores it as one vector (16- or
8-byte aligned) or as consecutive dwords, noting each written page.
Anything else takes the per-dword path as before, so faults and
range-spanning stores behave as they did. APS5_BDA_DWORD_STORES=1
restores per-dword stores.
AI-assisted (Claude Code).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every BDA dword read site still emitted in the entry function the add
of its first dword address with the overflow fault, a probe_bda_pointer
call, the wide loads, and one read_bda_dword_bytes call per extracted
dword. NVIDIA's pipeline compile time grows with every call left in the
entry function, so large compute programs with a few hundred such reads
spend 47-77 s in vkCreateComputePipelines.
A read site is now one call to a DontInline read_bda_span function, in
a stop and a coherent variant, taking the address, the immediate offset,
the mask of extracted dwords and the pc, which does the same steps in
the same order. Fault reasons, addresses, pcs, Volatile, the stop
behaviour and the APS5_BDA_BYTE_READS path are unchanged.
A BDA dword read whose span the probe rejects falls back to four byte
lookups with a branch each, and every fault was recorded inline with a
compare-exchange and seven stores. Both were copied into every read
site and both wave64 lane halves, so one large compute program grows
to about a million SPIR-V words and spends about 95 s in
vkCreateComputePipelines.
Faults are now recorded by one DontInline function, record_bda_fault,
and the byte fallback of a dword is read_bda_dword_bytes, in a stop
and a coherent variant, returning the value and whether the invocation
stops. Read order, fault reasons, pcs, Volatile and the stop behaviour
are unchanged, and so is the probe path.
raghav314
Scott Schuster
Celegans12
boykopovar